Skip to content

gsoc26: Declarative Workflow Engine #79

Description

@DhanashreePetare

Feature Request

Describe the feature you'd like

  • A YAML-based pipeline system that chains download, deploy, and delete steps into a single sequential workflow, run via databusclient workflow run pipeline.yml. Each step's output can be referenced by later steps using ${steps.step_name.output_files} syntax, resolved at runtime.
  • Each step declares its own on_error behavior: fail, continue, or retry (with configurable max_attempts/delay_seconds).
  • Environment variables (${VAR_NAME}) are substituted at parse time, with a clear early error if a referenced variable is missing.

Why is this feature important?
Today, running a multi-step Databus operation (e.g. download → deploy → delete old version) requires manually chaining CLI commands, with no structured record of what was run and no way to reuse a pipeline definition. This closes that gap, building on the existing stateless download()/deploy()/delete() functions without modifying their internals.

Additional context
Depends on the manifest system (#64) and replay (#65) already being in place, since a workflow run reuses the same underlying API functions.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions