Workflow Backend
The Workflow Backend is the internal service that bridges the Cogrion UI and Airflow for DAG and workflow operations. It handles workflow lifecycle management — creating, triggering, and monitoring pipelines — and is the layer the BFF talks to for all workflow-related actions.
What It Does
- DAG management — Create, update, and delete DAG definitions. Propagates changes to Airflow via the DAG git repository.
- Pipeline triggering — Accepts trigger requests from the BFF and dispatches them to Airflow.
- Secrets management — Uses an IRSA role with AWS Secrets Manager access to create and update Airflow connection secrets (e.g.
airflow-{workspace}/connections/*) on behalf of other services.
DAG Delivery
Like Airflow itself, the Workflow Backend has a configured DAG git repository (SSH). It can commit new DAG definitions to this repo, which Airflow's gitSync then picks up automatically.
| Config | Description |
|---|---|
dag_git_repo | Git repo URL (SSH) |
dag_git_branch | Branch to commit to |
dag_git_ssh_key | Base64-encoded SSH private key |
Storage
PostgreSQL (KubeBlocks) stores workflow metadata — pipeline definitions, run history, and state.
Go Deeper
- Airflow — the execution engine; Workflow Backend writes DAGs that Airflow picks up via gitSync
- Pipeline Backend — depends on Workflow Backend for lower-level pipeline operations