Workflow
Workflow is the managed DAG authoring and execution capability for scheduled Spark notebook workloads. It is distinct from the connector-ingestion feature documented as Pipeline.
Assessment scope
| Field | Value |
|---|---|
| Actors | Workflow viewer, editor, administrator, platform administrator, and workflow services |
| Product validation | Not yet recorded |
| Components assessed | sparqd/oqullus, sparqd/bff-api, sparqd/workflow-backend, sparqd/workflow-resource-monitoring, sparqd/spark-recommendation, sparqd/platform-stacks, and Airflow |
| Assessed environment | Realm gj76anadpsc; workspace w-t5e2kcxjd |
| Source baseline | Oqullus 8c9f240410b2; BFF 9b2fd45a4da1; workflow-backend d3d6a3ad7313; workflow-resource-monitoring 8665dd6efe13; platform-stacks a71390d12463 |
| Last verified | 3 September 2026 |
This assessment covers workflow discovery, authoring, validation, persistence, DAG publication, synchronization, schedules, manual runs, pause, clone, delete, runs, tasks, logs, XCom, clear and mark-state actions, notification implementation readiness, resource monitoring, authorization, caching, and recovery. It also covers Airflow metadata lifecycle, long-running Spark log transport, workflow concurrency, and managed execution capacity.
Connector creation and ingestion execution are excluded; those belong to Pipeline. Airflow-native DAGs that have no managed workflow record are also outside the managed Workflow contract.
Current capability boundary
| Capability | Source-backed state | Deployment evidence |
|---|---|---|
| List, search, filter, sort, and pin | Workflow-backend is canonical; UI uses it | Workflow-list filters are source-wired; live role and scale matrix pending |
| Create and edit | UI submits managed workflow definitions; backend persists and publishes DAG artifacts | End-to-end create/update evidence pending |
| Run as User | Incomplete and incorrectly exposed | Oqullus displays and submits the field and workflow-backend persists it, but the reviewed DAG-generation and runtime paths do not apply it as the execution identity; hide it until completion |
| Spark task authoring | Single-task and multi-task Spark Kubernetes forms are exposed | Live execution matrix pending |
| Condition, Empty, and Workflow Trigger tasks | Backend contract and compiler support exist | Not exposed by the current Workflow form |
| Semantic validation warnings | Backend validation endpoint exists | Current UI does not call or display it |
| Projects and permissions | The frontend uses the backward-compatible single-project path; omitted project_id resolves to core | Current single-project mode is deployed. Multi-project selection and OpenFGA-backed capabilities are implemented in backend source but not rolled out |
| Synchronization state | Creating, Applying changes, Ready, and Needs attention are modeled | Failure and recovery evidence pending |
| Pause and unpause | Routed through workflow-backend | Role matrix and reconciliation evidence pending |
| Runs, tasks, logs, XCom, graph, grid, clear, and mark state | Workflow-backend has workflow-scoped routes and OpenFGA checks in source | Implemented but inactive: UI/BFF use the generic Airflow path and deployed OpenFGA is not activated; run-state filtering is not in the Airflow list-runs contract |
| Notifications | Incomplete and incorrectly exposed | Oqullus exposes workflow assignment controls without channel management; workflow-backend has unscoped channel CRUD, returns config, implements AWS-only secret synchronization, and emits DAG callbacks; deployed delivery is unverified |
| Resource monitoring | UI, BFF proxy, and dedicated backend source exist | Implemented in pieces but not integrated by current KCL; see the subfeature assessment for deployment, trust, security, and scale boundaries |
| Airflow metadata lifecycle | One 20 GiB, single-replica PostgreSQL database; no cleanup, capacity alert, or rehearsed upgrade lifecycle identified | Source-verified; deployed growth, backup/restore, and migration evidence pending |
| Long-running Spark logs | Airflow reads and combines the complete available log before offset slicing; the UI omits the continuation token and requests the growing complete log every five seconds | Source-verified; ten-minute buffering user-observed; deployed byte and scale evidence pending |
| Concurrency and capacity | Max Active Runs is user-configurable; KubernetesExecutor and Karpenter provide task-pod and node elasticity | Source-verified; workspace admission, queue feedback, scheduler HA, and tested scale boundaries pending |
Critical journeys
| Journey | Required result |
|---|---|
| Create | Validate, persist one canonical workflow, publish one revision, show synchronization, and preserve errors |
| Edit | Preserve identity, warn about unsaved changes, publish a new revision, and never leave an old DAG orphaned |
| Schedule | Show the effective cron and timezone, distinguish paused from unscheduled, and verify the next run |
| Run now | Apply the active access contract, create exactly one run, and handle partial unpause/trigger failure safely |
| Observe | Keep definition state separate from Airflow runtime state and label stale or unavailable observations |
| Recover | Expose actionable synchronization failure, retry, edit, delete, and operator escalation paths |
| Clear or mark state | Apply the supported legacy role today and the specific workflow capability after OpenFGA rollout; confirm scope, audit, and refresh consistently |
| Delete | Apply the active access contract to canonical workflow identity and remove definition, artifacts, authorization state, and runtime visibility coherently |
| Dependency failure | Distinguish workflow-backend, Git, Airflow, authorization, monitoring, and cache failures |
| Capacity and upgrade | Prevent metadata exhaustion, retain recoverable history, and rehearse database migration before production upgrade |
| Long-running workload | Keep log reads bounded and preserve workload observability without coupling application lifetime to an Airflow task pod |
| Load burst | Apply workflow intent and workspace admission while the platform scales schedulers, task pods, and nodes internally |
Findings
Use this table as the register. Components identifies code repositories or runtime systems involved; it is not a person or team assignment. Each detailed finding states what must change in each listed component.
| ID | Finding | Components | Evidence | Impact |
|---|---|---|---|---|
ARCH-WF-001 | Route runtime operations through the managed workflow boundary | sparqd/oqullus, sparqd/bff-api, sparqd/workflow-backend, Airflow | Deployed split path source-verified; target path inactive | High |
ARCH-WF-002 | Wire one canonical Resource Monitoring backend through KCL | sparqd/oqullus, sparqd/bff-api, sparqd/workflow-resource-monitoring, sparqd/spark-recommendation, sparqd/platform-stacks | Backend source exists; current KCL wiring absent; live request pending | High |
ARCH-WF-003 | Expose concurrency intent and manage Airflow capacity internally | sparqd/oqullus, sparqd/workflow-backend, Airflow, sparqd/platform-stacks, Karpenter, Kubernetes, PostgreSQL | Source-verified; scale boundaries pending | High |
DATA-WF-001 | Preserve canonical identity when a workflow name changes | sparqd/workflow-backend, sparqd/oqullus, DAG Git, Airflow, monitoring | Source-verified | High |
DATA-WF-002 | Bind Resource Monitoring to task executions, not workflows | sparqd/workflow-resource-monitoring, sparqd/spark-recommendation, sparqd/workflow-backend, sparqd/bff-api, sparqd/oqullus, Airflow, Spark | Source-verified | High |
DATA-WF-003 | Replace periodic snapshots with execution-lifecycle metrics | sparqd/workflow-resource-monitoring, sparqd/spark-recommendation, sparqd/platform-stacks, Prometheus, Kubernetes, Spark | Source-verified; collector not KCL-deployed | High |
OPS-WF-001 | Prevent Airflow metadata exhaustion and rehearse upgrades | sparqd/platform-stacks, Airflow, KubeBlocks, PostgreSQL, platform operations | Source-verified; deployed capacity and recovery evidence pending | Critical |
PERF-WF-001 | Bound Workflow polling and request fan-out | sparqd/oqullus, sparqd/bff-api, sparqd/workflow-backend | Source-verified; scale pending | High |
PERF-WF-002 | Stop re-downloading growing Spark logs through Airflow | sparqd/oqullus, sparqd/bff-api, sparqd/workflow-backend, Airflow, Spark, sparqd/platform-stacks, Loki, Alloy | Source-verified; deployed volume and failure evidence pending | High |
PROD-WF-001 | Expose backend task types and semantic validation coherently | sparqd/oqullus, sparqd/workflow-backend | Source-verified | High |
PROD-WF-002 | Make the single-project compatibility mode explicit before multi-project rollout | sparqd/oqullus, sparqd/bff-api, sparqd/workflow-backend, OpenFGA | Current mode source-verified; future mode inactive | Medium |
PROD-WF-003 | Never substitute generated metrics for unavailable monitoring | sparqd/oqullus, resource-monitoring service | Source-verified | High |
PROD-WF-004 | Hide notifications until the channel and delivery contract is complete | sparqd/oqullus, sparqd/bff-api, sparqd/workflow-backend, sparqd/platform-stacks, Airflow | Source-verified; deployed delivery unverified | High |
PROD-WF-005 | Hide Run as User until execution identity is enforced | sparqd/oqullus, sparqd/workflow-backend, Airflow, Spark, storage, Trino | Source-verified | High |
REL-WF-001 | Make synchronization failure and retry actionable | sparqd/oqullus, sparqd/bff-api, sparqd/workflow-backend | Source-verified | High |
REL-WF-002 | Fail closed while workflow state is unknown | sparqd/oqullus, sparqd/bff-api, sparqd/workflow-backend | Source-verified | High |
REL-WF-003 | Reconcile Git publication with workflow database state | sparqd/workflow-backend, DAG Git/publisher, operations | Source-verified; fault injection pending | High |
REL-WF-004 | Keep Run now and pause state consistent | sparqd/oqullus, sparqd/bff-api, sparqd/workflow-backend, Airflow, sparqd/platform-stacks | Root cause source-verified; deployed request capture pending | High |
REL-WF-005 | Reconcile a cleared run without adding a grid column | sparqd/oqullus, sparqd/workflow-backend, Airflow | User-observed; source risk verified | High |
SEC-WF-001 | Block multi-project rollout until runtime authorization is active | sparqd/oqullus, sparqd/bff-api, sparqd/workflow-backend, OpenFGA, Airflow | Single-project legacy path source-verified; future enforcement inactive | Critical rollout blocker |
SEC-WF-003 | Protect monitoring now and add resource scoping before multi-project rollout | sparqd/bff-api, sparqd/workflow-backend, resource-monitoring service, OpenFGA | Current service boundary source-verified; resource scoping inactive | High |
SEC-WF-004 | Never delete an unmatched DAG name without authorization | sparqd/bff-api, sparqd/workflow-backend, OpenFGA, DAG Git/operations | Source path verified; deployed role evidence pending | Critical |
SEC-WF-005 | Add Pin to workflow authorization before multi-project rollout | sparqd/oqullus, sparqd/bff-api, sparqd/workflow-backend, OpenFGA | Single-project path source-verified; future enforcement inactive | High rollout blocker |
UX-WF-001 | Protect unsaved workflow edits | sparqd/oqullus; sparqd/workflow-backend only for server drafts | Source-verified; live refresh pending | Medium |
UX-WF-002 | Make run filters match the Airflow query contract | sparqd/oqullus, sparqd/bff-api, sparqd/workflow-backend, Airflow | Source-verified; live blocked at sign-in | Medium |
UX-WF-003 | Present complete and trustworthy selected-run details | sparqd/oqullus, sparqd/workflow-backend, Airflow | Source-verified | Medium |
Boundaries and interim workarounds
- Treat Spark Kubernetes as the only task type exposed by the current UI.
- Treat
coreas the backend-resolved compatibility project, not a value the user selected. Multi-project Workflow is not rolled out in the frontend. - Do not infer authorization from visible buttons. The current UI does not consume the source-defined per-workflow permission flags, and OpenFGA is not activated in the assessed deployment.
- Treat Notifications as incomplete despite the visible multi-task assignment
controls. Hide or capability-gate them until scoped channel management,
secret redaction, environment-specific connection synchronization, runtime
delivery, and failure feedback are verified. See
PROD-WF-004. - Do not expose incomplete capabilities. Hide Run as User from create, edit,
clone, and detail until its runtime identity contract is implemented and
verified. Do not infer the Airflow, Spark, storage, or Trino principal from
the currently persisted value. See
PROD-WF-005. - Do not use generated resource-monitoring sample values for capacity or cost decisions. See the Resource Monitoring assessment.
- Treat Spark recommendations as advice for a future Workflow task run. Do not use the copied runtime Python to change driver/executor resources, and do not assume generated advice is validated or effective. See the Spark Recommendation assessment.
- Avoid renaming an existing workflow until old DAG artifact cleanup is part of the supported lifecycle.
- Preserve work externally before refreshing or leaving an unsaved form.
- Treat the run-state selector as unreliable until it uses a supported backend contract. The 365 and All options can exceed Airflow's configured page limit.
- After clearing, compare unique
dag_run_idvalues in the network response; do not infer a new run from an extra rendered column alone. - Use the authorized runtime API or Airflow when logical date, data interval, trigger origin, configuration, or note is required; the current selected-run panel omits them.
- Monitor Airflow PostgreSQL capacity externally and preserve cleanup headroom;
do not begin archive-producing cleanup only after the 20 GiB volume is nearly
full. See
OPS-WF-001. - Close or leave live Log views when continuous output is unnecessary. The
current five-second refresh omits Airflow's continuation token, causing a
complete growing-log response each time; even correct continuation would not
prevent Airflow 2.10/2.11 from reading and reconstructing the complete source
before slicing. Remote object logging is not a guaranteed independent live
tail. See
PERF-WF-002. - Use Max Active Runs to limit overlap. Keep scheduler, worker, pool, node, and
database controls platform-managed; they are not supported workflow inputs.
See
ARCH-WF-003. - For a workflow in Needs attention, edit or delete is available, but the backend synchronization retry is not exposed through the current BFF/UI.
- Runtime closure requires role-based evidence in
gj76anadpsc, workspacew-t5e2kcxjd; source verification alone does not prove deployed configuration.