Workflow architecture

Architecture status model

This assessment keeps three states in one document:

Label	Meaning
Deployed	Traffic and configuration active in gj76anadpsc, workspace w-t5e2kcxjd
Implemented but inactive	Code exists, but it is not reachable from the current product path or its required deployment capability is not activated
Target	Intended architecture and acceptance boundary; not a claim about current behavior

OpenFGA is implemented but inactive for this assessment. Workflow-backend contains OpenFGA relations, permission fields, and workflow-scoped runtime checks in source, but OpenFGA is not activated in the assessed deployment. Those checks must not be described as deployed protection.

Deployed architecture

Definition and selected lifecycle operations use:

Oqullus → BFF Workflow facade → workflow-backend

Most runtime detail and recovery operations use:

Oqullus → BFF generic Airflow facade → Airflow

This includes runs, task instances, logs, XCom, source, graph, grid, clear, mark-state, and manual trigger behavior. The deployed generic route validates the caller for its Airflow audience, but this assessment has no evidence of an active workflow/project-level OpenFGA decision on that path.

Implemented but inactive architecture

Workflow-backend source implements explicit name-based runtime routes that resolve a managed workflow and call operation-specific OpenFGA checks. This is not the deployed browser path:

Oqullus ⇢ BFF Workflow runtime facade ⇢ workflow-backend ⇢ OpenFGA ⇢ Airflow
          not migrated                 source exists       not activated

The BFF Workflow facade does not expose the runtime surface, Oqullus does not call it, and OpenFGA is not activated. This implementation is useful target scaffolding, not a current security control.

Target control plane

The workflow backend becomes the browser-facing control plane. It resolves a managed workflow, checks the requested action against the canonical workflow UUID, and uses backend credentials for Airflow. The browser should never select an arbitrary Airflow DAG identifier or receive infrastructure credentials.

The architecture must converge on one managed workflow boundary. Airflow remains the runtime authority, but it is not the product authorization or identity authority.

State and intended authority

State	Authority	Architecture status	Notes
Workflow UUID, project, name, definition, creator, revision	Workflow database	Deployed	Canonical product record
Per-workflow permissions	OpenFGA	Implemented but inactive	Source models and returns permissions; deployed enforcement is not active and UI does not consume them
Authorization synchronization	Workflow authorization outbox	Implemented but inactive	Source contains durable synchronization; effectiveness depends on activating OpenFGA
Generated DAG definition	DAG Git repository	Deployed	Published synchronously during create/update
DAG discovery and applied revision	Airflow observed through workflow backend	Deployed source path; live failure matrix pending	Projected as Creating, Applying changes, Ready, or Needs attention
Schedule execution, runs, tasks, logs, XCom	Airflow	Deployed	Current browser access uses the generic Airflow facade
Effective pause	Airflow, projected into workflow database	Deployed source path; live reconciliation pending	Backend changes projection only after Airflow accepts the command
Personal pin	Workflow database keyed by workflow and username	Deployed source path; live authorization pending	Current pin mutation lacks an active workflow authorization boundary
Notification channel and config	Workflow database plus environment secret/connection backend	Incomplete and incorrectly exposed	Oqullus exposes assignment; channel scope, authorization, redaction, AliCloud synchronization, and deployed runtime delivery are incomplete
Unsaved authoring changes	Browser component state	Deployed source behavior	No autosave, durable draft, or leave warning
Run as User	No established runtime authority	Incomplete and incorrectly exposed	Oqullus submits and workflow-backend persists the value, but reviewed DAG-generation/runtime paths do not apply it as execution identity; remove it from the UI until completion
Resource snapshots	One canonical producer and the Workflow resource Delta table	KCL configures the recommendation job as a producer; dedicated reader API is not wired	Current BFF route is keyed only by DAG name, validates authentication only, and has no KCL-provided backend URL

Run as User

Run as User is incomplete. Oqullus derives or defaults a value, displays it, and includes run_as_user_id in the workflow request. Workflow-backend requires, stores, updates, clones, and returns the value. The reviewed DAG publication, Airflow execution, Spark identity, storage credentials, and Trino identity paths do not consume it as an authoritative principal.

An incomplete identity control must not be exposed. Remove Run as User from create, edit, clone, and detail until an end-to-end identity design is implemented. Backend compatibility may retain the persisted field temporarily, but it must not be presented as user-controlled runtime behavior.

Re-exposure requires a server-derived or explicitly authorized principal, propagation into the runtime contract, and proof that Airflow, Spark, storage, Trino, and audit all report the same identity. Permanently removing the field is the simpler alternative if workflows intentionally use one platform runtime identity. See PROD-WF-005.

Target definition and publication flow

1. The user selects an authorized project and authors a definition.
2. Oqullus performs immediate form validation and requests semantic validation from workflow-backend.
3. The backend validates task types, graph structure, notification assignments, runtime identity, and project permission. Runtime identity here is a target requirement; the current Run as User field does not implement it.
4. The backend persists the canonical workflow and revision together with the authorization-outbox intent.
5. A publication worker or durable outbox publishes the requested revision to the DAG Git repository.
6. Airflow discovers and parses the revision.
7. Backend observation moves the workflow to Ready only when Airflow exposes the exact requested revision without an import error.

The current implementation performs Git publication inside the request before the workflow database commit. This avoids reporting success when publication itself fails, but it does not create an atomic boundary: a database commit failure can leave a Git revision without the corresponding product state. The same risk is reversed during delete. A durable publication intent, idempotent worker, and reconciliation process are required for recovery.

Workflow identity

The product identity is the workflow UUID. The name is a unique mutable label and currently also becomes the Airflow DAG ID and route identifier. Runtime requests may use the name only as a lookup key; the backend must resolve it to the UUID before authorization and must use the stored Airflow identity after that resolution.

Renaming requires an explicit lifecycle. Publishing the new name without removing or tombstoning the previous DAG can leave an orphaned runnable DAG. The UI must either make the name immutable or show rename as a coordinated operation covering Git artifacts, Airflow discovery, schedules, runs, links, monitoring, audit, and authorization.

Target authorization boundary

Workflow-backend source defines eight workflow permissions: view, edit, trigger, pause, clear execution, mark execution state, delete, and manage access. Because OpenFGA is not activated and runtime traffic bypasses those routes, these fields are not current deployed enforcement. In the target, the UI gates presentation from those capabilities while workflow-backend enforces them server-side.

All runtime reads and mutations must follow this sequence:

1. Validate the browser token and resolve the canonical principal.
2. Resolve the supplied managed workflow name to its workflow UUID.
3. Check the operation-specific OpenFGA relation.
4. Fail closed without contacting Airflow on denial or dependency failure.
5. Execute one explicit Airflow operation using backend credentials.
6. Return a product error category and correlation identity.

Generic Airflow proxying, UUID pin reads without authorization, unmatched name-based deletion, notification channel administration, and DAG-keyed resource monitoring do not currently satisfy this boundary.

Runtime and synchronization lifecycle

Read-only runtime data can remain visible while a new revision is applying, but it must identify the revision and observation time. Runtime mutations are disabled until state and permissions are known and the requested revision is Ready. Unknown status must fail closed.

The current UI polls managed detail every five seconds and several child views also poll Airflow data. Polling must pause when the page is hidden, stop for terminal state where freshness is not requested, back off on failure, coalesce duplicate reads, and display the observation timestamp. WebSocket or event delivery is optional; bounded conditional polling is sufficient.

Run identity is dag_run_id; execution, logical, start, and end timestamps are attributes and must never be used as UI keys. Clearing a run resets task instances inside that identity and may change state, tries, and timestamps; it does not itself create another run. Runtime responses should be normalized by dag_run_id, preserve one ordered column per unique run, and invalidate or refresh every clear-sensitive field after a mutation.

Run filtering belongs to the workflow runtime contract rather than a leaked set of generic Airflow parameters. The contract must define state-list semantics, which timestamp is filtered, timezone interpretation, pagination, maximum page size, totals, and observation freshness. The current v1 Airflow run-list path supports date bounds and pagination but not the state parameter emitted by the UI.

Run-now partial failure

Run now currently has two implementations. The list uses workflow-backend to unpause and therefore persists workflow_paused; detail patches Airflow directly and changes only the frontend's legacy is_paused field. Because the detail toggle prefers backend-owned workflow_paused, it can show paused while Airflow is already unpaused. Normal Ready-detail reads do not repair this drift; only the internal reconciliation operation can import the Airflow value, and no scheduled platform-stacks caller was identified.

Both implementations send the trigger separately. If unpause succeeds but trigger fails, the workflow remains unpaused and its schedule can resume despite no manual run being created. The product must either require explicit unpause, restore the prior state after failure, or offer Run once while paused with server-owned semantics. See REL-WF-004.

Notifications

Notifications are exposed but not implemented as a complete product capability. The source boundary is incomplete:

• Workflow-backend contains unscoped channel models and CRUD routes, validates workflow assignments, synchronizes Teams connection secrets only through its AWS path, and emits notification callbacks into generated DAG definitions.
• BFF proxies the channel routes.
• Oqullus exposes assignment controls in the multi-task form and sends workflow_config.notifications, but has no channel management, test, or delivery-status journey.
• Channel responses return config unchanged, including a Teams webhook URL, and the backend routes do not enforce project/workspace channel scope or a channel capability.
• The assessed Alibaba stack enables an AliCloud notification-sync flag that the reviewed backend does not define or consume; its synchronization class is AWS Secrets Manager-specific.

Hide or server-capability-gate the existing controls. Re-exposure requires channel scope, authorization, write-only secret handling, recoverable secret/database synchronization for the active environment, runtime provider health, delivery evidence, audit, and failure feedback. See PROD-WF-004.

Resource monitoring

Resource Monitoring is a nested Workflow subfeature with its own collection, storage, read, authorization, deployment, and recovery boundaries. The parent control plane must resolve the managed workflow before monitoring data is read; the current source and KCL path has not yet converged on that integration.

See the Resource Monitoring architecture and ARCH-WF-002 for the complete topology, alternatives, and deployment gates.

Spark recommendation

Spark Recommendation is a separate advisory Workflow subfeature. It consumes completed Spark application evidence and optional Resource Monitoring data, but must bind both to one exact Workflow run, task attempt, and Spark application. Submission-time sizing belongs to the saved Workflow task and the next SparkApplication; it cannot be applied to a running job through generic spark.conf.set(...) Python.

See the Spark Recommendation architecture for applicability, validation, Delta/Trino storage, retention, cost, access, and effectiveness boundaries.

Airflow metadata lifecycle

Airflow runtime state depends on a single-replica PostgreSQL database with a 20 GiB volume in the assessed stack. No source-defined metadata cleanup, capacity alert, tested backup/restore, or upgrade rehearsal was identified. Exhaustion is a platform outage condition, not merely loss of old history: schedulers and tasks need the database to create and update runs, task instances, jobs, and heartbeats.

The target lifecycle combines forecasted capacity alerts, controlled expansion, bounded operational retention, supported Airflow cleanup, PostgreSQL maintenance, and tested recovery. Cleanup must start while enough free space remains for its archive behavior. Airflow schema migrations are rehearsed on a representative database copy; the production procedure pauses new schedules, defines running-task handling, stops components for migration, and uses a measured timeout and rollback plan. See OPS-WF-001.

Spark log boundary

Airflow currently follows Spark application output, while Oqullus polls every five seconds without sending Airflow's continuation token. In Airflow 2.10 and 2.11, the file-log handler reads and interleaves the complete available source before applying a previous log_pos; without the token, every Oqullus request returns the complete growing log. A 1 GiB streaming-task log can therefore be read, rebuilt, proxied, transferred, and rendered again on each poll. Remote Airflow object logging uploads on handler close and is not an independent live stream for an indefinitely active task.

The target keeps only submission identity, state transitions, and bounded diagnostics in Airflow. Spark driver and executor logs flow directly to Loki through Alloy and are read with exact immutable correlation labels, a bounded recent tail, and timestamp or cursor continuation. Spark event logs and History Server serve completed execution analysis. This also allows streaming applications to outlive a finite Airflow deployment operation. See PERF-WF-002.

Concurrency and managed capacity

The user-facing contract is workflow intent: maximum overlapping runs, explicit queue behavior, and—only if needed—maximum concurrent tasks. Airflow scheduler replicas, global parallelism, worker counts, pools, task-pod sizing, Karpenter nodes, and database sizing remain platform-managed controls.

The assessed deployment already uses KubernetesExecutor and a Karpenter worker node pool, so runnable tasks create pods and pending pods drive node supply. The missing boundary is workspace admission, quota, overload feedback, scheduler availability, and measured scale limits. Capacity should respond to runnable work and scheduler/database health rather than registered-user count. Streaming jobs belong to a managed long-lived deployment lifecycle; Workflow performs finite deploy, update, stop, and reconcile operations. See ARCH-WF-003.

Deployment and observability boundary

Required correlation spans workflow request, workflow UUID and revision, Git publication attempt, commit SHA, Airflow DAG observation, run ID, Spark application, authorization decision, and user-visible error. Logs must never contain browser tokens, OpenFGA credentials, notification webhooks, XCom secret values, or raw secret-store payloads.

See Workflow components for implementation ownership and pinned source references.