Workflow live evidence

Evidence scope

Source was refreshed from the latest main references on 3 September 2026:

Component	Commit
Oqullus	8c9f240410b2
BFF API	9b2fd45a4da1
workflow-backend	d3d6a3ad7313

Runtime verification targets realm gj76anadpsc, workspace w-t5e2kcxjd. No finding is closed by source inspection alone. Missing deployment evidence is recorded explicitly without erasing source-backed behavior.

Source evidence summary

Observation	Evidence state
Workflow list, create, edit, clone, delete, explicit pause/unpause, and pin use the BFF Workflow facade	Source-verified
Detail Run now bypasses the managed unpause command, patches Airflow directly, and leaves backend workflow_paused unchanged	Source-verified; user-observed UI mismatch, network capture pending
UI runtime detail uses BFF Airflow routes for runs, tasks, logs, XCom, source, graph, grid, clear, mark state, and trigger	Source-verified
Workflow backend implements workflow-scoped runtime routes and OpenFGA checks	Source-verified; implemented but inactive
BFF Workflow facade does not expose those runtime routes	Source-verified
UI receives but does not use per-workflow permission flags	Source-verified
OpenFGA is not activated in the assessed deployment	Deployment status supplied for this review; independent configuration capture pending
UI supports only Spark Kubernetes tasks and does not request the task-type catalog or semantic validation warnings	Source-verified
UI intentionally uses the backward-compatible single-project path, supplies no project, and relies on backend core default	Source-verified; product label still absent
Run as User is displayed/submitted by Oqullus and persisted by workflow-backend, but reviewed DAG-generation/runtime paths do not apply it as execution identity	Source-verified; incomplete feature incorrectly exposed in UI
Oqullus multi-task create/edit exposes Add Notification, channel selection, four trigger events, and email recipients; its workflow service serializes assignments	Source-verified; incomplete feature incorrectly exposed
Workflow-backend channel rows and queries have no project/workspace scope or capability enforcement; list/delete do not resolve identity, and read responses return config	Source-verified security and lifecycle gap
Workflow-backend implements AWS Secrets Manager synchronization and DAG compilation; BFF proxies channel routes	Source-verified partial implementation
The Alibaba stack enables NOTIFICATION_SYNC_ALICLOUD_KMS_ENABLED, but workflow-backend has no matching setting or implementation and checks only the AWS flag	Source-verified deployment mismatch; runtime delivery unverified
Resource-monitoring BFF validates authentication only; its backend does not authenticate callers	Source-verified
sparqd/workflow-resource-monitoring implements a collector, Delta writer, and FastAPI latest/history reader	Source-verified at 8665dd6efe13
Current KCL does not deploy that API or configure RESOURCE_MONITORING_URL; it separately configures the recommendation job with the same Workflow resource table	Source-verified at platform-stacks a71390d12463; live request pending
Monitoring service failure can trigger generated sample metrics and history; the modal warns Showing sample data — the resource-monitoring service is not connected yet, but still renders utilization and right-sizing conclusions	Source-verified
Create/update publishes Git before database commit; delete removes Git before database commit	Source-verified
Form state has no draft persistence or leave warning	Source-verified
Run filters are forwarded, but state uses an unsupported comma-separated state parameter and 365/All can exceed Airflow's page limit	Source-verified; deployment sign-in blocked
Run columns use mutable start_date identity and their change fingerprint omits clear-sensitive timing and try fields	Source-verified; reported extra column not independently reproduced
Selected-run response fields are omitted and refresh failure has no visible stale/error state	Source-verified
Airflow uses one 20 GiB PostgreSQL replica with its exporter disabled; no metadata cleanup, capacity alert, backup/restore proof, or upgrade rehearsal was identified in the assessed stack	Source-verified; deployed capacity and recovery evidence pending
Spark tasks follow application logs; Airflow 2.10/2.11 reads and combines complete available file-log sources before offset slicing, while Oqullus omits the continuation token and polls the complete growing response every five seconds	Source-verified; ten-minute buffering user-observed; representative deployed measurements pending
Workflow exposes Max Active Runs; Airflow uses KubernetesExecutor with global parallelism 128 and one scheduler, while Karpenter scales worker nodes from pod demand	Source-verified; workspace admission, scheduler HA, and scale evidence pending

Reproduction scenarios

WF-S001 — Canonical route inventory

1. Sign in to realm gj76anadpsc and select workspace w-t5e2kcxjd.
2. Open Workflow list and one managed workflow with browser Network recording.
3. Visit Graph, Runs, task Details, Log, XCom, Code, Audit Log, Task Duration, and Resources.
4. Trigger only approved read operations and record whether each request uses /workflow, /airflow, or /resource-monitoring.
5. Match every request to the canonical workflow UUID and managed name.

Expected target: all managed operations enter through the Workflow facade and are resolved and authorized by workflow-backend before Airflow or monitoring.

WF-S002 — Create, save, and synchronization

1. Create a disposable one-task workflow with no schedule.
2. Record the create response, workflow UUID, revision, sync status, Git commit, Airflow discovery time, and final Ready transition.
3. Repeat with a Git publication failure, Airflow import error, Airflow outage, and discovery timeout.
4. Confirm the UI never labels a pending revision as successfully applied and shows an actionable error and correlation identity.
5. Retry the same requested revision and confirm no duplicate workflow or DAG.

WF-S003 — Edit and rename lifecycle

1. Edit a disposable workflow without changing its name and verify one revision moves from Applying changes to Ready.
2. Change its name and record workflow UUID, old/new Git files, old/new Airflow DAGs, schedules, runs, monitoring identity, links, and authorization.
3. Confirm the previous DAG cannot remain runnable or visible as unmanaged.
4. Reload during synchronization and verify the same revision and status return.

WF-S004 — Task types and semantic validation

1. Compare the backend task-type catalog with every type selectable in the UI.
2. Author a branching workflow with a convergence task using a trigger rule that can cause skip propagation.
3. Confirm the UI requests backend semantic validation and displays its warning before Save.
4. Test duplicate task IDs, missing dependencies, cycles, unsupported images, invalid resources, and a workflow-trigger target.
5. Confirm blocking errors prevent persistence and warnings remain reviewable.

WF-S005 — Single-project compatibility and future project rollout

Current deployed mode:

1. Open Workflow creation in realm gj76anadpsc, workspace w-t5e2kcxjd.
2. Confirm the frontend shows no project selector and omits project_id.
3. Record the created workflow response and verify workflow-backend resolves the omitted project to core.
4. Confirm the UI does not claim that another project was selected or that OpenFGA-backed per-workflow capabilities are active.

Future rollout acceptance, not a current-production test:

1. Activate OpenFGA and the managed Workflow runtime facade in a controlled environment.
2. Use actors with create-in-project, view, edit, trigger, pause, clear, mark-state, delete, and manage-access combinations.
3. Verify the selector lists only projects where creation is allowed and every action matches the returned capability.
4. Invoke every denied action directly and prove denial occurs before Airflow, Git, monitoring, or secret-store access.
5. Verify no implicit core fallback occurs after a user explicitly selects a project.

WF-S006 — Runtime read authorization

Current deployed mode:

1. Use two representative authenticated roles in the single configured project.
2. Record which runtime, log, XCom, source, graph, grid, event-log, and monitoring routes each legacy role can reach.
3. Confirm documentation makes no claim of project- or workflow-scoped OpenFGA enforcement.

Future rollout acceptance, not a current-production test:

1. Activate OpenFGA and choose workflows A and B with disjoint viewer access.
2. As the A-only actor, request B's runtime resources using guessed names and IDs.
3. Verify every request returns 403 or 404 without contacting Airflow or the monitoring backend.
4. Repeat after caches are warm under B's allowed actor.

WF-S007 — Runtime mutation authorization and audit

Current deployed mode:

1. For each supported legacy role, test Trigger, Pause, Unpause, Clear run, Clear task, Mark run, and Mark task on an approved disposable workflow.
2. Record which operations reach Airflow and describe the result as legacy-role behavior, not workflow-scoped OpenFGA enforcement.

Future rollout acceptance:

1. Activate OpenFGA and test each operation-specific capability.
2. Confirm allowed operations reach Airflow once and denied operations never do.
3. Verify confirmation copy states affected runs, tasks, and downstream scope.
4. Correlate principal, workflow UUID, DAG name, run/task, old/new state, authorization decision, Airflow request, and result in audit evidence.

WF-S008 — Synchronization failure and recovery

1. Produce an approved Airflow import error for a disposable revision.
2. Confirm Needs attention includes a safe message and operator correlation ID.
3. Reload and verify the failed revision persists.
4. Exercise corrected Edit, explicit Retry, and Delete paths.
5. Confirm Retry republishes the same intended revision idempotently and Ready appears only after that exact revision is observed.

WF-S009 — Notification exposure and readiness

1. Open multi-task workflow create or edit and select + Add Notification.
2. Confirm the UI requests /workflow/api/v1/workflows/notification-channels and offers Channel plus Failure, Success, Retry, and Execute events.
3. With no channels, confirm the selector is a dead end and no create/manage action is available. With an approved disposable channel, select it, add an email recipient where applicable, save, reopen, and inspect the submitted and returned workflow_config.notifications.
4. As users from two projects, verify list, assign, update, and delete enforce the intended channel scope. Confirm no read response contains a webhook, password, token, or other secret. Current source does not satisfy this gate.
5. In realm gj76anadpsc, workspace w-t5e2kcxjd, verify channel creation provisions the required Airflow connection through the active Alibaba secret backend. Current source consumes only the AWS synchronization flag.
6. Verify generated callbacks deliver email and Teams notifications for execute, retry, success, and failure events, and surface an auditable failure when SMTP, Teams, secret-store, callback, or provider execution fails.
7. Until all gates pass, verify the notification controls are hidden through a server-owned deployment capability rather than a frontend-only assumption.

Expected result: incomplete deployments expose no notification controls. An enabled deployment provides a scoped channel lifecycle, write-only secrets, assignment persistence, observable delivery, and recovery for every advertised event. See PROD-WF-004.

WF-S010 — Resource monitoring trust and access

The complete scenario now lives in the Resource Monitoring evidence page so collection, API, deployment, and authorization evidence stay together.

WF-S011 — Pin authorization

Current deployed mode:

1. Verify Pin and Unpin operate within the documented single-project legacy role boundary.
2. Confirm no UI or documentation claims per-workflow OpenFGA protection.

Future rollout acceptance:

1. Activate OpenFGA; as an actor denied workflow B, obtain or guess B's UUID.
2. Call Pin and Unpin for B directly.
3. Confirm the request is denied and returns no workflow definition, project, schedule, creator, or synchronization data.
4. Warm the pin state as an allowed actor and repeat the denied call.

WF-S012 — Delete-by-name authorization

1. Use a disposable unmanaged DAG name that has no workflow database record.
2. As a non-admin actor, request managed deletion by that name.
3. Confirm the service does not delete Git artifacts merely because lookup returned no record.
4. Verify only an explicit, separately authorized operator cleanup procedure can remove orphan artifacts.

WF-S013 — Polling, cache, and request boundaries

1. Open list and detail separately and record every request for sixty seconds.
2. Repeat with auto-refresh on/off, foreground/background tab, Ready, syncing, failed, Airflow outage, and ten concurrent browser sessions.
3. Record BFF and backend cache hits, TTL, principal scope, observation age, Airflow calls, response size, and browser memory.
4. Verify polling backs off on error, stops when hidden, and does not duplicate the same runtime observation through both Workflow and Airflow paths.

WF-S014 — Run-now pause consistency and partial failure

1. Pause a disposable scheduled workflow.
2. From detail, select Run now and record the direct Airflow PATCH, trigger request, Airflow is_paused, backend workflow_paused, and visible toggle.
3. Wait through at least two five-second managed-detail polls, then reload and confirm whether the mismatch persists.
4. Repeat from the list and verify it instead calls the managed workflow unpause endpoint and persists the response.
5. Force the trigger request to fail after unpause succeeds. Verify whether the workflow remains unpaused and whether a scheduled run can start.
6. Invoke the internal reconciliation endpoint through an approved operator path and verify it repairs backend state; separately determine whether a deployed scheduled caller exists.
7. Retry and confirm exactly one manual run exists, then repeat with concurrent schedule and pause changes.

WF-S015 — Unsaved draft persistence

1. Create or edit a workflow and change its schedule, resources, dependencies, and several tasks without saving.
2. Navigate to another page, close the modal, reload, expire the session, and open another tab.
3. Verify a dirty warning or durable draft protects the changes.
4. Save, reload, and confirm only the accepted backend revision is restored.

WF-S016 — Run-as implementation readiness

There is no supported Run as User execution journey to verify. Current evidence ends at frontend submission and workflow-backend persistence. While incomplete, the field must be absent from create, edit, clone, and detail surfaces. See PROD-WF-005.

Immediate containment:

1. Hide Run as User on every Workflow surface without changing the effective platform runtime identity.
2. Confirm create, edit, clone, schedule, and manual run continue to work through a backward-compatible backend default.

Before product rollout:

1. Decide whether workflows use one platform identity, an actor-derived identity, or an explicitly selectable service identity.
2. Derive or authorize that identity server-side; do not trust a submitted string as impersonation authority.
3. Propagate it through generated DAGs, Airflow tasks, Spark, storage, Trino, and audit.
4. Run create, edit, clone, scheduled, and manual executions and prove the displayed, stored, executed, and audited principals agree.
5. Verify denied identity selection fails before publication or execution.
6. If no runtime identity choice is required, remove the field from the API and persistence model as a separate compatibility migration.

WF-S017 — Git and database fault reconciliation

1. In a controlled environment, fail database commit after a successful create publication; verify orphan detection and cleanup.
2. Fail database commit after successful update publication; verify the canonical requested revision converges without silent rollback.
3. Fail database commit after successful artifact deletion; verify restoration or completion from durable intent.
4. Repeat each operation to prove idempotency and record recovery telemetry.

WF-S018 — Run filter contract

1. Sign in to realm gj76anadpsc, workspace w-t5e2kcxjd, and open a workflow with queued, running, successful, and failed runs spanning two dates.
2. Record the unfiltered dagRuns request, response, unique run IDs, and total.
3. Select each single state and at least two states. Confirm the request uses a documented parameter and that every returned row satisfies OR semantics.
4. Select and clear Base date around a known logical-date boundary. Record the browser timezone, serialized UTC bound, and included IDs.
5. Select 25, 50, 100, 365, and All. Compare the label, request limit, response size, total_entries, and configured Airflow maximum.
6. Confirm invalid or unsupported filters show an error rather than an apparently unfiltered success.

WF-S019 — Clear and grid reconciliation

1. Use a disposable paused workflow with exactly two known dag_run_id values and no active schedule during the test.
2. Capture the dagRuns response and the DOM run-column count and identity.
3. Select the older run and clear existing tasks. Record the mutation response, event log, every subsequent dagRuns response, task tries, and rendered columns through completion.
4. Confirm clear preserves the selected dag_run_id, updates task try/timing data, and never renders more than one column for each unique response ID.
5. If a third column appears, distinguish a true third dag_run_id caused by a schedule/trigger from a DOM-only duplicate. Reload and compare again.
6. Repeat with two runs that have absent or equal start dates and with a fast rerun whose sampled state is the same before and after clear.

WF-S020 — Selected-run detail completeness

1. Select scheduled and manual runs with known configuration, data intervals, notes, and start/end values.
2. Compare the run-detail response with every labelled field in Details.
3. Confirm logical date, data interval, trigger origin, safe configuration, note, run type, state, and actual timing are visible and correctly distinguished.
4. Deny the request, return 404, disconnect Airflow, and fail one background refresh after a successful read.
5. Confirm the panel distinguishes no selection, not found, denied, unavailable, and stale last-good data and offers an appropriate retry.

WF-S021 — Resource monitoring KCL wiring

The complete scenario now lives in the Resource Monitoring evidence page. Its associated deployment finding is ARCH-WF-002.

WF-S022 — Airflow metadata capacity, retention, and upgrade

Use an isolated database copy and approved non-production Airflow deployment; do not induce storage exhaustion or run destructive cleanup against the production metadata database.

1. Record database volume size, used and free bytes, daily and weekly growth, largest Airflow metadata tables, connection use, and current alert coverage.
2. Forecast exhaustion under representative workflow, run, task, XCom, log metadata, and schedule frequency. Confirm alerts fire with enough lead time to expand or clean safely.
3. Run airflow db clean in dry-run mode for the proposed retention boundary. Estimate rows, archive space, transaction duration, and PostgreSQL maintenance needs before changing data.
4. Take a recoverable backup or snapshot. On the isolated copy, run the approved cleanup path, export required archives, drop archived tables when permitted, and perform planned PostgreSQL maintenance. Confirm retained Airflow views and active runs remain correct.
5. Restore the backup into another isolated environment and verify scheduler, webserver, workflow history, and active-state recovery.
6. Clone a production-sized database and run the target Airflow schema migration. Record free-space peak, lock and component downtime, migration duration, deployment timeout, rollback, and smoke-test results.
7. Verify the production runbook pauses new schedules, defines running-task handling, stops components during migration, and uses the rehearsed timeout.

Expected result: exhaustion is detected before writes fail; cleanup and maintenance preserve required operational state; restore is proven; and the upgrade has a measured downtime and rollback boundary. See OPS-WF-001.

WF-S023 — Long-running Spark log transport

1. In an approved environment, run a Spark task that emits a stable volume of uniquely numbered log lines for at least one hour and grows its Airflow task log through 100 MiB, 500 MiB, and 1 GiB checkpoints.
2. Keep Workflow Log open. Every five seconds, record request and response bytes, returned first and last line, cumulative network transfer, browser memory, render latency, Airflow task-pod state, and Spark application state.
3. Capture response metadata and the returned continuation token. Confirm that Oqullus does not send that token on the next request and that each response contains the complete currently available log. Close, navigate away, and hide the tab separately; verify when requests stop.
4. While the application is active, compare the Airflow local task log, remote OSS/S3 object, Loki driver and executor streams, and Spark event log. Record collection delay, correlation labels, gaps, and retention.
5. In a disposable run, evict the Airflow task pod while Spark continues. Verify whether live and completed logs remain available and whether the application is incorrectly terminated or orphaned.
6. Repeat with a diagnostic client that sends Airflow's continuation token. Confirm response bytes contain only appended content, while server-side duration, I/O, CPU, and memory still reflect reading and reconstructing the full source before offset slicing.
7. Validate the target Loki path with a bounded recent tail, timestamp or cursor continuation, older-page retrieval, paused auto-refresh, exact task-attempt isolation, truncation and gap messages, and authorization denial.
8. Verify Alloy replaces Promtail before relying on the central-log path as a supported design.

Expected result: refresh advances with the returned continuation identity and transfers bounded new content, the browser remains bounded, exact application logs survive Airflow task-pod loss, and Airflow contains only finite orchestration diagnostics. See PERF-WF-002.

WF-S024 — Workflow concurrency and managed capacity

Current-state reproduction:

1. Create a disposable workflow with one task that runs for ten minutes and set Max Active Runs to 1. Trigger three runs close together.
2. Confirm only one DAG run becomes active and record how the other two runs are labelled, queued, and explained in the UI.
3. In an approved performance environment, start ten synthetic workflows with twenty two-minute runnable tasks each, creating 200 runnable tasks against the configured global parallelism of 128.
4. Record scheduler replicas, scheduling-loop duration, DAG-parse duration, queued-task age, metadata-database latency and connections, task-pod startup, pending-pod age, Karpenter node provisioning, quota denials, and recovery after the burst.
5. Verify the Workflow UI exposes only Max Active Runs and does not expose scheduler replicas, worker counts, pool names, node counts, or database sizing.

Future acceptance validation requires workspace admission, scheduler high-availability, and managed streaming lifecycle capabilities that are not available in the assessed product path:

1. Apply a documented per-workspace runtime limit in Airflow/platform policy and Kubernetes, then repeat the 200-task burst from two workspaces. Prove one workspace cannot exhaust the other's admitted capacity.
2. Fail one of two schedulers and confirm scheduling continues without duplicate task execution. Repeat an upgrade using the database procedure from WF-S022.
3. Deploy a disposable streaming Spark application through the managed service lifecycle. Confirm the finite Airflow deploy operation finishes while the application remains observable and can later be updated, stopped, and reconciled idempotently.

Expected result: user intent is predictable, platform capacity scales from work, workspace admission prevents noisy-neighbor exhaustion, and streaming application lifetime is independent of Airflow task lifetime. See ARCH-WF-003.

Evidence still required

• Signed-in route and role evidence for WF-S001, WF-S005, WF-S006, and WF-S007.
• Create, update, import-error, timeout, reload, retry, and delete evidence for WF-S002, WF-S003, and WF-S008.
• Notification controls are currently exposed by source. Deployed assignment, redaction, Alibaba connection synchronization, delivery, failure, and recovery evidence for WF-S009 remains pending; see PROD-WF-004.
• Monitoring authorization, no-data, unavailable, and scale evidence remains open in the Resource Monitoring evidence register.
• Approved destructive fault-injection environment for WF-S012 and WF-S017.
• Draft and partial-trigger evidence for WF-S014 and WF-S015.
• Run as User remains incomplete; WF-S016 records future implementation prerequisites rather than missing evidence for a supported journey. See PROD-WF-005.
• Signed-in deployed filter evidence for WF-S018; the attempted Chromium run reached the requested realm sign-in but had no authenticated session. The available local E2E account targets another realm/workspace and was excluded.
• A disposable paused workflow for the clear mutation and canonical network/DOM capture in WF-S019.
• Representative run metadata and dependency-failure evidence for WF-S020.
• Rendered and deployed Resource Monitoring API/BFF wiring, single-producer ownership, Alibaba Trino authentication, and failure recovery remain open in WF-S021.
• Database growth, alert thresholds, cleanup headroom, backup/restore proof, and a production-sized migration rehearsal for WF-S022.
• Long-run response-byte and browser-memory measurements, pod-loss behavior, Loki correlation/access proof, and Alloy migration evidence for WF-S023.
• Workspace admission decisions, burst and sustained-load results, scheduler availability evidence, and a managed streaming lifecycle for WF-S024.