Workflow Resource Monitoring evidence

Evidence summary

Observation	Evidence state
Oqullus exposes Resources and independently requests latest and history	Source-verified
Non-404 latest failure generates task-shaped sample metrics and provisioning conclusions	Source-verified
Missing or insufficient history generates a sample trend	Source-verified
BFF validates browser JWT but performs no workflow authorization	Source-verified
Dedicated FastAPI backend has no caller authentication	Source-verified at 8665dd6efe13
Dedicated API and collector are implemented	Source-verified at 8665dd6efe13
Current KCL does not deploy the dedicated API or configure its BFF URL	Source-verified at platform-stacks a71390d12463
Current KCL separately gives Spark Recommendation the Workflow resource table	Source-verified; live write pending
Dedicated history API is unbounded	Source-verified
Dedicated collector converts Prometheus failures to empty maps used as zero measurements	Source-verified; deployed impact not applicable until collector rollout
Persisted identity stops at snapshot date, DAG, and task; run, attempt, and Spark application are absent	Source-verified
Collector uses a documented 15-minute tick, mixed five-minute and instantaneous metrics, sequential reads, and RUNNING-only discovery	Source-verified; collector not KCL-deployed

WF-S010 — Resource monitoring trust and access

1. Compare current and history data across supported single-project legacy roles and record the deployed access boundary.
2. Warm data under an allowed role, then repeat with a role outside that boundary.
3. Stop or disconnect the backend and verify production shows Unavailable rather than generated utilization values.
4. Test a workflow with no runs and distinguish No data from dependency failure.
5. Measure response size and latency for 7, 30, 60, and 90 days and verify the service bounds the query rather than returning all history.
6. After OpenFGA activation, repeat with disjoint workflow grants and verify denial before monitoring access, including after caches are warm.

WF-S021 — Resource monitoring KCL wiring

1. Render the gj76anadpsc/w-t5e2kcxjd stack and locate the canonical producer, API Deployment/Service, BFF RESOURCE_MONITORING_URL, credentials, probes, and capability flag.
2. Confirm only one job writes cogrion_system.workflow_resource_monitoring and record its schema and retention owner.
3. Open Resources and capture latest/history requests. Confirm the BFF reaches the rendered internal service and does not use an empty base URL.
4. Verify the API authenticates to Alibaba Trino, reads the same table, and distinguishes no rows from dependency failure.
5. Stop the API and deny Trino access. Confirm readiness fails and Oqullus shows Unavailable without generated values.
6. Restore dependencies and verify current and bounded history recover without requiring a browser refresh.

RM-S001 — Collector measurement validity

1. Run the selected canonical producer against a short, a long, and a dynamically allocated Spark workflow.
2. Record the workflow/run/task identities, collection interval, Prometheus timestamps, executor count, and raw resource series.
3. Compare stored allocation, usage, waste, and status with those sources.
4. Fail CPU, memory, and disk queries independently. Confirm the collection is rejected or marked incomplete; it must not store zero usage and classify the workload as Over provisioned.
5. Run a workflow entirely between two collection intervals and confirm the design captures it or documents it as unsupported.

RM-S002 — Real, no-data, stale, unavailable, and demo states

1. Return a fresh real snapshot and confirm provenance plus observation time.
2. Return a structured 404 and confirm No data.
3. Return an old last-good snapshot and confirm Stale plus age.
4. Return 401, 403, 500, 502, timeout, invalid JSON, and connection refusal; confirm each produces no measurement or provisioning conclusion.
5. Enable any approved demo mode and confirm a persistent environment-level banner, non-production isolation, and response provenance.
6. Confirm backend sample mode cannot be mistaken for real data by the browser.

RM-S003 — History scale and cancellation

1. Populate multiple tasks up to and beyond the retention boundary.
2. Request each supported time range and page size.
3. Record Trino scanned bytes, execution time, API response size, BFF latency, browser memory, and rendered point count.
4. Cancel a slow request and verify cancellation propagates rather than leaving abandoned Trino work.
5. Attempt an unsupported or excessive range and confirm a bounded validation response.

RM-S004 — Identity and authorization

1. Create two workflows whose actors have disjoint view grants.
2. Request both by Workflow UUID and by guessed Airflow DAG ID.
3. Confirm the server resolves the managed workflow and denies the unauthorized request before querying monitoring data.
4. Verify the monitoring API accepts only authenticated internal callers.
5. Confirm logs and metrics contain safe workflow correlation without browser tokens, Trino credentials, or raw tenant data.

RM-S005 — Producer ownership and migration

1. Inventory every writer to the monitoring table and disable all but the selected canonical producer.
2. Record schema, partitioning, merge key, version, retention, and timestamp semantics.
3. If ownership moves, dual-read and compare during a bounded migration without concurrent writes to the same keys.
4. Roll back and forward, confirming no duplicate, lost, or reclassified observations.

RM-S006 — Task-execution granularity

1. Create a workflow with two Spark tasks and one non-Spark task.
2. Run it twice on the same date, then retry one Spark task so at least three task attempts and distinct Spark applications exist.
3. Confirm every observation contains managed Workflow UUID and revision, workflow run ID, task ID, attempt, Spark application ID, observation ID, timestamp, and source.
4. Verify no execution overwrites another and current/history reads can select each run and task attempt independently.
5. Confirm the workflow summary equals an aggregation of the selected run's task observations and the non-Spark task shows Not applicable.
6. Generate a recommendation and verify it references the exact observation set and task configuration rather than only the DAG/task name.
7. Rename or revise the workflow and confirm old observations remain attached to the original immutable workflow identity and revision.

RM-S007 — Collection window and coverage

1. Run Spark tasks lasting less than five minutes, between five and fifteen minutes, and longer than fifteen minutes; include dynamic executor scaling.
2. Capture their exact application start/end time, task attempt, Prometheus series, Kubernetes state transitions, and collector invocation times.
3. Verify every execution is discovered even when it starts and finishes between reconciliation ticks.
4. Recalculate CPU average/high percentile/peak, memory high percentile/peak, disk peak/final usage, and executor allocation over time from the source series; compare them with the finalized record.
5. Delay the Prometheus and Kubernetes calls independently and confirm each measurement retains its real window/timestamp instead of inheriting one misleading snapshot time.
6. Remove one metric series, fail one lifecycle event, and overlap a collector reconciliation run with its next schedule. Confirm the record is incomplete or recovered, never silently zero-filled or treated as complete.
7. Verify incomplete observations cannot produce provisioning or recommendation conclusions and that the UI explains the missing coverage.

Evidence still required

• Rendered and deployed KCL evidence for the API, BFF URL, identity, probes, and feature capability.
• Live latest/history responses from realm gj76anadpsc, workspace w-t5e2kcxjd.
• Role and OpenFGA denial evidence.
• Collector accuracy and partial-source-failure evidence.
• History scale, cancellation, retention, and recovery evidence.
• Confirmation of one canonical monitoring-table producer.
• Same-day multi-run, retry, workflow-revision, and recommendation-correlation evidence for RM-S006.
• Short-job, peak, mixed-window, delayed-read, missed-event, and overlapping-run evidence for RM-S007.

Missing evidence remains open; source presence is not deployment proof.