Workflow Resource Monitoring architecture

Architecture status

Label	Meaning in this assessment
Implemented	Source exists and can be built independently
KCL-defined	Current platform-stacks source renders the component or setting
Deployed	Active behavior verified in gj76anadpsc, workspace w-t5e2kcxjd
Target	Required supported architecture; not a statement of current behavior

The dedicated backend is implemented, but its API and collector are not KCL-defined. The current KCL path separately gives the Spark Recommendation job the Workflow resource table. Live deployment and data evidence remain pending.

Current source and KCL topology

The implementation is split across repositories:

• sparqd/oqullus renders current and historical metrics.
• sparqd/bff-api proxies latest/history requests by Airflow DAG ID.
• sparqd/workflow-resource-monitoring contains a standalone Prometheus/Kubernetes collector, Delta writer, and FastAPI reader.
• sparqd/spark-recommendation contains the job currently configured by KCL to receive the same Workflow resource table.
• sparqd/platform-stacks deploys Spark Recommendation but does not declare the dedicated monitoring API or BFF URL.

Implemented dedicated collector

The standalone collector performs this sequence:

1. Start a Spark session.
2. query Prometheus for namespace-level pod CPU, memory, and PVC metrics;
3. list RUNNING SparkApplication resources and running executor pods;
4. match application names and DAG/task labels to metric series;
5. calculate allocation, usage, wastage, and one provisioning status;
6. merge one row per snapshot date, DAG, and task into a Delta table; and
7. retain the latest configured number of snapshots per DAG/task.

Important data-quality boundaries:

• The collector observes only applications in RUNNING state at collection time. A scheduled snapshot can miss short or completed workloads.
• Prometheus request and decoding errors return empty maps. Downstream logic turns missing usage into zero and can classify a real workload as Over provisioned instead of marking the observation invalid.
• Pod/PVC matching uses naming conventions rather than canonical Spark run and volume identities.
• The merge key includes snapshot_date, so repeated collections on one day update one daily row and increment its version rather than preserving each observation.

These are source findings for future triage; the collector is not the current KCL-defined producer.

Granularity and canonical identity

The current entry point is workflow-scoped, but the measured resource consumer is a specific Spark application launched by a specific task attempt. The current schema stops at dag_id and task_id, and its daily merge key can overwrite multiple executions of the same task.

The supported hierarchy is:

Managed Workflow and revision
  → Workflow run
    → Task attempt
      → Spark application
        → Immutable resource observations

Workflow totals and daily trends are derived projections, not canonical stored identity. Non-Spark tasks remain visible in the task hierarchy but are marked not applicable. Recommendation output must reference the same workflow revision, run, task attempt, and Spark application as its input observations.

See DATA-WF-002.

Collection cadence and measurement semantics

The standalone repository documents a 15-minute CronJob, but current KCL does not deploy it. Each invocation performs Prometheus instant queries and current Kubernetes reads. CPU uses a preceding five-minute rate, whereas memory, disk, and executor count are current values. The sequential reads receive one shared snapshot timestamp and do not preserve intervening peaks.

The target collector is driven by Spark-application lifecycle and aggregates timestamped Prometheus series across the exact task-execution window. It stores resource-specific average, percentile, peak, allocation-over-time, and completeness semantics. Periodic collection remains only for reconciliation of missed start/completion events.

See DATA-WF-003.

Implemented dedicated API

The FastAPI service exposes:

GET /api/v1/workflow-resource-monitoring/{dag_id}
GET /api/v1/workflow-resource-monitoring/{dag_id}/history
GET /health

Production mode opens a Trino connection per request. Latest uses a window function per DAG/task; history returns every matching row ordered by timestamp. Local sample mode reads a JSON file when USE_SAMPLE_DATA=true.

Current boundaries:

• The API has no caller-authentication middleware.
• It resolves a caller-provided DAG ID, not a managed Workflow UUID.
• History has no server-side time range, cursor, page size, or result limit.
• /health proves only that the process responds; it does not verify Trino, credentials, the table, or observation freshness.
• The source uses Basic authentication for Trino and has no Alibaba JWT path.
• The default table name uses oqullus_system, while current KCL producers use cogrion_system; deployment must set one explicit fully qualified table.
• Backend sample mode is invisible to Oqullus: a successful sample response can look like production data because the browser's fallback flag is not set.

Browser presentation and fallback

Oqullus opens Resource Monitoring from the Workflow row actions. It requests latest and history independently.

• A structured API 404 becomes No monitoring data.
• Other latest failures create deterministic CPU, memory, and disk rows keyed to real task names and show Showing sample data — the resource-monitoring service is not connected yet.
• Missing or insufficient history generates a synthetic trend and shows a smaller sample trend marker.

The warning does not make the values safe: generated rows still drive the real utilization cards and Right sized, Under provisioned, or Over provisioned classification.

Security boundary

Current BFF routes validate that a browser JWT exists, but they do not resolve the DAG ID through workflow-backend or require workflow view permission. The dedicated backend accepts unauthenticated internal calls. This is inadequate as an internal service boundary today and would bypass project-specific grants after OpenFGA rollout.

The target read path is:

Oqullus
  → BFF
  → workflow-backend resolves Workflow UUID and view permission
  → authenticated monitoring API request using stored Airflow identity
  → bounded Trino query
  → provenance-bearing response

The browser must not choose an arbitrary infrastructure identifier as the authorization scope.

Target deployment

KCL must declare:

• one producer and its schedule, schema version, merge semantics, and retention;
• an API Deployment, ClusterIP Service, pinned image, and resource limits;
• Alibaba-compatible Trino identity and the explicit table name;
• authenticated service-to-service access;
• BFF RESOURCE_MONITORING_URL and timeout;
• dependency-aware readiness plus liveness;
• a server-owned feature capability consumed by Oqullus; and
• dashboards and alerts for collection age, rejected rows, query latency, response size, dependency failures, and generated-data attempts.

Ownership alternatives

Incremental deployment

Deploy the dedicated API only, retain Spark Recommendation as the sole writer, and wire BFF/KCL. This is the smallest route to real reads, but monitoring data ownership remains coupled to recommendation execution.

Consolidation in Spark Recommendation

Add the bounded monitoring read contract to the already deployed recommendation API. This reduces components but combines monitoring availability, authorization, and lifecycle with a separate product capability.

Dedicated monitoring subsystem

Deploy both the dedicated collector and API, migrate table ownership away from Spark Recommendation, then remove the old producer. This is the clearest long-term boundary but requires controlled schema, history, and retention migration.

Prometheus with embedded Grafana

Replace the custom collector, read API, and duplicated chart implementation for the monitoring-only journey. Prometheus remains the time-series authority; recording rules derive stable per-application series, and Cogrion embeds an authenticated Grafana dashboard or panel. A thin Workflow-authorized facade or data-source boundary is still required for per-workflow viewers because dashboard variables are filters, not authorization. Durable completion summaries remain optional inputs to Spark Recommendation rather than a second raw-metrics store.

Grafana OSS can be used as an unmodified, separately deployed service under its current AGPL-3.0-only license, subject to the license terms. Preserve required notices and applicable source availability, inventory plugin licenses, and obtain legal/commercial review before modifying Grafana, white-labeling it, or using Enterprise/OEM features. This is a constraint on this alternative, not a separate Resource Monitoring finding. See Grafana's licensing file, relicensing Q&A, and embedding documentation.

The assessment recommends the incremental API-only path for immediate recovery, with an explicit later decision between consolidation and full separation. Running both collectors concurrently is excluded.

Performance and cost boundaries

• History must accept a bounded range and page/cursor; browser-side windowing cannot protect Trino or network cost.
• Latest should avoid scanning and ranking all historical rows when a compacted latest projection can be maintained.
• Collection cadence must match workload duration. A daily point-in-time scrape cannot represent short workflows.
• Retention must be enforced by one producer and documented in time as well as row-count terms.
• API and Trino timeouts, cancellation, concurrency, and maximum response size must be explicit.

Recovery contract

No data, stale data, collector failure, authorization denial, Trino failure, and API unavailability are distinct states. Recovery must preserve the last known real observation with its timestamp, expose retry where safe, and never replace a failed dependency with invented production metrics.

See live evidence and the finding register.