DATA-WF-003 — Replace periodic snapshots with execution-lifecycle metrics

Field	Value
Status	Open; source-verified, standalone collector not deployed by current KCL
Architecture state	Dedicated collector is designed as a periodic CronJob
Severity	High
Affected surface	Workflow Resources utilization, history, provisioning status, recommendations, and cost decisions
Components	sparqd/workflow-resource-monitoring, sparqd/spark-recommendation, sparqd/platform-stacks, Prometheus, Kubernetes, Spark Operator
Associated issue	Not opened

Where the problem appears

Run a Spark task that completes between two collector invocations, or compare the Resources values with the task's complete Prometheus time series. The standalone collector can return no row or a single point that does not represent the execution's peak or overall utilization.

Concrete observed case

The dedicated collector repository documents this CronJob schedule:

*/15 * * * *

That is an intended 15-minute tick, not a deployed cadence in the assessed environment: current platform-stacks KCL does not deploy the collector.

On each invocation, the collector creates one snapshot timestamp and then makes four sequential Prometheus instant queries and two Kubernetes API requests:

Measurement	Source semantics
CPU usage	Five-minute rate from container_cpu_usage_seconds_total[5m]
Memory usage	Current container_memory_working_set_bytes value
Disk capacity and usage	Current kubelet PVC gauge values
Allocation	Spark specification plus the currently running executor count
Workload discovery	SparkApplications and executor pods currently in RUNNING state

These values have different time semantics and are collected at different moments, but the resulting record receives the one earlier snapshot timestamp. The collector does not request a Prometheus range, retain intermediate peaks, or finalize a measurement when a Spark application completes.

The documented CronJob example also uses concurrencyPolicy: Forbid. If an invocation still runs when the next tick arrives, Kubernetes skips concurrent execution rather than creating another collector run.

Expected condition

Monitoring follows the lifecycle of each Spark task execution:

1. discover and identify the Spark application when the task starts;
2. retain or query timestamped Prometheus series throughout its lifetime;
3. finalize an immutable execution summary when the application completes;
4. store measurement windows and source timestamps rather than assigning one synthetic time to differently timed reads; and
5. reconcile periodically for missed events and incomplete finalization.

The summary defines resource-specific statistics instead of treating one point as the execution:

• CPU: time-weighted average, high percentile, and peak;
• memory: high percentile and peak working set;
• disk: peak and final usage with allocation history;
• executors: allocation over time so dynamic scaling is represented; and
• completeness: source coverage, gaps, and finalization state.

The periodic job remains a recovery/reconciliation mechanism, not the primary source of execution coverage.

Cause and evidence

• The collector invokes Prometheus /api/v1/query, not /api/v1/query_range.
• Only CPU has a range expression, and that range is five minutes; memory and disk are instantaneous gauges.
• One snapshot timestamp is created before the Prometheus and Kubernetes calls.
• Kubernetes discovery filters to RUNNING SparkApplications and running executor pods.
• The documented schedule is every 15 minutes with concurrent invocations forbidden.
• The writer subsequently merges one daily DAG/task row; the separate identity and overwrite defect is tracked in DATA-WF-002.

Why it matters

• A task shorter than the interval can be missed entirely.
• CPU, memory, disk, and allocation do not describe the same time window.
• Peaks between ticks are invisible.
• Sequential reads can describe different cluster states while appearing as one atomic observation.
• Long collector execution can create an additional coverage gap.
• Incomplete snapshots can drive incorrect provisioning and recommendation conclusions.

Existing controls and workarounds

Treat the standalone collector output as coarse point-in-time evidence only. For troubleshooting or sizing, inspect the complete Prometheus series, Spark application state, and Spark event logs for the exact task execution. Do not infer that a missing row means the task did not consume resources.

Alternatives considered

1. Recommended: lifecycle-driven discovery plus Prometheus range aggregation and completion finalization, with a periodic reconciliation job.
2. Use Prometheus recording rules for stable per-application series and read the bounded execution window at completion. This is compatible with the recommended design and can reduce expensive ad hoc range queries.
3. For monitoring-only presentation, remove the custom collector/API path and embed an authenticated Grafana dashboard over Prometheus in Cogrion. Keep a thin Workflow authorization boundary for per-workflow viewers and retain only optional immutable completion summaries needed by Spark Recommendation. Unmodified, separately deployed Grafana OSS is subject to AGPL terms; modifications, plugins, branding, and Enterprise/OEM use require review.
4. Reduce the CronJob interval. This lowers but does not remove gaps, increases collector and storage cost, and still mixes measurement windows.
5. Use Spark event logs alone. Rejected as the only source because they do not contain every Kubernetes container and PVC utilization signal.
6. Keep one point and label it a summary. Rejected because a sample is not an execution summary.

Component involvement

• sparqd/workflow-resource-monitoring implements lifecycle discovery, timestamped range aggregation, completeness state, and finalization.
• Prometheus retains or records application-labelled series for the supported execution window.
• Kubernetes and Spark Operator provide start, terminal state, and canonical application identity.
• The canonical producer, whether monitoring or recommendation during transition, writes one shared measurement contract.
• sparqd/platform-stacks configures retention, reconciliation cadence, recording rules, timeouts, and alerts rather than assuming a fixed tick is complete coverage.
• Spark Recommendation rejects incomplete observations and references the finalized task-execution measurement set.

Reproduction and verification

Follow RM-S007.

Resolution conditions

Close when short and long tasks are captured independently of the reconciliation tick, recorded statistics are reproducible from timestamped source series, different metric windows are explicit, missed events recover, incomplete measurements cannot produce recommendations, and coverage remains correct when one reconciliation run overlaps or misses its next schedule.

Validation and blockers

Monitoring, Prometheus, Kubernetes, Spark, data, recommendation, infrastructure, operations, and FinOps reviewers must validate measurement semantics, retention, query cost, and acceptable source gaps. Live series and deployment evidence in realm gj76anadpsc, workspace w-t5e2kcxjd, remain pending.