REL-SR-001 — Bind advice to one execution and valid metrics
| Field | Value |
|---|---|
| Status | Open; source-verified |
| Severity | High |
| Components | sparqd/spark-recommendation, sparqd/workflow-resource-monitoring, Spark History Server, Prometheus |
| Associated issue | Not opened |
Current behavior and evidence
The job selects the latest completed History application per DAG/task and the latest monitoring row per DAG/task independently. It joins them only by the two names. They need not describe the same Workflow run, task attempt, Spark application, configuration revision, or observation window.
Several derived fields are also misleading: totalDuration is named CPU time,
executor maxMemory capacity is named peak memory used, and per-executor cores
and memory are presented as cluster totals. Missing resource metrics do not
prevent LLM generation.
Expected condition
One immutable envelope correlates Workflow UUID/revision, run, task attempt, Spark application/attempt, exact configuration, and complete timestamped observations. Metric definitions preserve source unit and semantics. Missing or invalid required inputs prevent recommendation generation.
Alternatives considered
- Recommended: lifecycle correlation by immutable IDs and execution window.
- Timestamp-nearest matching. Useful only for migration and still ambiguous under retries or overlap.
- Keep latest-by-name matching. Rejected because a plausible recommendation can be based on a different execution.
Reproduction and verification
Resolution conditions
Close when mismatched executions cannot join, all metric names reproduce from their sources, completeness is explicit, and incomplete evidence cannot produce advice.