REL-ML-002 — Preserve and explain failed AutoML runs
| Field | Value |
|---|---|
| Status | Open reliability finding; live and source-verified |
| Severity | High |
| Components | sparqd/agent-based-ml-platform, sparqd/bff-api, sparqd/oqullus |
| Associated issues | Not opened pending finding review |
Concrete observed case
On 8 September 2026, AutoML run dc103820 failed during Model Building. MLflow
retained five related runs: ingestion, EDA, and feature engineering finished;
model building and the parent workflow failed. The model-building run was
af8fbbc1bfc0458a9edfbde770862bd0.
The status API reported the failed stage but no reason. The backend's separate workflow-error API retained the message No valid candidate models passed validation, while the result API returned only a failed status. Oqullus showed a generic failure until the page was reloaded, after which the failure panel disappeared. Run History also prevents users from opening failed runs.
Expected condition
A failed run remains selectable and recoverable after reload and from Run History. Its page shows the failed stage, a safe user-readable cause, suggested next action, and stable run identity. For candidate-validation failure, retain a summary of each candidate's rejection reason so the user can correct the data or configuration. Raw tracebacks remain restricted to authorized operators.
Why it matters
The current page appears to lose a recorded experiment and leaves the user with no way to understand or correct the failure. Support staff must search pod logs, even though the backend already stores a structured error. Repeating the run without diagnostics wastes compute and can reproduce the same outcome.
Existing controls and workarounds
The backend persists workflow and agent-run state, exposes a workflow-error endpoint, and records the failed child and parent runs in MLflow. An operator can correlate the run ID with pod logs. These controls are not available through the normal user journey.
Alternatives considered
- Recommended: expose an authorized, sanitized failure-detail contract through BFF, keep terminal runs addressable, and allow Oqullus to reopen them from history or a stable route.
- Include a safe error summary directly in workflow status. This reduces one request but must avoid returning operator-only tracebacks on a polling API.
- Keep failures only in MLflow. Rejected because users should not need a second product or pod-log access to recover an AutoML workflow.
Component involvement
- AutoML backend owns durable structured failure details and per-candidate validation outcomes. It already exposes workflow-level error details but does not retain the individual candidate rejection reasons needed for diagnosis.
- BFF API must expose an authorized, sanitized failure-detail route; it currently proxies status and result only.
- Oqullus must fetch and render the safe error, preserve the selected failed run across reload, and enable failed runs in Run History.
Reproduction and verification
Follow ML-S014.
Resolution conditions
Close when a failed run can be reopened after reload and from history, displays an actionable safe reason, preserves its stage and run correlation, and keeps technical diagnostics behind an operator-authorized boundary.
Validation and blockers
The deployed failure, stored error, missing BFF route, Oqullus state deletion, and disabled failed-history action are verified. A post-change authenticated reload check remains pending.