REL-ML-001 — Make AutoML execution durable
| Field | Value |
|---|---|
| Status | Open finding; source and topology verified; failure exercise pending |
| Severity | Critical |
| Components | sparqd/agent-based-ml-platform, sparqd/oqullus, sparqd/platform-stacks, PostgreSQL |
| Associated issues | Not opened pending finding review |
Concrete observed case
AutoML accepts a request, records state, and runs work in the accepting FastAPI pod. There is no external durable queue, worker lease, heartbeat, or orphan reconciliation. A pod restart can stop work while the database remains running.
Oqullus stores the active ID in browser storage, while server history lists terminal runs. Its UI exposes no cancellation. The backend cancellation route marks state and waits for the active agent to finish before stopping later agents; it does not prove active compute stopped.
Synthetic generation uses the same in-process background mechanism but does not check cancellation while fitting or before publishing output. A generic cancel can mark its workflow cancelled and the still-running task can later overwrite that state as completed or failed.
Expected condition
A durable queue owns each immutable attempt. Workers lease and heartbeat work; lost leases reconcile to retryable or failed state. Cancellation propagates to active Python, Spark, Trino, and AI Gateway work and records a confirmed terminal outcome. Terminal state must be monotonic, including synthetic generation. Any browser can recover authorized active runs from server state.
Why it matters
Pod rollout, eviction, crash, or node loss can strand jobs and consume resources without trustworthy status. Users cannot distinguish queued, running, retrying, cancel-requested, cancelled, failed, or orphaned attempts.
Existing controls and workarounds
PostgreSQL preserves status and per-agent summaries, and workflow code checks for cancellation between agents. Operators can inspect pods and manually repair state. These are not automated recovery.
Alternatives considered
- Recommended: durable queue plus separately scalable workers, attempt leases, heartbeat, idempotency, reconciliation, and propagated cancellation.
- Kubernetes Job or SparkApplication per attempt with PostgreSQL as control state and a reconciler.
- Keep in-process work and increase replicas — rejected because replica count does not make an accepted task durable.
Component involvement
- AutoML owns the workflow state machine, attempts, lease, reconciliation, and cancellation propagation.
- Platform stacks deploys queue/worker capacity and lifecycle policy.
- Oqullus lists authoritative active runs and exposes retry/cancel outcomes.
Reproduction and verification
Follow ML-S005.
Resolution conditions
Close when controlled pod loss cannot strand a run, retry creates an auditable attempt without duplicated effects, cancellation stops active work within a declared bound, and active state recovers across browsers.
Validation and blockers
The source topology and UI gaps are confirmed. A controlled destructive failure test and retry/cancellation objectives remain pending.