OPS-ML-001 — Protect ML state and artifacts
| Field | Value |
|---|---|
| Status | Open finding; source and topology verified; restore pending |
| Severity | Critical |
| Components | sparqd/platform-stacks, sparqd/agent-based-ml-platform, MLflow, PostgreSQL, object storage |
| Associated issues | Not opened pending finding review |
Concrete observed case
AutoML and MLflow each use a single-replica PostgreSQL cluster with 20 GiB
storage, no database-native backup configuration, and termination policy
Delete. AutoML also accumulates uploads, processed data, generated metadata,
notebooks, models, and other outputs in shared object storage. No authoritative
retention, deletion, legal-hold, capacity-alert, backup, restore, or cross-store
consistency contract was found.
Expected condition
Classify every state and artifact, assign an owner and retention period, enforce authorized deletion, alert before storage exhaustion, and back up the minimum authoritative set. Restore tests must recover workflow-to-run-to-model-to-artifact lineage and permissions to declared recovery objectives.
Why it matters
Database loss can orphan expensive artifacts; object loss can leave successful runs unusable. Unlimited retention raises storage cost and data-governance risk, while destructive cluster deletion can remove the only metadata copy.
Existing controls and workarounds
Persistent volumes and object storage survive ordinary pod restart. Neither is a backup or proof of recoverability. A platform-level snapshot may protect infrastructure, but no assessment evidence shows an application-consistent, coordinated database-and-object restore.
Alternatives considered
- Recommended: scheduled database backups, object versioning/lifecycle policy, inventory, consistency audit, and routine isolated restore tests.
- Rebuild metadata from object storage — insufficient because authorization, metrics, parameters, workflow state, and registry identity are not wholly derivable from artifacts.
- Retain everything indefinitely — rejected for cost and governance.
Component involvement
- Platform stacks declares non-destructive lifecycle, backup, monitoring, and recovery resources.
- AutoML and MLflow expose retention-safe deletion and consistency identifiers.
- Operations owns objectives, restore exercises, and capacity alerts.
Reproduction and verification
Follow ML-S009.
Resolution conditions
Close when approved retention is enforced, backups are monitored, and a timed restore proves consistent authorized access to representative workflows, runs, models, and artifacts.
Validation and blockers
The current topology and missing declared controls are confirmed. Data classification, recovery objectives, aggregate inventory, and restore results remain unverified.