DATA-SR-001 — Implement logical and physical retention
| Field | Value |
|---|---|
| Status | Open; source-verified |
| Severity | High |
| Components | sparqd/spark-recommendation, Delta Lake, sparqd/platform-stacks |
| Associated issue | Not opened |
Current behavior and evidence
KCL supplies RETENTION_RUNS=7. The job ranks recommendations by run_ts
within each DAG/task and identifies rows beyond that limit, but the Delta delete
block is commented out. It counts the expired rows and prints Retention
skipped. No VACUUM command exists.
The setting therefore provides neither logical retention nor physical storage
retention. Even after logical deletes are implemented, obsolete Delta files
remain until a separate VACUUM passes its safety interval.
Expected condition
Define separate policies for:
- current recommendations retained per immutable Workflow task;
- generation/audit events retained by age and business requirement;
- failed or invalid generations;
- Delta time travel and disaster recovery; and
- physical file deletion.
Retention runs idempotently, reports rows and bytes removed, uses a dry-run control, and alerts on failure. Workflow deletion, renaming, and revision history have explicit behavior.
Alternatives considered
- Recommended: keep bounded operational state and apply age-based archival retention to optional immutable Delta events.
- Keep the latest N executions per task in Delta, followed by scheduled safe
VACUUM. Acceptable as a stabilization step. - Rely on table partition lifecycle alone. Rejected because the table is not safely disposable by entire day and logical history requirements differ.
Reproduction and verification
Follow SR-S005.
Resolution conditions
Close when the configured policy deletes the intended logical records, physical cleanup is safely scheduled and observed, recovery/time-travel limits are documented, and same-day executions plus Workflow revisions are retained correctly.