ARCH-SR-001 — Separate interactive state from analytical history
| Field | Value |
|---|---|
| Status | Open architecture decision; current behavior source-verified |
| Severity | Medium |
| Components | sparqd/spark-recommendation, Delta Lake, Trino, sparqd/bff-api, sparqd/platform-stacks |
| Associated issue | Not opened |
Current behavior and evidence
A daily SparkApplication writes a small recommendation batch to a Delta table
partitioned by run_date. The primary read filters normalized DAG name—not the
date partition—and computes latest-per-task through Trino. The BFF fans out one
API request per Workflow, while the API runs two replicas.
The job scans all History applications, reads the resource table, and can call the LLM again for an unchanged application. These compute and request costs are more material than the recommendation text stored in object storage. Repeated Delta merges also create obsolete files, and no compaction or vacuum policy is implemented.
The recorded LLM cost is not reliable chargeback: the writer applies one hard-coded token price to total tokens without a model/version or distinct input and output rates.
Expected condition
Interactive current-state reads are bounded, low-latency, authorized, and keyed by immutable Workflow/task identity. Immutable analytical/audit events use Delta only when fleet analysis, evaluation, or retention requirements justify it. Processing skips unchanged application/configuration fingerprints and reports cost per new valid recommendation.
Alternatives considered
- Recommended: operational database for current state and user disposition; optional append-only Delta archive for analytics and audit.
- Keep Delta only, remove daily partitioning for the small table, separate immutable history from latest state, bound reads, and compact only when file counts justify the Spark cost.
- Generate on demand and cache by application/configuration fingerprint. This reduces idle cost but adds first-view latency and still needs durable state.
Reproduction and verification
Follow SR-S006.
Resolution conditions
Close when measured workload volume, latency, scanned bytes, file growth, compute, API, and LLM costs justify the selected store; unchanged inputs incur no new generation cost; and operational versus analytical retention is clear.