OPS-WF-001 — Prevent Airflow metadata exhaustion and rehearse upgrades

Field	Value
Status	Open; source-verified, deployed capacity and recovery evidence pending
Architecture state	Deployed risk; target operating model absent
Severity	Critical
Affected surface	Scheduling, task and DAG-run state, heartbeats, history, cleanup, backup, restore, and Airflow upgrades
Components	sparqd/platform-stacks, Airflow, KubeBlocks, PostgreSQL, platform operations
Associated issue	Not opened

Where the problem appears

Inspect the Airflow metadata database capacity and growth while creating and retaining representative DAG runs, task instances, jobs, rendered templates, XCom, and other metadata. Then exercise cleanup, restore, and an Airflow schema migration against a production-sized database copy.

Concrete observed case

The assessed stack provisions one PostgreSQL 16.4 replica with a 20 GiB volume, 500 millicores and 512 MiB requested, and 1 CPU and 1 GiB limited. Its exporter is disabled. No scheduled Airflow metadata retention, archived-row export and drop, database-capacity alert, Airflow database backup/restore test, or documented PostgreSQL maintenance policy was identified.

Airflow waits for database migrations during deployment, with a 900-second Helm timeout. Airflow's documented upgrade flow runs migrations while components are stopped and recommends timing the migration against a database copy. The current deployment source does not define that rehearsal or an operational drain and rollback procedure.

Expected condition

Operate the Airflow metadata database as a capacity-critical stateful service:

• Alert on used bytes, free bytes, growth rate, forecasted exhaustion, failed writes, connection saturation, and migration duration before the volume is nearly full.
• Allow controlled volume expansion as an emergency guard, while retaining a bounded metadata lifecycle so expansion is not the retention policy.
• Schedule reviewed retention using dry-run, backup, cleanup, archive export or deliberate archive skipping, archive removal, and PostgreSQL maintenance.
• Keep only operational history required by Airflow in its metadata database; send longer-lived audit or analytical history to a separate system.
• Provide backup or point-in-time recovery and prove restore into an isolated environment.
• Before an Airflow upgrade, clone a representative database, time the schema migration, verify free-space headroom, and test rollback and smoke checks.
• During the production upgrade, pause new schedules, define how running tasks drain or are interrupted, stop Airflow components for migration, and use a timeout based on the rehearsal rather than an untested fixed default.

Cause and evidence

• sparqd/platform-stacks declares a single-replica 20 GiB KubeBlocks PostgreSQL cluster for Airflow and disables its exporter.
• No Airflow db clean, db export-archived, db drop-archived, metadata retention, metadata backup, free-space alert, or cleanup schedule was found in the assessed stack source.
• Airflow documents that db clean can archive rows in the same metadata database before deletion. Cleanup started near exhaustion can therefore need additional space unless archive creation is deliberately skipped.
• Airflow documents stopping components during migration and testing migration duration on a database copy for installations with substantial history.

Why it matters

If the 20 GiB volume fills, Airflow can no longer reliably create or update DAG runs, task instances, scheduler jobs, heartbeats, or other metadata. Scheduling and state transitions can fail together, making the workflow service unavailable and complicating recovery. A large unpruned database also makes schema migration slower and less predictable, increasing upgrade downtime and the chance that the fixed deployment timeout expires.

Existing controls and workarounds

The database is persistent and KubeBlocks manages it as a stateful cluster. Until lifecycle automation exists, operations must watch volume use externally, expand capacity before cleanup if headroom is unsafe, take a recoverable backup, and run Airflow cleanup and PostgreSQL maintenance as a controlled procedure. Do not first attempt archive-producing cleanup on an almost-full volume.

Alternatives considered

1. Recommended: bounded operational retention, tested backup/restore, capacity forecasting, controlled expansion, and rehearsed upgrades.
2. Retain everything and auto-expand indefinitely — rejected because database growth still degrades maintenance, query, backup, restore, and upgrade time.
3. Periodically delete database rows directly — rejected because bypassing Airflow's supported cleanup lifecycle risks referential and product-state damage.
4. Use airflow db clean --skip-archive — acceptable only when a recoverable backup or external archive satisfies the retention requirement; it is not a universal default.

Component involvement

• sparqd/platform-stacks declares database capacity, expansion, backup, monitoring, cleanup scheduling, migration controls, and production Airflow availability.
• Airflow supplies the supported metadata cleanup and migration commands.
• KubeBlocks and PostgreSQL provide storage, backup/restore, capacity, and database-maintenance behavior.
• Platform operations owns thresholds, retention decisions, rehearsal, runbooks, alerts, and recovery validation.

Reproduction and verification

Follow WF-S022.

Resolution conditions

Close when metadata retention and maintenance run safely on schedule, capacity alerts provide actionable lead time, expansion and backup/restore are proven, and a representative database-copy migration plus production drain/rollback procedure complete within documented boundaries.

Validation and blockers

Deployment source establishes the current capacity and missing lifecycle controls. Remaining evidence requires database size and growth measurements, retention approval, a safe cleanup trial, backup/restore proof, and an upgrade rehearsal outside the production database.

Reference behavior: Airflow database maintenance and database migration.