ARCH-WF-003 — Expose concurrency intent and manage Airflow capacity internally

Field	Value
Status	Improvement backlog; source-verified, scale boundaries pending
Architecture state	Deployed foundation; target management policy incomplete
Severity	High
Affected surface	Workflow concurrency, queued tasks, scheduler availability, Kubernetes task pods, worker nodes, quotas, and streaming applications
Components	sparqd/oqullus, sparqd/workflow-backend, Airflow, sparqd/platform-stacks, Karpenter, Kubernetes, PostgreSQL
Associated issue	Not opened

Where the problem appears

Create workflows with overlapping schedules and enough runnable tasks to exceed normal workspace concurrency. Observe user-visible queue semantics, Airflow scheduler health, task-pod creation, pending-pod age, worker-node scaling, database health, and recovery after the burst.

Concrete observed case

Workflow already exposes and compiles Max Active Runs, which limits overlapping DAG runs. The assessed Airflow deployment uses KubernetesExecutor, global parallelism 128, one scheduler, one webserver, and one triggerer. Each runnable task can create an ad hoc Kubernetes pod, while an airflow-worker Karpenter node pool can add nodes in response to pending pods and consolidate empty capacity.

No product contract was identified for workspace admission, maximum concurrent tasks, shared-resource quotas, queued-run expectations, or overload feedback. Scaling raw Airflow and Kubernetes settings directly with registered-user count would not represent actual demand: schedule frequency, runnable task count, task duration, pod startup, scheduler throughput, and metadata-database health are the controlling signals.

Expected condition

Keep the user contract about execution intent, and keep infrastructure capacity managed by the platform:

• Present Max Active Runs under a clear Concurrency or Run policy section, with overlap and queue semantics.
• Add a maximum concurrent-task control only if users need that workload-level boundary and the backend can enforce and explain it consistently.
• Do not expose scheduler replicas, worker counts, Airflow pool names, Karpenter node counts, database sizing, or raw infrastructure profiles as per-workflow settings.
• Scale task capacity from runnable or pending work. Guard each workspace with admission limits, Airflow pools or an equivalent managed policy, Kubernetes ResourceQuota, and clear queued or throttled states.
• Size schedulers from queued-task age, scheduling-loop and DAG-parse duration, database latency and connections, and pending-pod behavior. Use two schedulers for production availability after database capacity and migration behavior are proven; add more only with measured demand.
• Treat streaming applications as managed long-lived services or deployments. Airflow performs finite deploy, update, stop, and reconcile operations rather than owning a task that runs for the stream's entire lifetime.

Cause and evidence

• Oqullus and workflow-backend already persist and compile max_active_runs.
• The assessed Airflow configuration uses KubernetesExecutor with global parallelism 128 and a single scheduler replica.
• The Airflow worker node pool scales from Kubernetes pod demand and removes empty capacity; it does not need an end-user worker-count control.
• No UI/backend contract for per-workflow maximum active tasks, workspace admission, shared quotas, or overload feedback was identified.
• Spark task execution currently uses an Airflow operator whose lifetime can be coupled to application log following.

Why it matters

Unbounded frequent schedules and runnable tasks can saturate the scheduler, metadata database, Kubernetes API, namespace quotas, or node supply before the product explains why runs are queued. Exposing raw infrastructure knobs would shift unsafe platform concerns to users and produce configurations that are difficult to support. Treating streaming jobs as ordinary long-running Airflow tasks permanently consumes orchestration slots and makes Airflow maintenance part of application availability.

Existing controls and workarounds

Use Max Active Runs to prevent excessive overlap and keep conservative schedules. Platform operators can manage Kubernetes and Karpenter capacity and Airflow global parallelism. These controls are not yet a documented workspace-level admission or service-level boundary.

Alternatives considered

1. Recommended: simple workflow concurrency intent with managed background autoscaling, workspace quotas, and observable queue behavior.
2. Offer fixed workspace capacity tiers such as Small, Medium, and Large — a reasonable commercial or operational abstraction when predictable bounds matter, provided raw Airflow/Kubernetes settings remain internal.
3. Expose scheduler count, worker count, pools, node counts, or infrastructure profiles directly — rejected because these are shared platform mechanisms, not stable workflow intent.
4. Scale from registered or signed-in user count — rejected because it does not predict runnable-task load or scheduler and database pressure.

Component involvement

• sparqd/oqullus presents only supported concurrency intent and queued or throttled feedback.
• sparqd/workflow-backend persists and validates concurrency policy, compiles it into runtime controls, and applies admission to authoring and manual commands. It cannot admit scheduled runs that originate inside Airflow.
• Airflow and platform policy enforce runtime admission for scheduled and manual demand and expose scheduler, queue, and database health signals.
• sparqd/platform-stacks, Karpenter, Kubernetes, and PostgreSQL provide managed capacity, quotas, high availability, and scaling safeguards.
• A separate managed-service lifecycle owns streaming Spark applications while Workflow supplies finite deployment operations.

Reproduction and verification

Follow WF-S024.

Resolution conditions

Close when the product explains overlap, queueing, and throttling; workspace load cannot exhaust shared capacity; node scaling and scheduler availability meet documented boundaries; users do not configure raw infrastructure; and streaming applications survive normal Airflow task and deployment maintenance.

Validation and blockers

The current execution and node-scaling foundation is source-verified. Remaining evidence requires representative burst and sustained-load tests, scheduler and database telemetry, quota decisions, production availability objectives, and a defined managed lifecycle for streaming applications.

Reference behavior: Airflow KubernetesExecutor and high-availability schedulers.