PERF-SQL-001 — Add managed SQL capacity and workload admission

Field	Value
Status	Open finding; deployed topology and source verified, saturation behavior not tested
Severity	High
Affected surface	Classic SQL execution, concurrent queries, queueing, latency, failure recovery, and cost
Implementation issue	Not opened
Components	sparqd/platform-stacks; Trino; Trino Gateway; Kubernetes; Prometheus; sparqd/oqullus; BFF and Superset as request-path dependencies

Where the problem appears

Concurrent SQL Editor requests share the query capacity behind the Trino gateway. The editor presents only local Executing state; it does not expose a distinct queued, throttled, or capacity-unavailable state.

Concrete observed case

On 3 September 2026, realm gj76anadpsc, workspace w-t5e2kcxjd, had one Trino coordinator and one Trino worker with fixed resources. The gateway had a horizontal autoscaler, but the query worker did not. No query workload-admission or resource-group policy was evidenced. Two bounded queries had previously overlapped successfully; that does not establish a safe concurrency boundary.

Expected condition

The platform should protect interactive query service before saturation and scale query workers from workload signals where scaling is appropriate:

1. Apply admission, concurrency, queue, memory, and fairness policies by a trusted workload identity.
2. Return truthful Queued, Throttled, Capacity unavailable, Running, and terminal states to the SQL Editor.
3. Scale workers within explicit minimum, maximum, stabilization, cost, and safe-scale-down boundaries.
4. Keep coordinator and worker sizing, JVM settings, and Kubernetes resources internal to the platform rather than exposing raw infrastructure controls.

Cause and evidence

• Current deployment configuration selects a fixed capacity profile and worker count, and disables query-worker autoscaling.
• The deployed query cluster matched that fixed model: one coordinator and one worker. Only the gateway had horizontal autoscaling.
• Gateway scaling increases routing capacity; it does not create more Trino execution capacity.
• Trino resource groups support admission and queuing, but no active policy was evidenced for this deployment.

Why it matters

A burst, large scan, or memory-heavy query can contend with unrelated users. Without admission, the platform cannot state who queues, who receives priority, or how overload is communicated. Without managed worker scaling, increasing demand requires an operator change and fixed capacity remains allocated between bursts.

Existing controls and workarounds

• Operators can change the capacity profile and fixed worker count at deployment time.
• The gateway can route queries to registered clusters and auto-scales its own replicas.
• Users can reduce scan size, result size, and concurrency, but the SQL Editor does not publish an enforceable workload budget.

These controls do not supply query admission or managed query-worker capacity.

Alternatives considered

1. Recommended sequence: introduce resource-group admission and visible queue states, then add workload-aware worker scaling with safe scale-down.
2. Provision a larger fixed cluster. This raises the ceiling but retains idle cost and has no fairness or overload contract.
3. Scale only the gateway. Rejected as a query-capacity solution because the gateway does not execute queries.
4. Let users set worker counts or pod resources directly. Rejected for the product surface; expose logical intent such as Interactive or Batch instead.

Component involvement

• Platform stacks currently defines the fixed query-cluster profile, worker count, graceful shutdown, and metrics. It would need to add the selected admission and worker-scaling configuration.
• Trino reports query state and would enforce workload admission once a resource-group policy is configured.
• Trino Gateway currently routes requests but does not replace query-compute scaling.
• Oqullus, BFF, and Superset would need to preserve and present capacity states without retrying overload as fresh engine executions.
• Prometheus and the platform operations plane expose existing metrics and would need to supply the approved scaling and saturation signals.

Reproduction and verification

Follow SQLC-S049. Current state can be verified without a load test. Establishing the performance boundary requires an approved environment and controlled concurrency ramp; do not infer it from replica count or two successful overlapping queries.

Resolution conditions

Close when the platform has a documented admission policy, user-visible queue and overload states, measured concurrency and recovery boundaries, guarded worker scaling or a documented fixed-capacity decision, and alerts/runbooks for saturation and failed scale actions.

Validation and blockers

Controlled saturation, scale-out, safe scale-in, cancellation, and recovery evidence is pending. Product must validate workload classes; platform and data operations must validate capacity, cost, and failure boundaries.