PERF-ML-001 — Define AutoML execution capacity
| Field | Value |
|---|---|
| Status | Open finding; source and deployed resources verified |
| Severity | High and cost exposure |
| Components | sparqd/agent-based-ml-platform, sparqd/platform-stacks, Spark, sparqd/oqullus |
| Associated issues | Not opened pending finding review |
Concrete observed case
Two fixed AutoML API pods each request 1 CPU and 8 GiB and may accept concurrent
training work. Embedded Spark defaults to local[*]. There is no durable queue,
per-user or workspace concurrency limit, admission response, queue depth,
dedicated worker profile, or workload-based autoscaler. The fixed service
therefore reserves 16 GiB while idle but still lacks predictable isolation when
several users run expensive jobs.
Expected condition
Keep request-serving capacity small and stable. Admit training into explicit Python/Spark worker profiles with per-workspace and per-user quotas, queue visibility, dataset limits, timeouts, and autoscaling bounded by downstream Trino, MLflow, database, node, and cost capacity.
Why it matters
Increasing API replicas mixes availability with compute cost and does not prevent simultaneous runs from exhausting memory or CPU. Local Spark also makes "large" execution dependent on one pod rather than a managed distributed capacity envelope.
Existing controls and workarounds
Platform configuration exposes replica count, Spark master, driver memory, and data-tier thresholds. Operators can size a workspace manually and restrict user access. Users cannot see available capacity or queue state.
Alternatives considered
- Recommended: small API plus queue-driven workers or per-run Kubernetes/Spark workloads that can scale to zero and use declared profiles.
- Fixed worker pool with strict admission and measured low-load envelope.
- Scale API replicas from CPU alone — rejected because memory-heavy training, queue state, and downstream saturation are not represented reliably.
Component involvement
- AutoML estimates work, admits it, applies quotas, and emits queue/resource metrics.
- Platform stacks exposes safe profiles and bounded autoscaling.
- Oqullus shows queued state, limits, estimates, and actionable rejections.
Reproduction and verification
Resolution conditions
Close when representative concurrent workloads establish a documented throughput, wait-time, memory, dataset-size, timeout, and cost envelope and the system rejects excess work safely.
Validation and blockers
Configuration and deployed resources are confirmed. Representative workload measurements, quality targets, and a cost budget remain unverified.