ARCH-ML-001 — Define the AutoML engine and compute boundary
| Field | Value |
|---|---|
| Status | Open finding; source and deployed configuration verified |
| Severity | High; critical for large-data claims |
| Components | sparqd/agent-based-ml-platform, Spark, Trino, MLflow, sparqd/platform-stacks, sparqd/oqullus |
| Associated issues | Not opened pending finding review |
Concrete observed case
Cogrion implements its own AutoML selection and training flow rather than using
an established AutoML engine. For supervised validation, it samples at most
10,000 rows, applies basic encoding and zero filling, evaluates a deterministic
candidate list, and uses single-worker GridSearchCV. Only seven model families
have small hard-coded parameter grids; candidates without a grid are evaluated
at their defaults. The LLM supplies explanations for the deterministic
candidates but does not control the bounded search.
Validation records the candidate attempts as one serialized MLflow parameter, not as separately inspectable trial runs. There is no study-level search state, pruning, ensemble construction, resumable trial scheduler, per-trial resource contract, or accuracy-versus-latency objective. Preprocessing and task inference are custom heuristics that Cogrion must maintain and benchmark.
The later training stage selects small, large, or massive algorithm paths.
However, deployed Spark uses local[*] inside the same API pod with a 14 GiB
memory limit. Remote objects are downloaded through the API process, and Trino
can call fetchall() before engine selection. The tier therefore changes the
algorithm implementation but does not currently establish a distributed
compute boundary.
Expected condition
Define an explicit AutoML capability contract before selecting or retaining an engine. At minimum it should cover:
- supported problem types, data types, algorithms, preprocessing, ensembles, explainability, and model-size or inference-latency constraints;
- time, trial, CPU, memory, and concurrency budgets with admission before data materialization;
- durable trial identity, progress, cancellation, retry, pruning, recovery, and complete failure diagnostics;
- independently inspectable MLflow runs for candidates, the selected artifact, input/configuration lineage, and reproducibility evidence;
- a measured local-data boundary and a separately verified distributed path;
- an honest UI presentation of the effective engine, candidates, ranking, limits, and compromises made for accuracy, duration, and inference cost.
Why it matters
The current flow can produce a model, but that alone does not establish a mature AutoML engine. Fixed grids can spend work on poor trials while missing better configurations; one summarized tracking run hides candidate behavior; and a size label cannot create distributed capacity. Cogrion also carries the ongoing cost of maintaining preprocessing, search spaces, compatibility, ensembles, model comparison, and quality benchmarks itself.
Open-source engine options
These are alternatives or complementary components, not a predetermined selection. None replaces authentication, durable job execution, admission control, data authorization, or Cogrion's product-level workflow.
| Option | What it strengthens over the current engine | Costs and limitations | Suitable assessment boundary |
|---|---|---|---|
| AutoGluon | Automated tabular preprocessing, a broader model portfolio, leaderboard, time-bounded fitting, and strong bagged/stacked ensembles. Its presets explicitly trade accuracy against training and inference cost. | Primarily a supervised tabular engine; clustering remains separate. Strong ensembles increase training, artifact, memory, and inference cost. Tabular workloads remain sensitive to single-machine memory, and Cogrion must map artifacts and candidate detail into MLflow and its APIs. | Compare as a bounded classification/regression engine when predictive quality and candidate exploration are primary. Evaluate its separate time-series package independently. |
| FLAML | Cost-aware model and hyperparameter search under time or resource budgets, MLflow integration, and optional parallel execution through Ray or Spark. It supports classification, regression, forecasting, and learning-to-rank tasks. | Less focused on automated feature engineering and high-quality stacked ensembles than AutoGluon or H2O. Spark support still requires a real Spark cluster and supported Spark estimators; it does not make local[*] distributed. | Compare where bounded cost, fast search, or reuse of the Spark execution environment matters more than maximum ensemble accuracy. |
| H2O AutoML | Distributed model training, automatic HPO, cross-validation, leaderboards, stacked ensembles, and model explainability through a mature AutoML runtime. | Introduces H2O's cluster, frame, artifact, and operational model. Spark interoperability uses Sparkling Water and is an additional integration boundary rather than native reuse of Cogrion's current Spark code. MLflow lineage and intended artifact-export licensing must be validated for the selected versions and deployment model. | Compare for large supervised tabular workloads when distributed training and ensembles justify another runtime. Do not infer fit solely from the presence of Spark. |
| PyCaret | Broad low-code workflows for preprocessing, model comparison, tuning, ensembling, analysis, forecasting, and direct MLflow experiment logging. This closely matches an interactive analyst or notebook journey. | It is primarily a high-level Python workflow over other libraries, not a durable distributed AutoML control plane. Its process state and mostly single-node model execution would still need isolation, recovery, and scale controls. | Compare as a notebook or guided low-code experience, or as a reference for Cogrion UX, rather than assuming it should own backend orchestration. |
| Optuna | Modern samplers, pruning of unpromising trials, persistent studies, and parallel or multi-node optimization through shared storage. It would replace exhaustive hard-coded grids while preserving Cogrion's chosen estimators. | It is an HPO framework, not a full AutoML engine: Cogrion still owns preprocessing, task inference, candidate models, ensembles, training jobs, and model selection policy. Durable compute and resource isolation must be supplied separately. | Compare as a search component when Cogrion deliberately wants to retain a custom engine and its product decisions. |
| Ray Tune | Distributed trial scheduling, search algorithms, early-stopping schedulers, per-trial CPU/GPU resources, failure policies, and MLflow callbacks. | Adds Ray and potentially KubeRay as another compute control plane. It does not provide automatic feature preparation or a complete model portfolio by itself. Data access and model training still require another library. | Compare when concurrent trials, GPU workloads, or multi-node Python training justify a general distributed tuning runtime. |
| Spark ML tuning | Distributed data preparation and training with CrossValidator or TrainValidationSplit, while reusing Cogrion's Spark and Delta/Trino direction. It avoids collecting genuinely large training data into the API process. | It has a narrower algorithm and AutoML feature set, and grid cross-validation can be expensive. It does not provide a complete AutoML product, ensemble strategy, or durable run orchestration. | Compare as the large-data execution path, independently of which local AutoML or HPO engine is selected. |
Comparison dimensions
| Dimension | Current custom engine | Questions every option must answer |
|---|---|---|
| Problem coverage | Classification, regression, heuristic time-series handling, and clustering | Are these native, separately implemented, or unsupported? |
| Preparation | Custom type detection, imputation, scaling, encoding, text handling, and simple lag features | How are leakage, high cardinality, imbalance, time ordering, and user overrides handled? |
| Search | Fixed model candidates and small grids; n_jobs=1 | Which sampler, stopping policy, budget, and parallelism are used? |
| Selection | Cross-validation score or silhouette score | Can users choose accuracy, latency, artifact size, explainability, or cost objectives? |
| Ensembles | Individual candidate selection only | Are ensembles supported, explainable, portable, and bounded for inference? |
| Trial state | Attempts serialized into one MLflow parameter | Are trials individually visible, resumable, cancellable, reproducible, and attributable? |
| Scale | API-process Python or Spark configured as local[*] | What is the measured local limit, and which path is genuinely distributed? |
| Integration | Custom APIs and artifacts already exist | What changes are required in AutoML, MLflow, Oqullus, platform stacks, and serving? |
Evaluation method
Run the same representative, non-sensitive datasets through viable options and the current engine. Include binary and multiclass classification, regression, mixed and high-cardinality features, missing values, imbalanced targets, time-ordered data, and a dataset beyond the local-memory boundary. Record:
- validation and held-out quality with leakage checks;
- elapsed and queue time, peak CPU/memory, trial count, failure isolation, and cancellation or recovery behavior;
- preprocessing decisions, candidate visibility, reproducibility, MLflow lineage completeness, artifact size, and batch/online inference latency;
- platform integration work, runtime dependencies, upgrade surface, licensing, and steady-state infrastructure cost.
No engine should be selected from feature lists alone. The assessment should retain multiple profiles if the measured requirements differ materially; for example, bounded supervised tabular work, clustering, forecasting, and distributed large-data training need not share one engine.
Existing controls and workarounds
KCL exposes Spark master, driver memory, tier thresholds, and API replica count. BFF caps an explicitly supplied Trino limit at 250 rows. Operators should require a bounded row limit and keep inputs within a measured local profile until a distributed runtime is verified. The current engine remains usable as a baseline for the comparison, but its fixed grids and tier names must not be presented as proof of broad or distributed AutoML support.
Component involvement
- AutoML owns the engine adapter, preprocessing and task contract, search budget, candidate state, artifact contract, and effective-engine evidence.
- MLflow stores run, metric, artifact, and model lineage; it is not the AutoML search or execution engine.
- Platform stacks and the selected compute runtime supply coherent local or distributed profiles, job isolation, capacity, cancellation, and recovery.
- Trino and object storage support bounded local reads or direct distributed reads without API-process materialization.
- Oqullus presents problem type, constraints, candidates, progress, ranking, engine, limits, and failure detail without exposing unsupported controls.
Reproduction and verification
Follow ML-S011.
Resolution conditions
Close when the engine decision is backed by representative benchmarks; each advertised problem and data-size profile maps to verified preparation, search, compute, tracking, cancellation, and recovery behavior; oversized data is rejected before API materialization; and the UI accurately states the effective engine and operational limits.
Validation and blockers
Current source and the deployed local Spark configuration establish the baseline gaps. No comparative benchmark, integration spike, representative large-data run, or approved product capability envelope has been recorded. Those decisions remain open; this finding intentionally does not select an engine.