AI/ML architecture

System boundary

AutoML, Experiments, and Models belong to one ML lifecycle because training should produce traceable run and model identities. AI Gateway is a separate GenAI control plane called during AutoML reasoning. It does not host the trained artifact.

State and identity ownership

State	Current authority	Important boundary
Active AutoML selection and run	Oqullus browser plus AutoML polling	Browser storage is not authoritative recovery across devices
Workflow and agent state	AutoML PostgreSQL	Work itself remains attached to an API process rather than a durable queue lease
Uploaded, processed, generated, and model artifacts	Workspace object storage	Retention and deletion ownership are not declared
Synthetic dataset and synthesizer	AutoML object storage plus workflow summary	The dataset combines original and generated rows; quality below threshold does not block completion
Experiment run, metrics, and artifact lineage	MLflow	AutoML uses a separate workflow ID and different ownership tags from the BFF listing policy
Registered model and version	MLflow Model Registry	Supervised AutoML does not register; unsupervised registration uses one shared name
Training data authorization	Trino/Ranger under an AutoML service account	The signed-in user identity is not propagated
GenAI provider routing	AI Gateway	This identity is unrelated to MLflow registered-model identity
Served endpoint	No supported product authority	The Oqullus route is disabled and AI Gateway is not this serving plane

Query and execution routing

• BFF authenticates the browser-facing request, but the deployed AutoML ingress is also externally reachable without an inbound authentication gate.
• AutoML creates its own workflow record and schedules work inside the accepting API pod. PostgreSQL records status; it does not lease work to a durable worker.
• CSV inputs go to object storage. Trino inputs are queried through a shared service-account token. Data may be materialized into the API pod for Python or an embedded local Spark session.
• All uploaded files enter Spark first. The service estimates size and selects an algorithm tier, but the deployed Spark master is local[*]; the labels large and massive do not create distributed compute. Remote files are first downloaded through the API process, and an omitted Trino row limit can be fetched before the engine tier is selected.
• Synthetic generation is another in-process background task. It accepts up to 200 MB and limits original plus requested rows to 500,000. It warns rather than fails when measured quality is below 0.65 or quality evaluation fails.
• AutoML calls AI Gateway through BFF for LLM assistance and logs nested work to MLflow using a service identity.
• Oqullus polls AutoML directly through BFF for workflow state. Experiments and Models use separate MLflow proxy paths and resource-filter conventions.
• Notebook deployment source does not inject MLFLOW_TRACKING_URI, an MLflow-audience credential, or a supported registration helper. Direct notebook-to-MLflow use therefore requires manual setup and can bypass the ownership metadata assumed by BFF listing filters.

Data-access enforcement

Three enforcement layers must not be conflated:

1. Workspace authentication decides whether a person can enter Oqullus.
2. AutoML resource authorization must decide whether that person owns or may access a workflow, prediction, notebook, input, or output. This is missing on the directly exposed backend and incomplete in BFF.
3. Trino/Ranger decides which physical data may be read. Current AutoML reads use a shared service identity, so user-scoped SQL access is not preserved.

MLflow has its own experiment and model permission system. BFF filters some resources by owner tags, but checks are not consistently applied to every name-, ID-, run-, metric-, or artifact-based route.

Model registration should cross a server-authorized boundary that derives the workspace and owner from the signed-in principal. Notebook-supplied tags are useful metadata, but must not be the authorization authority.

Deployment and scale boundary

The assessed workspace runs two fixed AutoML replicas. Each requests 1 CPU and 8 GiB and can execute training in its API process. There is no external work queue, dedicated worker deployment, workload-based autoscaler, admission controller, or disruption budget. This reserves 16 GiB while idle and offers no safe contract for concurrent expensive runs.

MLflow was observed as one application pod. KCL declares one-to-two autoscaling, but no live HPA was found. AutoML and MLflow each depend on a single-replica PostgreSQL cluster with 20 GiB storage, no database-native backup policy, and a destructive termination policy. A platform-level backup mechanism, if used, still requires an application-consistent cross-store restore exercise.

AutoML engine boundary

The current backend is itself the AutoML engine: it owns preprocessing, candidate selection, fixed hyperparameter grids, cross-validation, clustering, training, explanation generation, and MLflow logging. MLflow records lifecycle state but does not perform those AutoML decisions. The small-data validation path samples at most 10,000 rows and runs single-worker GridSearchCV; the large and massive labels select Spark algorithms but deployed Spark remains local[*] in the API pod.

Established open-source engines and components could strengthen different parts of this boundary. AutoGluon, FLAML, H2O AutoML, and PyCaret provide different levels of end-to-end automation; Optuna and Ray Tune strengthen search or distributed trial scheduling; Spark ML provides a distributed large-data training path. They are not equivalent replacements, and none removes the need for Cogrion's durable execution, authorization, resource, and product controls. See ARCH-ML-001 for the decision-neutral comparison and evaluation method.

Observability and failure state

Workflow and per-agent statuses are persisted and Oqullus polls them. The current design does not provide a durable queued/running lease, heartbeat, orphan reconciliation, authoritative retry attempt, or confirmed cancellation of active compute. A pod can disappear while the database still says a run is active. MLflow initialization can fail without preventing AutoML readiness, and system-metric logging is disabled for AutoML-created runs.

Required operational correlation is one stable chain:

workspace + user + workflow + attempt + MLflow run + model/version + Trino query + pod/worker

It should support queue time, run duration, resource use, dependency errors, cancellation latency, retry reason, artifact lineage, and final disposition without relying on a process-global user value.

Target lifecycle

1. Authenticate and authorize the request and input under an explicit user-delegation policy.
2. Create a durable workflow and attempt, then admit it to a bounded worker queue with a run-scoped identity.
3. Execute with heartbeats and cooperative plus forceful cancellation.
4. Bind an immutable input snapshot, validated configuration, effective engine, dependency versions, and random seeds to the attempt for reproducibility.
5. Create a visible authorized MLflow experiment and parent/training run linked to the workflow attempt.
6. Let the user explicitly register an accepted notebook or AutoML artifact through a visible, authorized action under a stable workspace-scoped model name and version.
7. Treat deployment as a separate lifecycle with endpoint identity, rollout, monitoring, rollback, and cost controls. Do not reuse AI Gateway's external chat-model registry for this purpose.

MLflow's registry preserves the relationship between a model version and its source run; registration alone does not provide serving. See the official MLflow Model Registry.