AI/ML evidence and reproduction

Source was reviewed first. Live checks were limited to deployed topology and safe reads needed to establish whether the source behavior is active. No training job, data mutation, credential, user record, model, or artifact was created or disclosed during this assessment.

Backend source baseline and traceability

Backend claims were reverified on 8 September 2026 from a clean local checkout of sparqd/agent-based-ml-platform at full commit 3112ee9f7e721a5ad730073bca26bbb755218dd3. The links below are immutable commit permalinks; they remain reviewable without the assessor's local clone. This is the current main source baseline, not the deployed AutoML image baseline 6b3fa2a. A source-only conclusion must not be assumed deployed unless the scenario also records live or deployment evidence.

Scenarios	Backend evidence	Reverified conclusion
ML-S002	main.py application and router setup, workflow_error.py	No inbound authentication dependency or resource-ownership check protects the application or workflow-error route. Live negative probes separately prove the deployed exposure.
ML-S003	trino_connector.py	Alibaba Trino access obtains a Keycloak service-account token and uses TRINO_USER or the client ID, not signed-in user identity.
ML-S004, ML-S006	main.py workflow/MLflow identity, base_agent.py nested-run tags, model_training_agent.py registration, JupyterHub forwarded environment, JupyterHub single-user environment, MLflow OAuth boundary, BFF model reads, Oqullus model service	Workflow IDs are eight hex characters; user email is also placed in process-global environment state; agent runs read that state; only unsupervised training registers, under fixed name agent_based_model. Notebook pods have no MLflow tracking/auth bootstrap, while the BFF and Models UI provide reads but no registration path.
ML-S005	main.py accepted background work, workflow_orchestrator.py cancellation check, main.py cancellation route	FastAPI background tasks execute accepted work in the API process. Cancellation changes database state and is observed only between agents; the response explicitly says the current agent will finish. No queue, lease, heartbeat, or reconciler appears in this execution path.
ML-S009	db/models.py, config.py state paths	Source identifies workflow state and artifact classes but implements no application retention or coordinated database/object-store recovery contract. Deployment and restore claims still require KCL and live evidence.
ML-S010	main.py deployment router registration, routers/deployment.py	Backend deployment and prediction prototypes exist. The finding is therefore about the absence of an enabled, validated product journey—not the absence of backend code.
ML-S011	model_validation_agent.py, model_factory.py, model_training_agent.py, input_resolver.py, trino_connector.py materialization, spark_manager.py, spark_size_estimator.py	Supervised validation samples at most 10,000 rows, uses small fixed grids with GridSearchCV(n_jobs=1), and records attempts in one MLflow validation run. Trino calls fetchall() before wrapping data in Spark; remote objects are read into API memory; tier selection is size/row based; the Spark master defaults to local[*]. The deployed master and pod limits remain deployment evidence.
ML-S012	synthetic_data_generator.py limits and modes, synthetic_data_generator.py validation, synthetic_data_generator.py output and quality, main.py in-process enqueue	Limits, augmented output, warning-only quality, and in-process execution are confirmed. The synthetic task does not read cancellation state before publishing terminal output.
ML-S014	workflow_error.py, workflow_result.py, model_validation_agent.py	The backend persists and returns workflow/agent error detail separately; result returns only failed. When every candidate fails, validation raises before its attempt list is logged.
ML-S015	main.py optional target and fixed plan, workflow_orchestrator.py conditional Labeling, yaml_service.py generated rules, labeling_agent.py, model_building_agent.py, model_validation_agent.py	Missing target selects clustering. The default Labeling stage is retained and generates a fallback rule because schema columns are absent from the request summary; that rule produces no useful label, while the stage can still complete. Clustering later evaluates numeric features and writes a separate clustered result.

Evidence register

Scenario	Evidence	Result
ML-S001	Live deployment	Fixed AutoML replicas and storage topology confirmed
ML-S002	Source + live negative auth probe	Unauthenticated reads confirmed
ML-S003	Source + KCL	Shared Trino service identity confirmed
ML-S004	Source	No supported notebook registration path; AutoML identity and ownership are discontinuous
ML-S005	Source + topology	Durability and UI cancellation gaps confirmed; destructive live test pending
ML-S006	Source	Process-global user attribution risk confirmed; concurrency test pending
ML-S007	Source	Route inconsistency confirmed; cross-user live test pending
ML-S008	Source	Pagination loss confirmed; representative scale test pending
ML-S009	KCL + live topology	Single replicas and no database-native backup confirmed; restore test pending
ML-S010	Source	No supported trained-model serving journey
ML-S011	Source + KCL	Custom fixed-grid AutoML and automatic tiering use local-pod compute; engine and runtime boundaries are not exposed
ML-S012	Source	Visible flow confirmed; quality, output, and download limitations identified; live run pending
ML-S013	Source	Generic stage formatting renders eda as Eda without expanding the acronym
ML-S014	Live + source	Failed run retained server-side but loses its UI context after reload
ML-S015	Source	Missing target silently selects clustering while Labeling remains in the stage plan
ML-S016	Source	Machine Learning navigation currently contains both ML lifecycle pages and AI Gateway

ML-S001 — Deployment baseline

1. Select Kubernetes context dev-north-alicloud.
2. Inspect application workloads, images, replicas, resources, autoscalers, disruption budgets, PostgreSQL clusters, and ingresses in workspace w-t5e2kcxjd.
3. Confirm AutoML has two replicas at image 6b3fa2a, each requesting 1 CPU and 8 GiB; AI Gateway has one replica at 0b9c3d2; MLflow reports version 3.14.0.
4. Confirm no AutoML HPA or disruption budget is present.
5. Confirm AutoML and MLflow PostgreSQL each have one replica, 20 GiB storage, no database-native backup policy, and termination policy Delete.

Last observed: 4 September 2026. Workload demand and recovery objectives remain unrecorded.

ML-S002 — Direct AutoML access

1. Inspect the AutoML application for inbound authentication middleware and per-resource ownership checks.
2. Inspect its external ingress for an authentication middleware or policy.
3. Without an Authorization header, issue safe GETs for a nonexistent workflow, an intentionally nonexistent email's run list, and the model catalog.
4. Record only status, content type, and payload size. Do not print or retain the returned catalog.

On 4 September 2026 all three requests returned HTTP 200 JSON. The run-list probe returned an empty two-byte array; the model-catalog response was 5,399 bytes. This establishes the deployed access-control failure without exposing another user's data. Cross-user reads and all mutation probes were deliberately not attempted.

ML-S003 — Training-data identity

1. Trace the signed-in identity from Oqullus through BFF to AutoML.
2. Trace AutoML's Trino connection and deployed credential configuration.
3. Compare the Trino principal with the signed-in user used by SQL Editor.
4. Ask the security and data-governance reviewers to define whether AutoML is delegated user execution or a privileged service operation.

Source and KCL show AutoML obtains a service-account token and sends the service principal to Trino. User-scoped Ranger behavior and an authorized-denied table test remain pending.

ML-S004 — Notebook and AutoML-to-MLflow lifecycle

1. Run a controlled supervised and unsupervised AutoML job with a non-admin ML persona.
2. Record the AutoML workflow ID, MLflow parent run, training run, experiment, registered model, and version without recording credentials or training data.
3. Open Experiments and Models as the same user and try to navigate between all identities.
4. Repeat as an administrator only to distinguish missing objects from hidden objects.
5. In a Cogrion notebook, inspect configuration names only—never token values— for a supported MLflow tracking URI, MLflow-audience authentication, and a platform registration helper.
6. Log a disposable model artifact using the supported notebook path, if one is present. Confirm whether the notebook or UI explains that logging is not registration and offers Register model.
7. Register the disposable artifact through that action and confirm the resulting model/version has authoritative workspace and owner identity, appears in Models for the author, links back to the source run, and is denied to another non-admin user.
8. If no supported helper or action exists, record the journey as absent. Do not manually inject credentials merely to turn the assessment green.

Source confirms AutoML creates a separate short workflow ID, tags MLflow runs with user email, and creates per-user experiment names without the experiment owner tag used by BFF. Supervised training does not register a model; unsupervised training registers one fixed model name without the owner tag. Deployment source also confirms notebook pods receive general Cogrion/Jupyter OAuth context but no MLflow tracking URI or MLflow-audience credential, while the BFF and Oqullus Models service expose model reads only. A direct manual MLflow call is therefore not evidence of a supported notebook journey. Representative live completion and cross-user authorization remain pending.

ML-S005 — Execution recovery and cancellation

1. Start a controlled long-running job with disposable data.
2. Reload Oqullus, open a second browser, and observe whether each can recover the active run from server state.
3. Request cancellation and measure time until active compute stops and the authoritative state becomes terminal.
4. In a controlled maintenance window, remove the accepting AutoML pod and verify retry, attempt history, orphan reconciliation, and final status.

Source confirms work runs in the accepting API process, history lists terminal runs, active selection is browser-local, Oqullus exposes no cancel action, and backend cancellation waits for the active agent to finish. Pod-loss evidence is pending because this test changes deployed execution.

ML-S006 — Concurrent run identity

1. Start two controlled jobs as different users with overlapping execution.
2. Compare the AutoML owner, MLflow experiment, and all nested run owner tags.
3. Confirm no run or artifact is attributed to the other user.
4. Repeat at the declared concurrency limit.

Source confirms user attribution is held in process-global state while two replicas may execute concurrent requests. A controlled overlap test and the supported concurrency limit remain pending.

ML-S007 — MLflow authorization

1. Create two users and one experiment/model for each in a disposable scope.
2. As each user, list resources and request the other user's experiment by name, run by ID, run overview, metric history, model by name, versions, and artifacts.
3. Expect one consistent deny or not-found policy without metadata leakage.
4. Repeat through MLflow's own authorized path to ensure BFF and MLflow agree.

BFF source applies owner checks to some list, detail, version, and artifact paths but omits them from other name-, run-, metric-, and model-detail paths. The cross-user live matrix is pending.

ML-S008 — Large-collection discovery

1. In a disposable environment create more than 1,000 experiments, more than 100 runs in one experiment, more than 25 registered models, and more than 100 versions where supported.
2. Search and sort for objects outside the first server response.
3. Compare the UI count and pages with authoritative MLflow counts.
4. Verify forward/back page tokens and stable ordering under concurrent writes.

Source confirms experiment, run, model, and version tokens are not preserved end to end and Oqullus filters fetched subsets locally. Representative scale data is pending.

ML-S009 — Retention and recovery

1. Inventory workflow rows, run metadata, registered models, uploads, processed data, generated notebooks, model artifacts, and MLflow artifacts by age and owner using aggregate counts only.
2. Obtain approved retention, legal hold, deletion, recovery-point, and recovery-time requirements.
3. Back up both databases and required object prefixes in a disposable environment.
4. Restore to a new workspace and verify workflow-to-run-to-model-to-artifact consistency and authorization.

No application-specific retention, database-native backup, or completed coordinated restore evidence was found. A platform-level snapshot alone would not prove database-and-object consistency. Recovery remains unverified.

ML-S010 — Model-serving boundary

1. Train a controlled candidate and inspect the visible Models actions.
2. Confirm whether a user can register, promote by alias, deploy, observe, roll back, or delete it.
3. Inspect AI Gateway's endpoint model source and distinguish external provider models from MLflow versions.

Source confirms Models is a read-only browser, the Model Serving route is disabled, and AI Gateway stores external GenAI provider models. Serving an AutoML/MLflow artifact is not a supported current journey.

ML-S011 — AutoML engine and data-engine configuration

1. Inspect the candidate algorithms, parameter spaces, preprocessing, sampling, cross-validation, parallelism, MLflow trial representation, and problem-type inference in the pinned backend source.
2. Upload a controlled CSV and request task suggestions without starting a workflow. Record the proposed task, target, warnings, and elapsed time.
3. Start equivalent CSV and Trino runs at each declared data tier and record the effective engine, algorithm, input size, row count, pod, and Spark master.
4. Omit the optional Trino row limit only in a disposable bounded table and verify whether extraction is bounded before materialization.
5. Confirm the UI communicates the effective tier and rejects work outside the deployed capacity before accepting it.
6. Benchmark viable open-source options using the dimensions and dataset cases in ARCH-ML-001.

Source confirms task suggestion is profiling rather than execution. Supervised validation samples at most 10,000 rows, uses fixed parameter grids with single-worker GridSearchCV, and stores candidate attempts inside one MLflow validation run. AutoML estimates an uploaded Spark DataFrame and chooses small, large, or massive algorithm paths, but deployed Spark uses local[*] inside an API pod. Remote files are downloaded before Spark reads them, and Trino extraction can fetch rows before tier selection. The UI exposes row limit and train split, not a compute profile or the effective engine. Comparative engine benchmarks and representative tier runs remain unverified.

ML-S012 — Synthetic-data contract

1. Upload disposable CSVs covering empty data, missing target, high missingness, 200 MB and 500,000-row boundaries, and multipliers below, within, and above the backend range.
2. Generate supervised and unsupervised outputs and compare source rows, generated rows, deduplication, model choice, quality metrics, warning, and terminal status.
3. Force quality evaluation failure in a disposable environment and verify whether the workflow succeeds, warns, or fails.
4. Reload during generation, reopen from history, and attempt an authorized download without using an internal object-storage path.
5. Request cancellation during fitting and verify compute stops, no artifact is published, and cancelled state cannot later become completed.
6. Verify retention, source-data classification, access, and deletion of the upload, synthesizer, combined dataset, and quality report.

Source confirms a visible CSV-only generation and history flow. The backend accepts multipliers from 0.1 to 5, while the UI restricts entry to integers of at least 1 without displaying the maximum. The backend caps input at 200 MB and the combined row count at 500,000. The saved dataset shuffles original and generated rows together. Low quality and quality-evaluation failure are warning-only, despite stale README text claiming rejection. A backend download exists, but BFF/Oqullus expose only status, history, and the internal output path. The task does not observe cancellation before publishing output and final state. A representative live generation remains pending.

ML-S013 — EDA stage label

1. Open AutoML and start a controlled workflow, or inspect a retained workflow whose status contains the stage key eda.
2. Observe the corresponding item in Pipeline Stages while it is pending, running, and completed.
3. Confirm that Oqullus renders the item as Eda and does not expand the term in the progress stepper.
4. Inspect the Oqullus AutoML stage order and formatStageName call used by the stepper.

Source confirms that the pipeline order contains the machine-readable key eda. The shared formatter capitalizes only the first character of each word, so the visible label becomes Eda. The same source identifies the concept elsewhere as Exploratory Data Analysis (EDA), confirming that Eda is a presentation error rather than a different stage. A post-change deployed visual check remains pending.

ML-S014 — Failed-run diagnostics and reload recovery

Environment: realm gj76anadpsc, workspace w-t5e2kcxjd. Last observed: 8 September 2026.

1. Open AutoML and run a controlled workflow that reaches model validation but produces no valid candidate. Record only workflow and tracking identifiers.
2. Observe the failure panel, then reload /admin/automl.
3. Open Run History and attempt to reopen the failed workflow.
4. Compare the AutoML status, result, and error responses with the Oqullus endpoint map, failed-status branch, and Run History action state.
5. Inspect server logs only when the user-safe responses cannot explain the failure; do not expose raw tracebacks to end users.

For run dc103820, the status response reported failed, failed stage model_building, three completed stages, and no error or message. The result response returned HTTP 200 with only a failed status. The error response retained category internal_error and the safe message RuntimeError: No valid candidate models passed validation. Logs and MLflow correlated failed child run af8fbbc1bfc0458a9edfbde770862bd0 with the failed parent workflow; three earlier stage runs finished.

Source explains the reload behavior. Oqullus deletes the workspace-scoped active-run key as soon as polling observes failed but leaves the failure state in React memory, so it remains visible only until reload. BFF has routes for workflow status and result but not workflow error. Oqullus therefore never requests the stored reason, and Run History disables its result action for every non-completed run. The user's authenticated observation establishes the visible reload loss; the API, logs, and three source paths establish its cause.

ML-S015 — Optional-target mode semantics

1. Open Auto ML → Connect to Lake House and select a table containing at least two usable numeric feature columns.
2. Leave Target Column at None (Unsupervised) and start the workflow.
3. Record the submitted request, returned workflow stages, Labeling summary, Model Building problem type and candidates, and final generated columns.
4. Repeat with a valid categorical or numeric target and compare the request, stages, candidates, metrics, and result semantics.

Source confirms that Oqullus omits target when the field is empty and marks the run unsupervised. The backend interprets an absent target as clustering and evaluates K-Means, HDBSCAN, and Gaussian Mixture against numeric features using silhouette score. It does not infer a prediction target.

The backend re-audit corrected the earlier Labeling explanation. The fixed workflow retains Labeling when target is absent. Its deterministic YAML generator receives no schema-column list on the normal request path and emits the fallback rule 1 == 1 -> default_label. The rule cannot produce the expected row mask, is recorded as an internal rule error, and the agent still returns success with zero labels and unchanged data. Model validation later writes a separate clustering result containing cluster. Representative live comparison remains pending.

ML-S016 — AI and Machine Learning umbrella

1. Inspect the Oqullus navigation definition at source baseline f76e57dbb88dbbfdeb2b9c4d076ad382fb326ca7.
2. Confirm the section title and its visible child pages.
3. Classify each child as an ML lifecycle surface or a separate AI capability.
4. Compare the deployed navigation wording with the documentation umbrella.

Oqullus defines one Machine Learning section containing Experiments, Models, Auto ML, and AI Gateway. The first three are related ML lifecycle surfaces. AI Gateway routes external GenAI provider requests and is only a dependency of AutoML reasoning; it is not MLflow model serving. The existing section name is therefore narrower than its contents. AI/ML is the appropriate umbrella, while each child retains its precise capability name. Source: src/routes.ts.