Workflow Spark Recommendation evidence

Evidence summary

Observation	Evidence state
Latest completed Spark application is selected per DAG/task	Source-verified at 9576355935d4
Monitoring enrichment selects a separate latest row per DAG/task	Source-verified at 9576355935d4
Several metric names do not match their Spark source semantics	Source-verified at 9576355935d4
One LLM call is made per selected task with no processed-input checkpoint	Source-verified at 9576355935d4
LLM output receives formatting checks but no key/value or policy validation	Source-verified at 9576355935d4
Oqullus converts all suggested settings to runtime spark.conf.set(...) code	Source-verified at 8c9f240410b2
Workflow-backend models submission-time resource fields and spark_conf	Source-verified at d3d6a3ad7313
Oqullus Spark-task serialization does not send spark_conf	Source-verified at 8c9f240410b2
Workspace recommendation data remains null	Source-verified at 8c9f240410b2
Retention delete is commented out; no VACUUM exists	Source-verified at 9576355935d4
Delta is partitioned by recommendation date and latest reads do not filter that partition	Source-verified at 9576355935d4
API interpolates dagName into Trino SQL	Source-verified at 9576355935d4
BFF bulk retrieval fans out one API request per DAG	Source-verified at 9b2fd45a4da1
KCL defines a daily job and two API replicas	Source-verified at a71390d12463
Current pods, schedule, rows, role behavior, latency, and cost in the assessed realm	Live evidence pending

SR-S001 — Applicability to a Workflow task

1. Create a managed Workflow with one Spark task and record its saved driver, executor, disk, node-type, scaling, and spark_conf values.
2. Run it to completion and open its recommendation.
3. For every suggested key, classify it as submission-time Workflow field, supported pre-query runtime setting, or informational only.
4. Compare the displayed/copyable application method with that classification.
5. Confirm submission-time settings are not presented as effective runtime spark.conf.set(...) calls.
6. Confirm every mapped value passes workflow-backend schema and environment policy validation before being described as applicable.

SR-S002 — Exact execution correlation

1. Run the same Spark task twice with different executor sizes, then retry one run so three distinct Spark applications exist.
2. Capture Workflow UUID/revision, run ID, task attempt, Spark application ID, Spark configuration, and monitoring observation IDs for each execution.
3. Run recommendation generation after each execution.
4. Confirm each recommendation references one exact execution and never joins the latest History application to monitoring data from another run.
5. Rename or revise the Workflow and confirm identity remains attached to the original revision without fuzzy DAG-name matching.

SR-S003 — Metric and recommendation validity

1. For a completed task, capture raw History Server executor/job/stage payloads and the complete Prometheus series over its execution window.
2. Recalculate CPU time/utilization, current and peak memory, executor count over time, shuffle, spill, GC, duration, and allocation.
3. Compare those values and units with the recommendation input.
4. Remove or fail one input source and verify generation becomes incomplete, not silently advisory.
5. Inject unknown keys, invalid quantities, static-at-runtime settings, and disallowed node types into a model response. Confirm they are rejected or explicitly marked unsupported before reaching the UI.

SR-S004 — Effectiveness experiment

1. Select repeatable workloads with stable input data and at least three comparable baseline runs.
2. Record runtime, executor CPU/memory distribution, shuffle/spill, failures, retries, infrastructure cost, and output correctness.
3. Apply one reviewed recommendation to a new Workflow revision.
4. Run at least three comparable trials and compare median plus variability against the baseline.
5. Record accepted/rejected, actual field changes, reviewer, expected outcome, observed outcome, regression, rollback, and confidence.
6. Confirm the product does not label advice effective from generation alone.

SR-S005 — Retention and physical cleanup

1. Generate more than RETENTION_RUNS distinct recommendations for one task, including multiple executions on the same day.
2. Query logical rows, Delta table history, active files, obsolete files, and object-store bytes before retention.
3. Run the current retention function and confirm it only reports skipped rows.
4. After an approved fix, confirm only the configured logical history remains.
5. Run VACUUM first as a dry run and then under the approved safety window.
6. Confirm physical bytes are reclaimed and document the resulting time-travel boundary and recovery procedure.

SR-S006 — Storage and cost boundary

1. Measure a no-new-application schedule and schedules with 1, 10, 100, and 1,000 eligible tasks.
2. Capture driver/executor runtime and node allocation, History request count, Trino scanned bytes, LLM calls/tokens/cost, Delta file count/size, and total job duration.
3. Repeat the unchanged workload and confirm already-processed fingerprints are skipped in the target design.
4. Measure latest-list and task-detail reads for growing retained history.
5. Compare current Delta/Trino serving with an operational latest-state lookup and optional append-only Delta archive.
6. Record the volume threshold at which archival analytics justify Delta and the file-count threshold that justifies compaction.

SR-S007 — Authorization and query safety

1. Use two Workflows with disjoint viewers and recommendations.
2. Request both through list and task-detail routes using names and immutable IDs; verify denial occurs before the recommendation query.
3. Submit DAG names containing quotes, SQL syntax, Unicode, hyphens, and underscores; confirm parameterized queries preserve identity without SQL interpretation or normalization collisions.
4. Call the internal API without the authenticated BFF identity and confirm denial.
5. Verify prompts and logs exclude credentials, secrets, and unapproved Spark properties, and record AI-gateway retention/data-residency behavior.

SR-S008 — Failure and recovery

1. Fail History Server, monitoring table, AI gateway, Delta write, Trino read, and one BFF fan-out request independently.
2. Confirm the UI distinguishes unavailable, incomplete, invalid, stale, no recommendation, and unsupported advice.
3. Retry a transient generation failure and confirm one idempotent record is produced without duplicate LLM billing.
4. Recover each dependency and confirm current advice returns without browser remount while preserving its generation and input timestamps.

Evidence still required

• Rendered and deployed resources in gj76anadpsc, workspace w-t5e2kcxjd.
• Current schedule, last job result, API readiness, and BFF endpoint evidence.
• Real table schema, partitions, file counts, bytes, history, and retained rows.
• At least one exact-run correlation capture and one deliberately mismatched latest-run case.
• Applicability validation against a real managed Spark Workflow task.
• Recommendation-quality experiments and reviewer decisions.
• Role denial, query-safety, partial failure, retry, and recovery evidence.

Missing live evidence remains open; it does not negate the source-confirmed findings above.