Workflow Spark Recommendation
Spark Recommendation analyzes a completed Spark application and presents configuration and resource-sizing advice for its managed Workflow task. Its current and target contract is advisory only: it does not modify a Workflow, restart Spark, or apply settings to a running application.
Assessment scope
| Field | Value |
|---|---|
| Actors | Workflow viewer, workflow editor, platform operator, data-platform reviewer, FinOps reviewer |
| Parent capability | Workflow |
| Assessed environment | Realm gj76anadpsc; workspace w-t5e2kcxjd |
| Source baseline | Oqullus 8c9f240410b2; BFF 9b2fd45a4da1; workflow-backend d3d6a3ad7313; spark-recommendation 9576355935d4; platform-stacks a71390d12463 |
| Last verified | 3 September 2026 |
| Review status | Source assessment drafted; live deployment and component review not yet captured |
This assessment covers input selection, recommendation generation, Cogrion applicability, storage, retention, read APIs, authorization, cost, deployment, failure handling, and effectiveness measurement. It excludes automatic changes to Workflow definitions and automatic execution of recommendations.
Current capability boundary
| Capability | Current state |
|---|---|
| Generate advice | A scheduled Spark job selects the latest completed application per DAG/task and calls the AI gateway once per selected task |
| Show advice | Workflow list/detail surfaces show the latest task recommendation; the Workspace surface has no data wiring |
| Apply advice | No supported apply path; the modal only copies generated Python |
| Validate advice | Output format is parsed, but Spark keys, values, policy, and compatibility are not validated |
| Measure effectiveness | No baseline comparison, accepted/rejected state, subsequent-run comparison, or savings verification |
| Store advice | Delta table partitioned by recommendation date and queried through Trino |
| Retain advice | RETENTION_RUNS exists, but deletion is commented out and no VACUUM runs |
| Deployment | API and scheduled job are defined and enabled by default in current KCL; deployment/runtime evidence for the assessed realm remains to be captured |
Environment and surface status
| Path | Source/KCL condition | Assessment status |
|---|---|---|
| Alibaba Delta Spark | KCL supplies Alibaba History URL, Keycloak JWT for Trino, OSS access, Alibaba image tag, daily job, and API | Defined; live state in gj76anadpsc/w-t5e2kcxjd not yet captured |
| AWS Delta Spark | KCL supplies History URL, Basic-auth Trino, S3 access, daily job, and API | Defined; not live-tested in this assessment |
| Workflow list/detail | Requests latest advice by DAG name through BFF | Implemented; exact-run identity, validation, and role evidence remain open |
| Workspace | Button exists, but recommendation data is initialized to null | Incomplete; must not be treated as a working surface |
| Automatic apply | No implementation and excluded from the approved scope | Not applicable; advisory only |
What advice can mean in Cogrion
| Recommendation | Cogrion application point | Current presentation |
|---|---|---|
| Driver/executor cores and memory | Saved Spark Workflow task, before submission | Incorrectly rendered as runtime spark.conf.set(...) Python |
| Driver/executor disk and memory overhead | Saved Spark Workflow task, before submission | Not mapped back to the corresponding task fields |
| Fixed or dynamic executor counts | Saved Spark Workflow task and generated SparkApplication | Not mapped or validated against min/max rules |
| Driver/executor node type | One of the environment-approved task node types | Generic C/M/R/I family text is not mapped to allowed Cogrion values |
| SQL runtime settings such as shuffle partitions or AQE | Saved task spark_conf, or a supported pre-query runtime setting when Spark permits it | Workflow-backend supports spark_conf; current Oqullus Spark-task payload does not expose it |
Advice must identify whether a setting is a submission-time Workflow field, a Spark configuration override, or informational only. The UI must say that a change applies to a future run.
Storage and cost conclusion
Delta Lake can store the records correctly, but the current shape is not cost-effective as an interactive product-state store:
- a few rows per daily run create small date partitions and files;
- latest-by-DAG reads do not filter the date partition;
- the scheduled Spark job repeatedly scans history and launches Spark even when no new application exists;
- the same unchanged application can be sent to the LLM again; and
- two API replicas plus Trino serve simple latest-record lookups.
Object-storage bytes are unlikely to dominate cost. Spark startup, History Server calls, Trino scans, continuously running API replicas, and LLM tokens are the material boundaries.
Recommended target: keep current recommendation state and user disposition in
an operational database keyed by an exact task execution and configuration
fingerprint. Export immutable recommendation/effectiveness records to Delta
only when lakehouse analytics or audit retention justifies it. The lower-change
stabilization option is to keep Delta, remove high-cardinality date
partitioning, skip unchanged applications, implement retention plus VACUUM,
and bound all reads.
Findings
| ID | Finding | Components | Impact |
|---|---|---|---|
PROD-SR-001 | Map advisory output to future Workflow task configuration | Oqullus, workflow-backend, recommendation | High |
REL-SR-001 | Bind every recommendation to one exact execution and valid metrics | Recommendation, monitoring, Spark History Server | High |
PROD-SR-002 | Validate recommendations and measure their effectiveness | Recommendation, workflow-backend, Oqullus | High |
DATA-SR-001 | Implement logical and physical retention | Recommendation, Delta Lake, platform-stacks | High |
ARCH-SR-001 | Separate interactive recommendation state from analytical history | Recommendation, Delta Lake, Trino, BFF, platform-stacks | Medium |
SEC-SR-001 | Parameterize reads and authorize the managed Workflow | Recommendation API, BFF, workflow-backend | High |
Interim guidance
- Treat every recommendation as a hypothesis for the next Workflow run.
- Do not paste the generated Python to change driver/executor resources.
- Verify the suggested value against current task fields, approved node types, Spark image/version, and platform policy before changing a Workflow.
- Do not claim savings or performance improvement until a comparable subsequent run confirms it.
- Do not rely on
RETENTION_RUNS=7to delete data in the current implementation.