PERF-WF-002 — Stop re-downloading growing Spark logs through Airflow

Field	Value
Status	Open; source-verified, deployed volume and failure evidence pending
Architecture state	Deployed behavior
Severity	High
Affected surface	Spark task execution, live logs, Airflow task pods, BFF traffic, browser memory, log retention, and streaming workloads
Components	sparqd/oqullus, sparqd/bff-api, sparqd/workflow-backend, Airflow, Spark, sparqd/platform-stacks, Loki, Alloy
Associated issue	Not opened

Where the problem appears

Run a Spark task long enough to produce a steadily growing driver log. Keep its Workflow Log tab open and measure each response size, cumulative bytes, browser memory, Airflow task-pod state, remote-log availability, and behavior after pod loss.

Concrete observed case

Generated Spark tasks enable Airflow operator log streaming. The operator task therefore remains active while it follows Spark output. Oqullus requests the task log every five seconds and replaces the displayed log with the response. The request sends neither Airflow's continuation token nor full_content=true.

In the Airflow 2.10 and 2.11 file-log path, full_content=false does not bound the underlying read. The handler reads complete available remote, local, executor, and served logs into strings, splits and interleaves their lines, builds one combined log, and only then removes characters before a supplied log_pos. On the first request, or every request that omits the returned token, there is no log_pos, so the API returns the complete currently available log. Oqullus therefore makes Airflow rebuild and return that growing log every five seconds.

A streaming task with 1 GiB of accumulated output can consequently make one Log page opening read, combine, serialize, proxy, transfer, and render approximately that whole log. While the page remains open, each poll can repeat the operation against a still-growing file. This explains the observed Airflow 2.10 behavior where opening a large log remained buffering for about ten minutes; deployed timing and byte capture are still required for gj76anadpsc.

The assessed Airflow stack enables remote OSS or S3 logging. Airflow's object storage handlers upload the local task log when the handler closes, so this is not a reliable independent live-log path for a task that remains active indefinitely. Loss of the Airflow task pod can also remove the current local tail before close/upload behavior completes.

Expected condition

Separate orchestration logs from workload logs:

• Airflow records Spark submission, immutable application identity, state transitions, and bounded terminal diagnostics.
• A central log system receives live Spark driver and executor logs directly from the cluster.
• The UI queries logs by exact immutable workspace, workflow, run, task attempt, Spark application, pod, and container identity.
• The first read is a bounded recent tail. Subsequent reads use a timestamp or cursor, older logs load on demand, auto-refresh can be paused, and the UI reports truncation, gaps, retention, and collection failure.
• Spark event logs and the Spark History Server serve completed execution analysis; text logs remain in the log system under an explicit retention policy.
• Long-running and streaming Spark applications do not require an Airflow task to remain alive for the application's full service lifetime.

After the alternative is validated, long-running Spark submission should stop following all application logs through the Airflow operator. The current Loki deployment already offers a destination and 14-day retention, but its collection labels, tenancy, completeness, and query contract still require verification. Replace Promtail with Grafana Alloy rather than extending it; Promtail reached end of life on 2 March 2026.

Cause and evidence

• Workflow DAG generation enables Spark operator log following.
• Oqullus polls log content every five seconds and overwrites the displayed value; no Airflow continuation token is sent.
• Airflow 2.10 and 2.11 FileTaskHandler reads each available log source in full, interleaves the complete line sets, calculates log_pos, and applies a previous offset only afterward.
• Without a token, each Oqullus poll is treated as an initial read and returns the complete currently available log. full_content=false does not impose a byte or line limit on this handler.
• BFF proxies the Airflow log request but adds no cursor or bounded-tail contract.
• Airflow's OSS and S3 task handlers upload on close.
• The assessed observability stack deploys Loki with 14-day retention and a cluster log collector with workflow, run, task, Spark application, pod, and container label inputs, but the current UI does not use it for this journey.

Why it matters

Long Spark runs repeatedly force a full upstream read and, because Oqullus does not return the continuation token, a full response. Line splitting, interleaving, JSON serialization, BFF proxying, browser buffering, and rendering can amplify memory and latency beyond the raw file size. A 1 GiB task log can therefore make the page unusable and repeat substantial work every five seconds. Very long or streaming runs also couple workload lifetime to an Airflow task pod and can leave live logs unavailable when that pod is evicted or the remote handler has not closed.

Existing controls and workarounds

Close the Log tab or navigate away when continuous viewing is unnecessary and use cluster logging for operator diagnosis where available. Keep application logs bounded at their source. Do not treat remote Airflow object logs as a guaranteed live stream for an active task.

Alternatives considered

1. Recommended: Loki-backed bounded and incremental live tail, Spark event logs for execution analysis, and short bounded Airflow orchestration logs.
2. Pass Airflow's continuation token correctly — reduces repeated response bytes, but Airflow 2.10/2.11 still reads and reconstructs the complete available file before slicing, so it does not solve backend I/O, CPU, memory, or long-lived workload-log ownership.
3. Add a genuinely bounded byte-range or tail endpoint backed by storage that supports ranged reads — reduces transfer and backend work but still leaves Airflow responsible for workload logs unless the storage boundary moves.
4. Lower the five-second polling frequency — reduces requests but still repeatedly transfers growing content and delays visibility.
5. Push log updates over WebSocket — transport can improve, but it still needs durable central storage, bounded replay, reconnection cursors, and retention.

Component involvement

• sparqd/workflow-backend defines Spark submission and the application identity returned to product surfaces.
• Airflow owns finite orchestration state and bounded orchestration diagnostics, not the complete lifetime log store for long-running Spark applications.
• sparqd/platform-stacks wires Spark logs to Loki through Alloy, defines retention, and ensures immutable correlation labels.
• sparqd/bff-api and sparqd/oqullus expose an authorized, bounded, incremental log contract with pause, pagination, gap, and error states.
• Spark event logging and History Server provide completed-execution detail.

Reproduction and verification

Follow WF-S023.

Resolution conditions

Close when long-running Spark logs remain available after Airflow task-pod loss, refresh transfers only bounded new data, browser memory remains bounded, exact task attempts cannot cross identities, retention is visible, and Airflow tasks are not kept alive solely to relay streaming application logs.

Validation and blockers

The current Oqullus request and Airflow 2.10/2.11 read paths are source-verified; the ten-minute buffering case is user-observed. Remaining work requires a representative long-running run in gj76anadpsc, response-byte and memory measurements, forced Airflow task-pod loss, Loki label/access validation, and a Promtail-to-Alloy migration plan.

Reference behavior: Airflow 2.11 file-log reader, Airflow 2.11 log endpoint, Airflow large-log pagination issue, Airflow OSS task handler, and Promtail deprecation and end of life.