REL-NB-011 — Operate bounded Maven dependency resolution

Field	Value
Status	Mitigated; package resolution works, but capacity controls and alerting remain open
Architecture state	Deployed controller-local cache in gj76anadpsc / w-t5e2kcxjd
Severity	High
Affected surface	Remote Spark-kernel creation when the kernel requests Maven coordinates
Components	sparqd/platform-stacks, Spark Operator, Enterprise Gateway, Maven repositories, Spark driver
Associated issue	platform-stacks #692; implemented by platform-stacks #693

Where the problem appears

When a remote kernel requests packages through Spark's Maven package mechanism, the Spark Operator controller resolves those coordinates while reconciling the SparkApplication. This happens before a driver pod exists. A controller cache failure therefore prevents kernel creation and cannot be fixed by mounting only the Spark driver.

Concrete observed case

The controller ran with a read-only root filesystem while Ivy attempted to use /home/spark/.ivy2. The path had no writable volume, package resolution failed, and the driver was never created.

On 9 September 2026, Spark Operator Helm release revision 3 was deployed with a bounded emptyDir at /home/spark/.ivy2 and a separate bounded writable /tmp. The replacement controller was running with no restarts and one replica. A disposable SparkApplication requested org.postgresql:postgresql:42.7.7; the controller cache received the Ivy metadata and JAR, and the application reached RUNNING with ready driver and executor pods. The driver received the resolved JAR in its submission working paths, not in the immutable Spark image's /opt/spark/jars directory.

A JDBC read then succeeded according to the user conducting the notebook test. That result demonstrates that this package was available to the runtime; the database rows and credential values were not independently retained as assessment evidence.

Expected condition

Approved Maven coordinates resolve before driver creation without requiring a writable controller root filesystem. The cache has an explicit capacity bound, cache exhaustion produces an actionable launch error, and operators can observe usage, failures, and cold-cache latency. Controller replacement may make the next launch cold, but must not leave partial state that prevents later launches.

Cause and evidence

Dependency resolution is part of controller reconciliation for a SparkApplication that declares Maven packages. The controller therefore needs a writable Ivy home even though the resulting package is ultimately supplied to the Spark runtime.

The deployed mitigation provides:

• an 8 GiB emptyDir for /home/spark/.ivy2;
• a 1 GiB emptyDir for /tmp; and
• controller ephemeral-storage request and limit values of 2 GiB and 12 GiB.

The cache is local to the controller pod. It survives successive application and driver lifecycles while that pod remains alive, but a rollout, eviction, rescheduling, or node loss replaces it with an empty cache. If controller replicas are introduced, each replica has an independent cache and may download the same artifacts.

Why it matters

Package resolution is on the kernel-start critical path. A missing or full cache blocks new package-bearing Spark kernels before users receive a driver. Repeated trial-and-error with different coordinates can consume the bounded volume until the controller is replaced, while every cold cache increases startup time and repository traffic.

Existing controls and workarounds

The cache and temporary directory are bounded and do not require a writable root filesystem. Controller pod replacement clears accumulated content and is the current recovery path for a full or corrupt cache, but it also makes later launches cold and should be an operator action rather than routine cache management.

Users should select only approved package coordinates and versions. A successful import from a prebuilt image does not prove Maven resolution; use a coordinate that is not already in the immutable image and confirm the controller and driver evidence described in NB-S023.

Alternatives considered

1. Current bounded emptyDir: simple, fast while the controller lives, and automatically cleared on replacement; requires monitoring and accepts cold starts.
2. Persistent shared cache: improves survival across replacement, but adds concurrent-writer, corruption, retention, cleanup, and storage-cost responsibilities. Do not introduce it without measured cold-start need and an explicit ownership model.
3. Curated packages in approved Spark images: deterministic and offline at launch, but slower to change and unsuitable as the only mechanism once users may choose packages.
4. External artifact proxy: centralizes policy and upstream caching, but does not remove the need for a bounded local working directory.

Component involvement

• sparqd/platform-stacks owns the controller volumes, resource bounds, replica configuration, and operational telemetry.
• Spark Operator resolves declared Maven packages and creates the driver only after reconciliation succeeds.
• Enterprise Gateway supplies the approved package coordinates as part of the SparkApplication request.
• Maven repositories supply artifacts and require availability, integrity, and allow-list policy appropriate to user-controlled code.
• The Spark driver and executors consume the resolved dependency but do not own the controller cache.

Reproduction and verification

Follow NB-S023.

Resolution conditions

Close the remaining operational work when cache bytes and inode usage, resolution failures, repository latency, and cold starts are observable; an alert and runbook cover near-full and full-cache behavior; cleanup or replacement is safe and documented; multiple approved coordinates and failure recovery are tested; and any future replica or persistent-cache design has explicit concurrency and lifecycle semantics.

Validation and blockers

One PostgreSQL JDBC coordinate, controller cache population, driver delivery, and a successful application launch are live-verified. Cache-full behavior, automatic cleanup, alerts, repository outage recovery, multiple package sets, and multi-replica behavior remain unverified.