Notebook execution architecture
Architecture status
| State | Meaning in this assessment |
|---|---|
| Deployed | Present in gj76anadpsc / w-t5e2kcxjd and observed or topology-checked |
| Source-backed | Present in reviewed source but not exercised end to end in the assessed workspace |
| Expected contract | Required behavior used to evaluate the current feature |
| Proposal | Future design that is not part of the current support boundary |
The deployed Spark path uses a custom Workspace UI and BFF in front of a per-user Jupyter Server. Jupyter Server delegates remote-kernel lifecycle to Enterprise Gateway, which creates a SparkApplication in Kubernetes.
The separate JupyterHub and Enterprise Gateway orchestration proposal is not current behavior.
System context
Notebook documents and live computation are separate state domains. Saving the document preserves cells and outputs, not Python variables, the Spark session, pending executions, or the kernel process.
Implementation references and source baselines are centralized in Notebook execution components.
Component responsibilities
| Component | Canonical responsibility |
|---|---|
| Oqullus | Notebook presentation, local editing, execution state, controls, save state, and recovery feedback |
| BFF API | Authenticated browser boundary for Jupyter and Workspace file operations |
| Workspace file service | Notebook object, current version, history, diff, and revert |
| JupyterHub | Authenticated personal Jupyter Server lifecycle and routing |
| Jupyter Server | Notebook sessions and their kernel references |
| Enterprise Gateway | Remote-kernel lifecycle and persisted reconnection metadata |
| Spark Operator and Kubernetes | SparkApplication, driver, executor, storage, and scheduling lifecycle |
| Platform stacks | Deployment composition, compute profiles, persistence, culling, and runtime policy |
Start and execute flow
The expected contract creates at most one authorized session and kernel for the canonical notebook identity. A cell reaches completed only after its matching execution reply and idle state. Any lost or failed branch must instead reach a visible failed, cancelled, or execution-lost state.
Dependency resolution lifecycle
When a kernel profile requests Maven coordinates, the Spark Operator controller
resolves them before driver creation. Its bounded Ivy cache is reusable across
applications handled by the same controller pod, not a durable platform cache:
controller replacement makes the next resolution cold, and each controller
replica would have independent contents. Capacity, repository, or cache-write
failures must be reported as launch failures because no driver may exist yet.
See REL-NB-011.
State and authority
| Domain | Authority | Required states | Persistence boundary |
|---|---|---|---|
| Personal server | JupyterHub | checking, stopped, spawning, running, stopping, failed, unreachable | Hub database and current deployment state |
| Notebook session | Jupyter Server | absent, starting, ready, deleting, failed | Lives with the personal server and links notebook identity to a kernel |
| Kernel | Jupyter Server and Enterprise Gateway | starting, idle, busy, restarting, dead | Live process plus gateway recovery metadata |
| Connection | Client and BFF channel | connected, reconnecting, disconnected | Recreated after page or transport loss |
| Spark runtime | Spark Operator | provisioning, initializing, ready, failed, terminating | SparkApplication and driver/executor resources |
| Cell execution | Oqullus, reconciled with Jupyter messages | queued, running, completed, failed, cancelled, execution-lost | Saved output after successful notebook persistence |
| Notebook document | Workspace file service | clean, dirty, saving, saved, save-failed, conflict | Current object version and history |
These domains must not be collapsed. A failed status request does not prove the server stopped; a disconnected channel does not prove the kernel died; a live kernel does not prove Spark is ready; and saved output does not preserve kernel variables.
Session identity and refresh recovery
The expected recovery sequence is:
- Restore the notebook's canonical server-side identity.
- Resolve and authorize its live Jupyter session during page restoration.
- Reconnect to the same kernel and reconcile current state before execution.
- Create a new session only when no authorized live session exists.
- Report expiry, restart, or replacement and the resulting loss of in-memory state.
Current recovery is later and path-dependent: browser persistence restores the
tab and document metadata, while a session lookup occurs only when another cell
needs a kernel. Detailed findings are
REL-NB-001 and
ARCH-NB-001.
One runtime can serve multiple cells and tabs. Kernel and Spark readiness
therefore belong to a shared session/kernel record, while each cell retains only
its own queue and execution outcome. See
ARCH-NB-002.
Failure and recovery contract
- Transport recovery may reconnect automatically to the same authorized kernel.
- Kernel restart or replacement must always disclose loss of the in-memory namespace.
- A lost cell is never automatically rerun because it may perform writes or external side effects.
- Partial output is retained and labelled incomplete.
- Queue, kernel preparation, Spark initialization, execution, and cancellation each have a bounded terminal outcome and an available cancel path where safe.
- Server absence, transition, failure, and unreachability remain separate.
The detailed protocol and recovery findings are
REL-NB-002,
REL-NB-003,
REL-NB-007, and
REL-NB-008.
Expected save and version lifecycle
The following sequence is the target contract. The reviewed current source path uses global tab save state, performs unconditional updates, and has no established end-to-end payload or retention limits; those differences remain open findings.
Dirty state, in-flight save, failure, and current version belong to each notebook, not to the active tab globally. Save and Close, Discard and Close, and Cancel must be atomic and truthful. View closure remains separate from compute shutdown.
See REL-NB-004,
REL-NB-005,
REL-NB-006,
UX-NB-001,
UX-NB-002, and
PERF-NB-001.
Compute lifecycle controls
Users need intent-level operations—Start Server, Stop Server, Interrupt, Restart, Restart and Run All, and Shut Down—not raw Jupyter object management. Every action must identify its scope, warn about state loss, and report a terminal result. Change Kernel is meaningful only when multiple compatible, policy-approved kernels are available.
Closing a notebook view must not implicitly destroy its compute session. An
explicit shutdown can terminate it immediately; otherwise server-side idle
policy governs cleanup. See
PROD-NB-001 and
UX-NB-001.
Identity and data access
The BFF is a privileged boundary because it resolves the authenticated user and connects to that user's Jupyter Server through platform credentials. HTTP and kernel-channel paths must enforce equivalent token validation, workspace binding, server ownership, and kernel ownership before upstream access.
Kernel launch authorization is separate from dataset authorization. Spark data access depends on the propagated user/workspace identity, service account, storage credentials, metastore, and policy services. A successful kernel launch does not prove access to any dataset.
Network reachability is also separate from data authorization. The current
deployment allowed a Spark driver to connect to an internal platform PostgreSQL
service. The expected boundary denies direct platform-database access from
user-controlled servers, drivers, and executors while explicitly allowing DNS,
driver/executor communication, object storage, governed services such as Trino
and the Hive Metastore API, and approved external data targets. Network policy
and database authorization are both required. See
SEC-NB-003.
Credential propagation is a separate trust boundary. The current source path
exchanges the browser token for the workspace-specific JupyterHub audience,
passes it in Hub spawn options, and injects the resulting bearer token plus an
OpenBao token into the personal server. The single-user environment also holds
the confidential JupyterHub OAuth client secret. Matching Jupyter client and
Enterprise Gateway allow-lists permit those names in remote-kernel launch
environments. Once present there, normal notebook code can retrieve the raw
values with os.environ or os.getenv(), and subprocesses can read them through
the inherited environment.
The expected boundary keeps confidential client secrets, refresh tokens, shared control-plane credentials, and reusable secret-management tokens out of both personal servers and kernels. When direct user delegation is unavoidable, it must be short-lived, single-audience, workspace- and user-bound, minimally scoped, revocable, and redacted from outputs and logs. Prefer a trusted broker for token exchange and secret access.
See SEC-NB-001 and
SEC-NB-002.
Deployment, availability, and capacity
The assessed deployment uses one Hub, one proxy, one JupyterHub database, and
one Enterprise Gateway. The deployed Hub configuration preserves personal
servers when the Hub shuts down. A controlled Hub replacement on 4 September
2026 preserved the personal-server pod and SparkApplication and restored the
server record without duplicates. Oqullus nevertheless interpreted two
temporary status 503s as a stopped server, cleared local kernel state, stopped
polling, and required a page refresh to reconcile. Kernel-memory preservation,
existing-route behavior during interruption, proxy interruption, and database
recovery remain unverified; see
REL-NB-010.
A personal Jupyter Server pod is created per active user, but heavy Spark compute runs remotely as a driver and dynamically allocated executors.
Single-user pods are therefore a secondary capacity concern, not the principal compute unit. They still consume one pod and persistent volume per user and carry notebook sessions, files, outputs, and kernel channels. On 4 September 2026 the workspace had no active single-user pod but retained six user volumes totalling 120Gi.
Every active Spark kernel retains a driver and at least one executor until it is stopped or culled:
| Profile | Driver | Executor range |
|---|---|---|
| Small | 1 core / 6Gi | 1–2 executors at 2 cores / 10Gi each |
| Medium | 2 cores / 10Gi | 1–4 executors at 3 cores / 22Gi each |
| Large | 4 cores / 16Gi | 1–10 executors at 6 cores / 40Gi each |
These values exclude runtime memory overhead and storage. For the current workspace-isolated model, no finding claims that the topology is insufficient. Capacity should be assessed from concurrent server starts, active kernels, kernel-launch rate, channels, and Spark demand—not registered-user count. Vertical sizing is the first response to a measured gateway throughput limit.
Enterprise Gateway uses standalone availability with persisted kernel-session
metadata. A replacement attempts to reconnect to a still-running remote kernel,
but gateway requests and channels can be interrupted while the sole instance is
replaced. Replication mode is a future alternative only if measured capacity or
an explicit availability objective requires it. See
REL-NB-009.
JupyterHub and Enterprise Gateway share a 1,800-second idle policy. Connected
but inactive kernels are eligible, with a server-side check approximately every
300 seconds. The browser reconciles after load, reconnect, or channel closure;
it does not own idle cleanup. See
GOV-NB-001.
Observability boundary
Evidence must correlate one user action across browser request, notebook, session, kernel, BFF channel, Enterprise Gateway, SparkApplication, driver, and executor identities without recording notebook content or credentials.
Operations need counts and durations for server spawn, session/kernel creation, Spark readiness, cell execution, reconnect, restart, cull, failure, and orphan cleanup. User-visible status must include the last observation time and explain which authority is unavailable. No reviewed evidence establishes a complete correlation ID, notebook-execution SLO, orphan alert, restart alert, or recovery objective.
Architecture questions still open
- What concurrent active-server, kernel, channel, and launch-rate envelope is supported for one workspace?
- What interruption and recovery objective applies to standalone gateway replacement?
- Which Jupyter message families and output sizes are supported?
- What notebook size, save rate, history retention, and object-version limits are enforced?
- Which roles may start compute, access each notebook, operate another session, and query each dataset?
- Which metrics, alerts, runbooks, and recovery exercises constitute operational readiness?