Notebook execution
Notebook execution lets an authenticated Workspace user edit a notebook, start a personal Jupyter Server, run cells on a remote Spark kernel, inspect output, save the document, and manage compute lifecycle.
See Architecture for state and component boundaries, Live evidence for reproducible scenarios, and Notebook execution components for implementation references.
Assessment scope
| Field | Value |
|---|---|
| Actors | Authenticated notebook user, workspace administrator, and platform operator |
| Product validation | Not yet recorded |
| Components assessed | sparqd/oqullus, sparqd/bff-api, sparqd/workspace-file-management-backend, sparqd/platform-stacks, JupyterHub, Jupyter Server, Enterprise Gateway, Spark Operator, and Kubernetes |
| Assessed environment | Realm gj76anadpsc; workspace w-t5e2kcxjd; Kubernetes context dev-north-alicloud |
| Source baseline | Oqullus 363355f968fe; BFF API 93ea18e72414; Workspace file backend a6c5eb793e6b; platform-stacks a71390d12463 |
| Focused credential baseline | BFF API 0ac008a4c581; platform-stacks 0904a8417f05 |
| Last verified | General assessment: 4 September 2026; credential path: 8 September 2026; Maven cache and internal database reachability: 9 September 2026 |
| Tracking | platform-docs #39; component validation is not recorded |
This assessment covers server, session, kernel, connection, Spark runtime, cell execution, refresh, multiple tabs, kernel operations, cancellation, output, save, history, conflict, idle culling, gateway recovery, authorization, credential propagation, Maven dependency resolution, internal-service network reachability, observability, and performance boundaries.
The separate JupyterLab interface, local Python kernels, providers other than the reviewed AliCloud Enterprise Gateway path, and unauthorized production-data testing are excluded. This assessment did not execute destructive kernel or cluster fault injection and did not benchmark Spark.
Current capability boundary
| Capability | Source-backed state | Deployment evidence |
|---|---|---|
| Open and edit a notebook | Workspace restores tab and file metadata and loads the notebook document | Available in the assessed workspace; refresh can lose live runtime association |
| Start and stop the personal server | UI and BFF route JupyterHub server lifecycle | Deployed; detailed start, stop, failure, and unreachable states are not preserved end to end |
| Create or reuse a notebook session | UI lists Jupyter sessions and matches the notebook path before creating one | Source-verified; canonical identity and concurrent recovery remain open |
| Direct kernel path | Source can create a kernel without a persisted notebook session | Source-backed; it has no durable notebook-session identity and must not be presented as notebook recovery |
| Run Spark cells | Per-user Jupyter Server delegates to Enterprise Gateway, which launches a SparkApplication | Deployed AliCloud path; one approved Spark kernel type is configured |
| Resolve Maven packages | Spark Operator resolves requested coordinates before creating the driver | One PostgreSQL JDBC coordinate live-verified; controller-local cache operations remain conditional |
| Reach platform data services | Governed service interfaces are the intended path | A Spark driver could directly reach an internal PostgreSQL service; default-deny egress is not established |
| Interrupt execution | Cell cancellation invokes kernel interrupt with a bounded local fallback | Source-verified; deployed completion and retained-state evidence pending |
| Restart or shut down one kernel | Underlying lifecycle operations exist | Incomplete UI: visible toolbar controls are inert and the full Kernel menu is disabled |
| Render output | Basic stream, result, display, error, reply, and status messages are handled | Advanced message compatibility and output limits are not established |
| Save and recover document content | Code and output changes mark the notebook dirty; accepted saves create object versions and history | Happy paths exist; tab targeting, failure truthfulness, conflict detection, size, and retention are open |
| Recover after refresh or disconnect | Lazy exact-path session lookup runs when another execution needs a kernel | Deployed refresh state loss observed; deterministic reattachment is incomplete |
| Recover JupyterHub | Hub replacement preserved the personal server and SparkApplication and restored the server record | Oqullus falsely changed to stopped after two temporary 503s and required refresh; kernel-memory, proxy, and database checks remain |
| Recover Enterprise Gateway | Standalone availability persists remote-kernel connection metadata across gateway replacement | One-replica topology deployed; controlled replacement and client recovery pending |
| Release idle compute | JupyterHub and Enterprise Gateway use a shared 1,800-second idle policy | Current configuration source-verified; full idle and orphan-cleanup evidence pending |
Critical journeys
| Journey | Required result |
|---|---|
| Start and execute | Create one authorized server, session, kernel, and Spark runtime and correlate their identities end to end |
| Refresh and reconnect | Reattach to the same live kernel or explicitly report replacement and lost in-memory state |
| Invalid code and retry | Show the exception, return the same kernel to idle, and execute the corrected cell |
| Unauthorized access | Deny notebook, data, and kernel access without leaking identities, content, or credentials |
| Interrupt and cancel | Stop upstream work, preserve prior kernel state, and reach a truthful terminal cell state |
| Timeout and culling | Distinguish launch timeout, execution duration, idle expiry, and cleanup |
| Kernel or engine failure | Separate reconnecting, dead, restarted, and replaced states without automatically rerunning code |
| Large or partial output | Stay responsive, enforce limits, and label incomplete output truthfully |
| Multiple cells and tabs | Share one runtime state, queue deterministically, and prevent one tab from destroying another's work |
| Save and recover | Save the correct document once, retain errors, reject stale writes, and restore through history |
| Hub control-plane interruption | Keep existing servers and kernels alive, preserve healthy proxy routes, report unavailable controls, and reconcile state without duplicates |
| Gateway replacement | Permit a visible interruption, then reconnect to the same surviving remote kernel without duplicates or orphans |
| Package-bearing kernel start | Resolve approved coordinates through a bounded controller cache and surface repository or capacity failures |
| Internal data-service access | Permit governed APIs and approved external data targets while denying direct platform-database connections |
Findings
Use this table as the assessment register. Components identifies involved repositories and runtime systems, not people or teams. Each linked finding contains the observed case, expected condition, alternatives, component work, reproduction, closure conditions, and blockers.
| ID | Finding | Components | Evidence | Impact |
|---|---|---|---|---|
ARCH-NB-001 | Recover notebooks by canonical session identity | sparqd/oqullus, Jupyter Server | Source-verified | Medium |
ARCH-NB-002 | Model kernel and Spark readiness once per runtime | sparqd/oqullus, Jupyter Server, Enterprise Gateway, Spark | Deployed inconsistency + source | High |
GOV-NB-001 | Enforce and observe connected-idle kernel culling | sparqd/platform-stacks, JupyterHub, Enterprise Gateway, Spark | Source configuration + prior operational observation | High cost exposure |
PERF-NB-001 | Bound notebook save size, rate, and retention | sparqd/oqullus, sparqd/bff-api, sparqd/workspace-file-management-backend | Source-verified; scale pending | Medium |
PROD-NB-001 | Expose complete notebook kernel controls | sparqd/oqullus, Jupyter kernel APIs | Source-verified | Medium |
REL-NB-001 | Restore the active kernel after refresh | sparqd/oqullus, Jupyter runtime dependencies | Deployed observation + source | High |
REL-NB-002 | Define the supported Jupyter message contract | sparqd/oqullus, sparqd/bff-api, Jupyter protocol | Source-verified | Medium |
REL-NB-003 | Surface kernel death and state loss | sparqd/oqullus, sparqd/bff-api, Jupyter Server, Enterprise Gateway | Source-verified; controlled crash pending | High |
REL-NB-004 | Scope autosave state to each notebook | sparqd/oqullus | Source-verified | Critical |
REL-NB-005 | Keep failed saves visible and retryable | sparqd/oqullus, Workspace file API | Source-verified | Critical |
REL-NB-006 | Reject stale concurrent notebook saves | sparqd/workspace-file-management-backend, sparqd/oqullus, BFF API | Source-verified | High |
REL-NB-007 | Preserve authoritative user-server state | sparqd/bff-api, sparqd/oqullus, JupyterHub | Source-verified | High |
REL-NB-008 | Guarantee terminal state for every cell execution | sparqd/oqullus, sparqd/bff-api | Source-verified; fault injection pending | High |
REL-NB-009 | Validate standalone Enterprise Gateway recovery | sparqd/platform-stacks, Enterprise Gateway, sparqd/bff-api, sparqd/oqullus | Source + deployed topology; recovery pending | Operational boundary |
REL-NB-010 | Recover truthfully across JupyterHub control-plane interruption | sparqd/platform-stacks, JupyterHub, proxy, PostgreSQL, sparqd/bff-api, sparqd/oqullus | Live Hub replacement: runtime survived; Oqullus false-stopped and required refresh | Operational boundary + high client reliability impact |
REL-NB-011 | Operate bounded Maven dependency resolution | sparqd/platform-stacks, Spark Operator, Enterprise Gateway, Maven repositories | PostgreSQL JDBC coordinate and controller cache live-verified | High |
SEC-NB-001 | Verify identity and kernel ownership on WebSocket upgrade | sparqd/bff-api, JupyterHub, Jupyter Server | Source-verified; security validation pending | Critical |
SEC-NB-002 | Notebook code can read injected credentials through os.getenv() or printenv | sparqd/bff-api, sparqd/platform-stacks, JupyterHub, Enterprise Gateway, Keycloak, OpenBao | Credential propagation source-verified; safe deployed presence check pending | Critical |
SEC-NB-003 | Restrict notebook workloads from platform-owned databases | sparqd/platform-stacks, Kubernetes networking, JupyterHub, Enterprise Gateway, Spark workloads | Driver-to-internal-PostgreSQL TCP connection live-verified; JDBC result user-reported | High |
UX-NB-001 | Separate closing a view from shutting down compute | sparqd/oqullus, Jupyter Server | Source-verified | High |
UX-NB-002 | Make Save, Discard, and Cancel truthful | sparqd/oqullus | Source-verified | High |
Findings are observations, not accepted remediation commitments. Missing evidence remains open, and severity and disposition require review.
Implementation subissues
| Finding | Issue | Finding | Issue |
|---|---|---|---|
ARCH-NB-001, REL-NB-001 | oqullus #519 | ARCH-NB-002 | oqullus #520 |
SEC-NB-001 | bff-api #229 | REL-NB-007 | bff-api #228, then oqullus #516 |
REL-NB-002 | oqullus #521 | REL-NB-003, REL-NB-008 | oqullus #517 |
PROD-NB-001, UX-NB-001 | oqullus #522 | REL-NB-004, REL-NB-005, UX-NB-002 | oqullus #518 |
REL-NB-006 | workspace-file-management-backend #18, then oqullus #523 | GOV-NB-001 | project-management #1349 |
PERF-NB-001 | bff-api #230, workspace-file-management-backend #19, oqullus #524 | REL-NB-009 | Evidence only; open remediation only after failed gateway continuity or an approved availability objective |
REL-NB-010 | oqullus #516 for confirmed false-stopped recovery | REL-NB-010 proxy/database exercises | Evidence only until a failed exercise or approved availability objective |
REL-NB-011 | platform-stacks #692, implemented by platform-stacks #693 | SEC-NB-003 | Evidence recorded; remediation issue not opened |
Boundaries and interim workarounds
- Save notebook content before refresh and expect to rerun initialization cells until canonical runtime restoration is implemented.
- Do not interpret an open tab, connected channel, or visible kernel indicator as proof that the original kernel and namespace remain alive.
- Avoid simultaneous editing from multiple contexts; history can recover an overwrite only after it is discovered.
- Stay on the notebook that changed until save completes, and verify persistence after any network or service error.
- Avoid unbounded stream or rich output and clear unnecessary output before saving.
- Use cell interrupt for a running cell. Treat Stop Server as affecting every notebook on the personal server.
- Do not use inert restart controls. Hide them until their operation, scope, confirmation, and failure state are complete.
- Treat closing the final tab as potentially destructive until Close and Close and Shut Down are separate actions.
- Retain the current standalone Enterprise Gateway design for the expected workspace size unless a capacity measurement or availability objective requires replication. A controlled restart must still prove recovery.
- Retain the current single Hub and proxy while workspace capacity remains within a measured envelope. Existing servers are configured to survive Hub shutdown, but login, spawn, proxy, and database continuity still require a controlled exercise.
- Detect idle and orphan Spark workloads through server-side metrics and alerts, not browser polling.
- Treat the controller Ivy cache as bounded pod-local state. Monitor it and expect a cold cache after controller replacement.
- Use governed data services rather than direct platform-database endpoints from notebooks until default-deny egress is verified.
Assessment maturity
Scale: 0 absent, 1 ad hoc, 2 partly documented or automated, 3 routinely verified, 4 measured and improving.
| Lens | Score | Evidence-backed assessment |
|---|---|---|
| Product and UX | 1 | Core execution is available, but refresh, kernel controls, close semantics, and failure feedback remain incomplete. |
| Architecture and data | 2 | Routing and state authorities are mapped; canonical recovery and conflict-safe save identity are incomplete. |
| Security and privacy | 1 | HTTP identity routing exists; kernel-channel token and ownership enforcement needs security validation. |
| Infrastructure and operations | 2 | Idle policy and standalone gateway recovery are configured; replacement, cleanup, telemetry, and recovery are not exercised end to end. |
| Reliability and performance | 1 | Interrupt and save happy paths exist; terminal execution, output, concurrency, reconnect, and scale boundaries lack deployed proof. |
| Delivery, cost, and lifecycle | 1 | Source baselines and idle controls exist; capacity, retained storage, budgets, recovery objectives, and reviewer decisions remain open. |