Notebook execution live evidence

Evidence scope

Item	Value
Realm	gj76anadpsc
Workspace	w-t5e2kcxjd
Kubernetes context	dev-north-alicloud
Actor	Authenticated assessment participant; exact role must be captured per run
Source baseline	Oqullus 363355f968fe; BFF API 93ea18e72414; Workspace file backend a6c5eb793e6b; platform-stacks a71390d12463
Focused continuity source check	Oqullus f76e57dbb88d; deployed BFF image public.ecr.aws/quantdata/sparqd/bff-api:9b2fd45
Focused credential source check	8 September 2026; BFF API 0ac008a4c581; platform-stacks 0904a8417f05
Last general source and deployment-topology check	4 September 2026
Existing live observation	Refresh lost active notebook kernel state on 2 September 2026

Use disposable notebooks and synthetic values. Do not retain secrets, tokens, notebook contents, or unauthorized data. Record timestamps, HTTP status, session and kernel IDs, SparkApplication identity, and relevant pod identity without publishing user identifiers.

Source evidence summary

Observation	Evidence state
Workspace restores tab and file metadata but not live session, kernel, connection, or execution state	Source-verified; refresh state loss observed
Recovery is deferred until execution and matches a Jupyter session by exact notebook path	Source-verified
Kernel and Spark startup are represented on individual cells rather than one shared runtime	Source-verified; inconsistency observed
The client implements a bounded subset of Jupyter messages	Source-verified; protocol matrix pending
Unexpected channel closure does not produce a complete kernel or cell failure state	Source-verified; controlled crash pending
User-server status does not preserve stop direction, failure, or unreachable state	Source-verified; focused source tests cover current behavior
Queue, initialization, send, and message-loss branches can remain non-terminal	Source-verified; controlled injection pending
Kernel toolbar controls are visible but incomplete while broader server stop is exposed	Source-verified; live capture pending
Closing the last same-path tab deletes its Jupyter session	Source-verified; live capture pending
Connected-idle culling is configured for 1,800 seconds with a periodic server-side check	Source-verified; full deployed idle test pending
Standalone Enterprise Gateway persists recovery metadata but provides no uninterrupted second serving instance	Source- and deployment-topology-verified; recovery pending
Hub shutdown preserves the personal server, but Oqullus maps two temporary status failures to stopped and stops reconciliation	Hub replacement live-verified; kernel-memory, proxy, and database checks pending
Save state is global across tabs and later resolves the active tab as target	Source-verified
Failed saves can resolve as success and clear dirty state	Source-verified
Close-dialog Save and Discard paths do not implement their labels reliably	Source-verified
Concurrent saves have no server-enforced expected-version condition	Source-verified
Autosave can repeatedly upload and version the complete notebook and outputs	Source-verified; scale boundary pending
Notebook code can read access, OpenBao, and OAuth client credentials from the allow-listed kernel environment through os.getenv() or printenv	Source-verified propagation; deployed presence and effective authority pending safe validation

Reproduction scenarios

NB-S001 — Start server and execute

Status: Source path reviewed; live identity correlation pending.

1. Sign in to realm gj76anadpsc and select workspace w-t5e2kcxjd.
2. Start the personal server and open a disposable notebook.
3. Run 1 + 1.
4. Record the server, notebook session, kernel, SparkApplication, driver, and executor identities and their creation timestamps.
5. Confirm the output is 2 and only one authorized runtime was created.

Expected: one traceable server/session/kernel/Spark chain reaches ready and the cell reaches completed.

NB-S002 — Refresh and continue

Status: Refresh state loss observed; exact deployed branch not yet captured.

1. In a disposable notebook run:

   import os
   import uuid
   
   refresh_probe = str(uuid.uuid4())
   (refresh_probe, os.getpid())

2. Record the matching session ID, kernel ID, notebook path, process ID, and SparkApplication identity.

3. Refresh without closing the notebook or stopping its server.

4. Capture the session list again before running another cell.

5. Run (refresh_probe, os.getpid()) and inspect the old runtime.

Expected interpretation:

• unchanged value, process, session, and kernel means the UI reattached;
• the same session/kernel with missing state indicates a kernel restart or remote reconnection problem;
• changed identity means a replacement and must be disclosed;
• simultaneous old and new kernels mean recovery leaked an orphan;
• no prior session requires classification as culling, shutdown, path mismatch, or another lifecycle action.

NB-S003 — Multiple tabs and session identity

Status: Source suggests same-path sharing; deployed behavior pending.

1. Open the same disposable notebook in two tabs.
2. Run distinct stateful cells from both tabs.
3. Refresh one tab and compare session/kernel identity and state.
4. Close each tab separately and record session and Spark lifecycle.
5. Repeat while opening both tabs concurrently from a cold state.

Expected: both tabs bind predictably to one authorized session, one close does not terminate the other's work, and concurrent restoration creates no duplicate kernel.

NB-S004 — Invalid code and retry

Status: Not yet exercised.

1. Run raise RuntimeError("assessment probe").
2. Record error output and kernel status.
3. Run 2 + 2 without restarting.
4. Confirm the session and kernel IDs did not change.

Expected: the first cell fails visibly, the kernel returns idle, and the second cell completes with 4 in the same kernel.

NB-S005 — Authorization and data denial

Status: Controlled actor and resource required; security review pending.

1. Use an approved restricted test actor and disposable denied resource.
2. Attempt notebook read/write and a denied data operation.
3. Verify direct HTTP access is denied at the same boundary as the UI action.
4. Review the kernel-channel upgrade path without retaining or forging tokens in the deployed workspace.
5. Capture safe audit identities and confirm another user's kernel cannot be addressed.

Expected: every denied action fails closed without exposing notebook names, data, credentials, session IDs, or kernel access, and the decision is auditable.

NB-S006 — Interrupt and continue

Status: Not yet exercised.

1. Define a unique variable.
2. Run import time; time.sleep(120).
3. Interrupt after the running state appears.
4. Execute the prior variable and record state transitions and identities.

Expected: upstream work stops promptly, the cell becomes cancelled, the same kernel returns idle, and prior variables remain usable.

NB-S007 — Execution duration and idle culling

Status: Source policy verified; controlled runtime and idle window required. The reviewed configuration supplies a 300-second chart launch value and a 180-second kernel launch timeout; their effective precedence and user-visible behavior are not yet confirmed.

1. Run a safe bounded cell long enough to distinguish execution duration from kernel launch timeout.
2. Leave a separate disposable kernel connected but inactive beyond 1,800 seconds, including once after closing the browser tab.
3. Record server-side idle observations and cleanup time.
4. Confirm the kernel, SparkApplication, driver, executors, and transient storage are removed as intended.
5. Reopen the notebook and record expiry feedback.

Expected: duration and idleness have distinct states; active work is retained; idle compute is culled within the agreed tolerance; no orphan remains.

NB-S008 — Kernel or Spark driver failure

Status: Controlled fault injection required; source risk verified.

1. Start a disposable kernel, define a unique variable, and begin a bounded execution.
2. Terminate either the kernel or its Spark driver through an approved operator action.
3. Record connection, kernel, Spark, and cell state plus retry timing.
4. Attempt recovery and compare old/new identities.
5. Confirm the interrupted cell was not automatically resubmitted.

Expected: reconnecting is distinct from dead; partial output is labelled; state loss is persistent and explicit; recovery is bounded and creates no orphan.

NB-S009 — Large and partial output

Status: Limits and supported message types are not established.

1. Emit bounded increasing stream output while measuring browser memory and response time.
2. Repeat with a bounded rich-display payload.
3. End one probe with an exception after partial output.
4. Exercise supported clear/update-display behavior and one unsupported message type in a controlled test.
5. Record render, save, and recovery behavior at each size.

Expected: explicit limits prevent page failure; partial output remains labelled; unsupported messages cannot corrupt another cell; notebook saving stays bounded.

NB-S010 — Network and BFF reconnect

Status: Not yet exercised.

1. Define a unique variable in a disposable kernel.
2. Briefly disconnect the browser network while idle, then while running.
3. Repeat across an approved BFF restart.
4. Record reconnect attempts, channel count, duplicate messages, cell outcome, and kernel identity.
5. Run the variable after recovery.

Expected: transient transport returns to the same authorized kernel, messages are not duplicated, and unrecoverable state is explicit and terminal.

NB-S011 — Standalone gateway recovery

Status: Source and deployed one-replica topology verified; controlled recovery pending.

1. Create a disposable variable and record session, kernel, SparkApplication, and Enterprise Gateway instance identities.
2. Perform an approved replacement of Enterprise Gateway without terminating the remote kernel.
3. Record channel interruption, replacement readiness, and total recovery time.
4. Reconnect through Workspace and evaluate the variable.
5. Confirm kernel/Spark identity did not change and no duplicate or orphan was created.

Expected: the client reports a temporary interruption, then reconnects after standalone recovery restores the same remote kernel. Uninterrupted service is not expected.

NB-S012 — Kernel operation controls

Status: Incomplete controls source-verified; live capture pending.

1. Start a disposable notebook server and inspect every visible kernel action.
2. Exercise cell Interrupt and each toolbar or menu action.
3. Record request count, scope, confirmation, state loss, result, and failure.
4. Repeat with two notebooks sharing the personal server.

Expected: Interrupt, Restart, Restart and Run All, and Shut Down either work with correct notebook scope or are absent; Stop Server clearly affects all sessions; no inert control is visible.

NB-S013 — Close view versus shut down

Status: Implicit deletion source-verified; live capture pending.

1. Define a unique variable and open the notebook in one tab.
2. Close that view without selecting a shutdown action, reopen it, and compare session/kernel identity and variable state.
3. Repeat with two tabs and close only one.
4. Repeat using an explicit Close and Shut Down action when available.

Expected: Close only detaches the view; explicit shutdown warns and deletes the session; one tab cannot terminate another tab's active work.

NB-S014 — Shared runtime and cell queueing

Status: Multi-cell inconsistency observed; source behavior verified.

1. From a cold runtime, quickly run three cells.
2. Repeat with Run All and two tabs on the same notebook.
3. Record server, session, kernel, connection, Spark, and cell states, readiness probes, and execution order.
4. Refresh during initialization.
5. Replace the kernel without changing the notebook path and repeat.

Expected: one shared runtime and readiness probe exist per kernel ID; waiting cells explain their queue; order is deterministic; tabs agree; refresh rehydrates; replacement invalidates old readiness.

NB-S015 — Output persistence

Status: Successful and cleared-output paths are source-tested; deployed capture pending.

1. Run a cell that returns a unique value.
2. Wait beyond the save interval and capture the notebook update.
3. Refresh and inspect Version History.
4. Repeat with an exception, cancellation, Clear Output, and Clear All Outputs.
5. Compare execution counts and stored output after each action.

Expected: every output change saves the correct notebook once after inactivity, refresh restores it exactly, and success or failure remains visible.

NB-S016 — Tab-scoped autosave

Status: Shared dirty state and active-tab targeting source-verified; deployed reproduction pending.

1. Open notebooks A and B.
2. Edit A and switch to B before autosave runs.
3. Record the public ID and payload actually updated.
4. Repeat while A executes and finishes in the background.
5. Refresh both notebooks and inspect independent dirty/save status.

Expected: only A is dirty and written, B is untouched, A's result persists, and changing the active tab cannot retarget the save.

NB-S017 — Save failure and close choices

Status: Failure and dialog defects source-verified; deployed reproduction pending.

1. Force the notebook update request to fail.
2. Edit a disposable notebook and allow autosave, then attempt reload and close.
3. Separately exercise Save and Close, Discard and Close, and Cancel on active and background dirty tabs.
4. Record target ID, payload, request count, dirty state, dialog, and tab/session outcome.
5. Restore the service and use Retry.

Expected: failure remains dirty and visible; Save writes the selected tab once and closes after success; Discard performs no write; Cancel changes nothing; compute shutdown is separate.

NB-S018 — Concurrent save and scale boundaries

Status: Unconditional update, full-object versioning, and BFF buffering are source-verified; deployed concurrency and scale measurements pending.

1. Open one notebook in two isolated browser contexts at the same object version.
2. Make different changes, save context A, then save stale context B.
3. Inspect final content, conflict response, history, diff, and revert.
4. Repeat bounded saves while increasing output size.
5. Record request size/rate, BFF memory, latency, object versions, history rows, and retention behavior.

Expected: stale writes are rejected or reconciled before overwrite; recovery works; payload, memory, save-rate, and retention boundaries are enforced and visible.

NB-S019 — User-server state contract

Status: Current state collapse source-verified; deployed fault sequence pending.

1. Record the authoritative Hub server model while starting, running, and stopping a disposable server.
2. While running, block only the status request for two health intervals and restore it.
3. Exercise an approved spawn failure and a spawn beyond the client timeout.
4. Attempt one cell during each state and observe queued work.
5. Compare Hub state with Workspace labels and recovery.

Expected: spawning, running, stopping, failed, and unreachable remain distinct; transport loss preserves last known state; only confirmed absence is stopped; queued work is not retargeted or lost.

NB-S020 — Terminal execution branches

Status: Non-terminal source branches confirmed; controlled protocol and lifecycle injection pending.

1. Independently suppress a matching execution reply and matching idle status.
2. Close the channel after partial stream output.
3. Force send failure, prevent Spark readiness, and fail server startup with a queued cell.
4. Record cell metadata, spinner, partial output, pending requests, queue controls, and recovery for each branch.
5. Confirm no branch automatically reruns code.

Expected: every branch reaches completed, failed, cancelled, or execution-lost within a documented bound; partial output is labelled; queue and initialization remain cancellable; reconnect reconciles authoritative state.

NB-S021 — JupyterHub control-plane continuity

Status: Hub replacement partially completed on 4 September 2026. Runtime resources survived, but Oqullus falsely reported the server stopped and required a page refresh to reconcile. Kernel-memory, proxy, and database checks remain.

1. Start a disposable personal server and remote kernel, define a unique variable, and record Hub, proxy, server, session, kernel, and Spark identities.
2. Confirm the deployed Hub is configured to preserve user servers on shutdown.
3. Perform an approved Hub-only replacement while leaving the proxy, user server, Enterprise Gateway, and Spark runtime intact.
4. During replacement, test the existing routed notebook and separately record new login, server-start, and server-stop feedback.
5. After the Hub returns, verify it recovers the same server and routes without creating a duplicate server or kernel, then evaluate the variable.
6. In separate controlled exercises, interrupt only the proxy and simulate recoverable Hub-database unavailability; record connection, control-plane, state, and recovery behavior without deleting durable data.

Observed Hub-only result in gj76anadpsc / w-t5e2kcxjd:

• The Hub pod was replaced successfully.
• The personal-server pod retained its Kubernetes UID, IP, age, and zero restart count. Enterprise Gateway was unchanged, and the same SparkApplication remained RUNNING with one attempt. No duplicate runtime appeared.
• The replacement Hub rediscovered the existing personal server.
• Two server-status requests returned HTTP 503, 20 seconds apart, while the Hub was unavailable. Oqullus then displayed Server not started, cleared its local kernel selection, and stopped health polling.
• Refreshing the page, without pressing Start, restored the running state. The recovered page received successful status and session responses and rejoined the existing kernel WebSocket.
• The pre-restart probe recorded UUID 4ff52917-53bf-4a69-95a5-8652f3b0061d and process ID 61. A post-restart evaluation was not recorded, so in-memory namespace preservation remains Unverified.

Expected: existing servers and kernels survive Hub replacement; temporary status failure is shown as unreachable or reconnecting rather than stopped; the client keeps reconciling and recovers without refresh; healthy proxy routes remain usable; and Hub state and routes reconcile without duplicates. Proxy interruption reconnects to the same runtime, and database interruption does not silently destroy running compute.

NB-S022 — Runtime credential containment

Status: Injection and both Enterprise Gateway allow-lists are source-verified; deployed presence, authority, expiry, revocation, and redaction checks are pending.

Use a disposable notebook and synthetic credentials. Never print, screenshot, copy, or persist a real credential value.

1. Record the deployed BFF and platform-stack revisions for realm gj76anadpsc, workspace w-t5e2kcxjd.
2. In a remote-kernel cell, use bool(os.getenv(name)) to record only presence or absence for each access-token alias, OAuth client-secret alias, and the OpenBao token. Do not return the raw value, run printenv, or display os.environ because notebook output may be autosaved.
3. At personal-server scope, repeat the presence-only check for refresh-token aliases and the shared Enterprise Gateway token.
4. Decode JWT metadata locally and retain only issuer, audience, scopes, issued-at time, and expiry; never retain the encoded token or signature.
5. Verify the intended workspace operation with an approved synthetic identity, then verify a different audience and workspace are denied.
6. Confirm expiry or revocation stops reuse and produces a recoverable re-login path without silently retrying privileged operations.
7. Search notebook outputs and versions, BFF/Jupyter logs, Spark event logs, and termination diagnostics for the synthetic marker.

Expected: no confidential OAuth client secret, refresh token, or reusable OpenBao token is available to the remote kernel or other notebook-executable code. The shared gateway credential is isolated to the trusted Jupyter Server and Enterprise Gateway boundary. Any unavoidable user delegation is short-lived, single-audience, workspace- and user-bound, minimally scoped, and absent from persisted output and logs.

NB-S023 — Maven dependency resolution

Status: One PostgreSQL JDBC coordinate and controller-local cache population were live-verified on 9 September 2026; capacity and failure-path exercises are pending.

Use a disposable kernel profile and an approved Maven coordinate that is not already present in the Spark image.

1. Record the Spark Operator release revision, controller identity, restart count, replica count, and cache capacity without recording environment variables or repository credentials.
2. Start one disposable Spark kernel that declares the approved coordinate.
3. Confirm the SparkApplication is created, the controller reports successful resolution, and the driver and executor become ready without Ivy write or repository errors.
4. Confirm the controller cache contains metadata and the requested artifact, then confirm the driver receives the resolved JAR in its submission working paths. Do not treat /opt/spark/jars absence as failure; that directory represents the immutable image.
5. Exercise one harmless operation that requires the dependency against an approved target. Record only success, timing, identities, and sanitized error classes—never a credential or production row.
6. Delete the disposable runtime. In a separate approved exercise, validate an invalid coordinate and near-full cache produce actionable launch failures and recover without corrupting later launches.

Observed: release revision 3 had one running controller with no restarts and a fresh bounded cache. org.postgresql:postgresql:42.7.7 populated the controller cache, reached the driver, and the SparkApplication ran. A JDBC read was user-reported as successful.

Expected: dependency resolution occurs before driver creation, cache usage stays within its bound, failures identify the package or capacity cause, and later valid launches recover without operator-state corruption.

NB-S024 — Internal service network boundary

Status: Driver-to-internal-PostgreSQL DNS and TCP reachability were live-verified on 9 September 2026; denial policy validation is pending.

Use a disposable synthetic target or an explicitly approved service endpoint. Do not query production data, enumerate internal services, retrieve credentials, or perform destructive database operations.

1. Identify the labels and effective egress policy for the disposable notebook server, Spark driver, and executor.
2. Verify required DNS, Enterprise Gateway, driver/executor, object-storage, metastore API, governed query, and approved external-data paths still work.
3. Attempt a TCP connection—not a login or query—from each user-controlled workload type to a labeled synthetic platform-database endpoint.
4. Confirm the connection is denied by policy and that denial telemetry records workload and destination identity without payload or credentials.
5. Test one explicitly approved external JDBC destination through its governed egress path and confirm the exception cannot address internal database services.

Observed before remediation: one Spark driver resolved and connected to the Hive Metastore PostgreSQL service on port 5432. A JDBC read using an application credential was user-reported but not retained as independent evidence.

Expected: required platform and approved external paths work, while every direct connection from user-controlled notebook and Spark workloads to labeled platform database ports is denied and observable.

Evidence still required

• Before/after runtime identities for refresh and multiple-tab restoration.
• Restricted-role, notebook-ownership, data-denial, cross-workspace, audit, and kernel-channel security evidence.
• Runtime credential presence by boundary, effective Keycloak audience/scope/ lifetime, OpenBao authority, revocation, and output/log redaction evidence.
• Ivy cache capacity, cleanup, alert, repository-failure, and replica behavior.
• Effective notebook/Spark egress policy, required-flow allow-list, and direct platform-database denial across supported compute profiles.
• Interrupt, timeout, idle culling, server failure, kernel/Spark failure, BFF restart, Hub kernel-memory confirmation, proxy/database continuity, and standalone gateway replacement evidence.
• Output, payload, memory, save-rate, retention, concurrent-cell, concurrent-tab, and concurrent-kernel boundaries.
• Deployed image and configuration provenance, metrics, alerts, SLOs, runbooks, incident history, recovery objectives, and cost attribution.
• Product, UI, BFF, platform, data, security, and operations review decisions.