REL-NB-010 — Recover truthfully across JupyterHub interruption

Field	Value
Status	Partially validated; runtime survived Hub replacement but the client reported a false stopped state
Architecture state	Deployed single-instance Hub, proxy, and database; Hub replacement exercised in gj76anadpsc / w-t5e2kcxjd
Severity	Operational boundary
Affected surface	Existing notebook access, login, server start/stop, proxy routing, profile discovery, and Hub state recovery
Components	sparqd/platform-stacks, JupyterHub, configurable HTTP proxy, PostgreSQL; sparqd/oqullus and sparqd/bff-api own user feedback and reconnection
Associated issue	oqullus #516 for truthful client state and automatic reconciliation; proxy and database validation remain evidence-only

Where the problem appears

Restart the Hub while a disposable user server and remote kernel remain alive. The runtime can survive while the Workspace UI incorrectly reports that the server stopped. Proxy interruption and database recovery remain separate exercises.

Concrete observed case

On 4 September 2026, an approved Hub-only rolling replacement was performed in gj76anadpsc / w-t5e2kcxjd while a disposable personal server and remote Spark kernel were active. Before the replacement, the probe variable was 4ff52917-53bf-4a69-95a5-8652f3b0061d and the kernel process ID was 61.

The Hub pod was replaced. The personal-server pod retained the same Kubernetes UID, age, IP, and zero restart count; Enterprise Gateway was unchanged; and the same SparkApplication remained RUNNING with one attempt. The replacement Hub logged that the user server was still running, and no duplicate personal server or SparkApplication appeared.

The Workspace UI nevertheless changed to Server not started. BFF evidence showed two GET /jupyterhub/status-server responses with HTTP 503, 20 seconds apart, while the Hub was unavailable. After the Hub recovered, refreshing the Workspace page—without pressing Start—returned the UI to running. The refreshed client received HTTP 200 from server status and sessions, then rejoined the existing kernel WebSocket.

The final evaluation of the probe variable and process ID was not recorded, so preservation of the kernel's in-memory namespace remains Unverified. Existing route behavior during the interruption, login and lifecycle feedback, proxy interruption, and database recovery also remain open.

Expected condition

Existing user servers and remote kernels survive a Hub restart. While the Hub is unavailable, login, spawn, stop, and control-plane operations fail truthfully without changing runtime identity. The client presents the server as temporarily unreachable, retains its last confirmed identity, and keeps reconciling. Existing proxy routes remain usable when the proxy is healthy. After recovery, the Hub and client reconnect to durable state without a refresh and without creating duplicate servers or kernels.

Proxy interruption can disconnect notebook traffic, but the client reconnects to the same surviving server and kernel. Database unavailability blocks Hub state operations without silently destroying running compute.

Cause and evidence

JupyterHub separates control-plane lifecycle from the proxy and per-user servers. The assessed deployment enables preservation and successfully restored the existing server record after replacement.

The failed client recovery is explained by sparqd/oqullus source. While the server is running, useWorkspaceServer polls status every 20 seconds. Two non-Active responses or request errors call transitionToStopped(), which sets the server to stopped, clears kernelActive, and stops health polling. The two observed 503 responses therefore made a temporary control-plane outage indistinguishable from authoritative server absence. With polling stopped, the page could not discover recovery until it was remounted by refresh. The current unit test explicitly expects two network errors to produce this stopped state.

The Hub, proxy, and database still each have one serving instance. The Hub replacement included an observable control-plane gap while the new image was pulled and became ready; proxy and database interruption semantics must still be validated independently.

Why it matters

Kernel survival does not guarantee notebook reachability or truthful controls. Without continuity evidence, operators cannot distinguish an acceptable brief control-plane interruption from duplicate compute, lost routes, or state drift.

Existing controls and workarounds

Running user servers are configured to survive Hub shutdown, and the proxy is a separate deployment. Existing work may continue when its route and connections remain healthy. Until automatic reconciliation is corrected, do not press Start when the UI reports Server not started after a platform interruption. Refresh the page, confirm that the existing server is running, and reconcile the session and kernel identity before executing more work.

Alternatives considered

1. Recommended client correction: represent status transport failure as unreachable or reconnecting, preserve the last confirmed server and kernel identity, continue bounded polling with backoff, and transition to stopped only after an authoritative Inactive response or successful explicit stop.
2. Retain the simple workspace-sized topology, complete continuity validation, and establish an interruption objective.
3. Vertically size the Hub, proxy, or database after measured login, spawn, routing, or state latency demonstrates a throughput limit.
4. Add replicas or a more complex high-availability design only when an explicit availability objective requires it and the full state/routing behavior is supported and tested.
5. Move compute-profile discovery out of the Hub only if it needs an independent availability or scaling boundary.

Component involvement

• sparqd/platform-stacks owns Hub, proxy, database, resource, disruption, and lifecycle configuration. The exercised Hub setting preserved the user server.
• JupyterHub owns login, server state, spawn/stop, and route reconciliation; it rediscovered the existing server after replacement.
• The proxy owns continued access to existing routes.
• PostgreSQL owns durable Hub state and recovery.
• sparqd/bff-api exposed the temporary Hub unavailability as HTTP 503 and later returned healthy status and session responses.
• sparqd/oqullus owns the confirmed false-stopped transition, loss of local kernel state, missing automatic reconciliation, and recovery presentation.

Reproduction and verification

Follow NB-S021.

Resolution conditions

Close validation when the Hub exercise also confirms the in-memory probe, oqullus remains reconnecting through temporary 503 responses and recovers without refresh, and controlled proxy and database exercises preserve runtime identity, avoid duplicates, recover routes and state, and provide actionable user feedback under an agreed interruption objective.

Validation and blockers

Hub replacement, user-server preservation, SparkApplication preservation, Hub state restoration, and the false-stopped client behavior are live-verified. Kernel namespace preservation is unverified because the post-restart probe was not recorded. Existing-route behavior during interruption plus proxy and database dependency-failure exercises still require approved disposable runtime and operator coordination.