Notebooks
A System9 notebook is a governed, versioned, catalog-bound artifact. You author cells, version them immutably, branch them, bind the notebook into a pipeline as a task, attach a schedule, publish a cell as an API endpoint, and read its run history. Every one of those persists to real storage under the same identity, policy and audit as anything else on the platform.
What makes it more than a hosted notebook is that it sits inside the data platform rather than beside it. A SQL cell runs on the platform's own planner, a graph cell on its graph engine, and both inherit row and column policy, lineage capture and cost attribution without the notebook doing anything to earn them.
Six kinds of cell
A notebook is not only code and prose. Six cell types are first-class, and the useful ones are the middle four:
| Cell type | What it is |
|---|---|
| Code | Executable code |
| Markdown | Rich text, with executable fenced blocks |
| Query | A catalog-aware query cell with schema binding, so the cell knows what it is reading |
| Streaming | A continuously executing cell fed by change data capture or messaging |
| Visualization | A declarative chart or graph specification rather than plotting code |
| Raw | Text that is neither executed nor rendered |
Output is typed to match: text streams, rich display data, return values, structured tables carrying a schema, graph data as vertices and edges, errors with tracebacks, and progress updates for long-running work.
A table result is a table, not a rendered picture of one. That is what lets the next cell consume it.
Four languages, one notebook
Cells execute on kernels by language:
| Kernel | What it runs |
|---|---|
| Python | General code, in a confined worker process |
| SQL | Delegated in-process to the data platform's own SQL engine and planner |
| GQL / openCypher | Delegated to the graph engine |
| Gremlin | Delegated to the Gremlin traversal engine |
The SQL and graph kernels are delegations, not reimplementations. A query in a notebook cell runs through the same planner, the same governance and the same lineage capture as the identical query issued over the wire. There is no notebook dialect to learn and no second policy path to audit.
Results flow between languages
This is the feature that makes a polyglot notebook actually polyglot. Results are bridged across languages automatically, under a stable naming convention:
| Name | What it holds |
|---|---|
_sql_1, _sql_2, … | The result of the nth SQL cell |
_gql_1, _gql_2, … | The result of the nth graph cell |
_out_1, _out_2, … | The output of the nth Python cell |
_last_sql | The most recent SQL result |
_last_gql | The most recent graph result |
So a SQL cell queries the lakehouse, and the next Python cell picks up
_last_sql without any export, serialization or file in between. Types are
bridged at the boundary rather than left for the author to reconcile.
Execution
Notebook execution runs through one engine, reached from several places:
| Caller | What it runs |
|---|---|
| A pipeline task | A notebook bound into a flow as a step |
| A scheduled job | A durable queued job drained by a worker that is enabled by default |
| A workflow graph node | A notebook as a step inside an agent workflow |
| An agent tool | A notebook invoked as a governed tool call |
| A continuous-delivery step | A notebook in a change-control pipeline |
All of these share one execution implementation, deliberately. The scheduled path does not get its own copy of "run a notebook" that can drift from the pipeline path.
Runs land in a hash-chained run ledger, which is what the run history reads from, so what you see is the record rather than a summary assembled for display.
Failure is typed, never faked
A run that cannot proceed fails with a named cause: the notebook was not found, no kernel pool was available, the engine errored, or the notebook itself failed. Each is distinct.
That matters more than it sounds. "The notebook failed" and "there was nowhere to run it" call for completely different responses, and a single generic error would send you looking in the wrong place. Nothing is ever reported as a completed run that did not happen.
The enqueue side of scheduled notebook runs, the authoring surface that creates the recurring job, is on the roadmap. The worker that drains and runs those jobs is live and enabled by default, and the pipeline, workflow-node, agent and delivery paths all execute today.
Streaming cells
A streaming cell runs continuously rather than once. The manager separates two layers that compose but stay independent: a lifecycle (register, start, pause, resume, stop, checkpoint) and live streaming (publish, subscribe, shut down).
Output fan-out is bounded, there are checkpoints, and isolation is by workspace and clean room, so a long-running cell in one room cannot flood or leak into another.
Publishing a cell as an API
A notebook cell can be published as a governed HTTP endpoint. The registry stores the contract, route, method, authentication, rate limit, timeout, and request and response schemas, and points at the cell that fulfils it.
Routes are unique per workspace, and every change to the publication carries an audit stamp. This is how an analysis becomes a service without being rewritten as one.
In a clean room
Clean-room notebooks add an approval gate on top, with one property worth highlighting:
Editing resets approval. Approvals are keyed to a version hash, so submitting a new version invalidates every prior approval.
A notebook cannot be approved, then quietly edited, then run. Approval requires a threshold of distinct approvers, and the decision logic is pure and separately auditable from the storage that feeds it. Submissions are also linted and reviewed before they reach the approvers.
Notebooks as pipelines
The intended way to run a notebook is not a standalone "run this notebook" button. It is a notebook as a task in a governed pipeline, the same model a data engineer knows from a notebook task inside a job, but governed, provenanced and portable across cells. This is the reconciliation of two true-sounding statements: a coverage review that said "nothing runs a notebook" and the design intent that "notebooks run as pipelines." Both are accurate, the pipeline path is the real design, and its executor is built, but the concrete engine that would actually execute the cells is not yet supplied, so a notebook pipeline task fails closed today.
What is genuinely built at this seam, and worth documenting, because it is the governance that makes the path different from a bare notebook runner, is the task executor and its gates:
- A clean-room-aware task node. The pipeline layer must not import the compute service, so the notebook task node takes an injected runner port and stays independent of the engine. This is a deliberate layering boundary, not an accident.
- Two independent clean-room gates before any run. When the task runs inside a clean room, the runner is not called until admission passes (the carrying pipeline is admitted to the room and the notebook is approved for the room with a matching version hash) and egress passes (every output destination is admissible under the room's resolved egress policy, the same sink-admission chokepoint pipeline sinks use). Any admission failure or egress denial returns a terminal failure with the notebook unrun and no sink opened.
- A reactive skip gate. The task can skip a recompute when the notebook's content hash is unchanged from the last completed run, and the gate is fail-safe: if the runner does not report a content hash, skipping is disabled and every run recomputes.
- A hash-chained run recorder for provenance and artifacts.
*Figure: the notebook-as-a-pipeline-task path. Admission and egress gates (amber) are fail-closed; the reactive skip (cyan) is fail-safe; provenance is recorded (green). Ceiling: the task node resolves its runner itself, defaulting to the same notebook engine and kernel pool the durable worker and the flow task use, so there is one execution implementation per task kind rather than a second path to drift from. The port stays injectable for a deployment that needs to substitute it.
Table equivalent
| Stage | State | Behaviour |
|---|---|---|
| Notebook task node | neutral | Carries notebook id/version and input/output bindings; stays independent of the compute service via an injected runner port |
| Clean-room admission | refusal (fail-closed) | Pipeline admitted, notebook approved, and version hash matched, or the node fails with the notebook unrun |
| Egress check | refusal (fail-closed) | Every output destination admissible under the room's egress policy, or the node fails with no sink opened |
| Reactive skip | candidate (fail-safe) | Skips recompute on an unchanged content hash; recomputes when the hash is unknown |
| Runner | verified | Executes the cells. Defaults to the engine notebook runner, the same engine and kernel pool the durable worker and the flow task use; the port remains injectable |
| Run + artifact record | verified | Hash-chained provenance for the run |
The kernel a cell runs on
Notebook cells execute on a pooled, governed kernel rather than a shared interpreter.
The pool maintains pre-warmed Python processes, claims one on demand and binds it to a workspace, tracks active kernels by workspace, and reaps idle kernels past their time to live. Warm-pool claiming is why the first cell in a session does not pay process-start cost, and reaping is why an abandoned notebook does not hold a process indefinitely.
Kernels are quota-admitted: creating one is subject to the same capacity governance as other compute, so notebooks cannot quietly consume a cell.
Isolation
A Python worker runs under a seccomp-BPF filter that blocks dangerous system calls, including process creation, execution, mounting and tracing. The filter is applied after resource limits and before the worker accepts any work, so all user code runs under it rather than racing it.
Where the platform provides it, the filter is not optional; where the host cannot support it, the condition is logged rather than silently assumed.
Policy runs inside the kernel
The property worth carrying away: row- and column-level security is enforced inside the kernel, not in a wrapper around it.
A notebook cell reading a table sees exactly what its author is entitled to see, because the filters and masks are compiled into the query plan by the same in-engine enforcement that governs every other read. There is no notebook-shaped hole where a user drops into Python and reads around the policy.
In the deployment matrix
The metadata plane behaves identically across the whole matrix: single-cell and multi-cell, sovereign and cloud, a notebook authors, versions, branches, binds, schedules and publishes the same way, and run-history reads are resolved and replicated so the reactive gate's decision is identical single-cell and multi-cell. Execution, by contrast, is not corner-specific. It is uniformly unreached: no corner of the matrix has a demonstrated notebook run, and every attempt fails closed rather than degrading silently. When execution is wired, the design places it in-process on whichever cell runs the task, sharing one CPU admission cap, with cross-cell placement decided by the pipeline's remote dispatch, so a notebook task and a script task in the same pipeline compete for compute under the same admission control.
Related topics
- Data Pipelines, the pipeline model a notebook task binds into, and how a pipeline distributes across cells.
- AI functions, the function object, its five call sites, and the computed isolation tiers a governed run inherits.
- compute service, the execution and agent plane that owns kernels and notebooks.
- Workspaces & clean rooms, the boundary the clean-room notebook path runs inside.