Compute & Workbench
The compute service carries two commercially distinct things on one substrate: the workbench (what an analyst or engineer works in) and the intelligence engine agents run on. This page covers the workbench: notebooks, kernels, job scheduling, CI/CD, secrets, and the coordination beneath them. The point of putting them on one substrate is that they share one sandbox, one metering path, one audit trail, and one security model with the coding agents and the AI system, an analyst's kernel and an agent's sandbox are governed the same way.
In one paragraph
Notebooks are versioned immutably with server-side content hashing, bound into scheduled DAG pipelines, and published as API endpoints; kernels run under seccomp-BPF confinement with row- and column-level security enforced inside the kernel, all on the same governed sandbox, metering, and audit path as agents and coding runs.
The six pillars
| Pillar | What it provides |
|---|---|
| Notebooks | Immutable, server-hashed version snapshots on named branches; notebook pipelines with scheduling and governed run history; a notebook-as-API-endpoint registry; catalog binding and lineage on every executed query; and six cell types (code, markdown, query, streaming, visualization, raw). |
| Kernels | Python, SQL, GraphQL, and Gremlin kernel controllers with a warm pool, seccomp-BPF confinement, a builtins sandbox, import hooks for module filtering, network restriction, and row- and column-level security enforced inside the kernel. |
| Sandboxed execution | Leased warm sandbox pools, sub-agent spawn and supervision, a net-potential-gain spawn gate, isolation-tier admission, shared blackboards, and a key broker held outside the sandbox. |
| CI/CD | A stage-by-stage pipeline runner with transformation profiles, manual approval gates, dev → staging → prod promotion and rollback, and per-workspace library management with CVE scanning on dependencies. |
| Coordination | Leader elections for singleton work, distributed query execution, remote workflow dispatch, and startup recovery that reconciles in-flight work and scans for orphans after a crash. |
| Infrastructure | Scope-scoped secrets over local/AWS/Azure/GCP backends with scheduled rotation, resource policies and distributed quotas, a compute meter over a write-ahead log, and a fault-injection harness: chaos engineering as a shipped feature group. |
One substrate, three consumers
An analyst's notebook, an engineer's coding run, and an agent's sub-task are three consumers of the same sandbox, kernel pool, metering path, and audit chain, which is why a governance decision made once holds for all three.
*Figure: one substrate, three consumers. An analyst, an engineer, and an agent all run on the same leased, seccomp-confined sandbox and kernel pool, with row- and column-level security enforced inside the kernel and the key broker held outside the sandbox, feeding one metering path and one audit trail.
Table equivalent
| Consumer | Enters through | Shares |
|---|---|---|
| Analyst | Notebook + query kernel | The sandbox, metering, and audit |
| Engineer | Coding run + CI/CD | The same sandbox, metering, and audit |
| Agent | Sub-task + tool call | The same sandbox, metering, and audit |
| All three | Kernel pool | Row/column security inside the kernel; key broker outside it |
Why it is different
- Security is inside the kernel, not in front of it. Row- and column-level policy is enforced within the execution kernel, so a query run in a notebook is governed like a query run anywhere else. See in-engine enforcement.
- The sandbox is shared, so governance is not re-invented per surface. An agent's spawn and an analyst's kernel pass the same isolation-tier admission and the same net-potential-gain spawn gate.
- Chaos is a shipped feature, not an external tool. A fault-injection harness exercises graceful shutdown, startup recovery, and reliability events from inside the platform.
What is live, and what is not
The split falls along metadata versus execution:
- Live and persisting to real storage: create a notebook, version it immutably with server-side content hashing, branch it, bind it as a task into a DAG pipeline, attach a cron schedule, publish it as an API endpoint, and read its governed run history. Creating, listing, and stopping a governed, quota-admitted kernel is also real.
- Execution substrate live, enqueue side owed: a workspace notebook outside a clean room runs headless through the production notebook engine and kernel pool. A durable job queue partition carries the work, a default-enabled worker drains it on every tick, and a failure resolves to a typed cause (notebook not found, kernel pool unavailable, engine error, cell failed) rather than a fabricated success. The drain is cell-local and needs no coordinator, so it holds in the sovereign, air-gapped corner. What is owed is the enqueue side: no authoring action yet places a notebook run on that queue, so the substrate is drainable before it is fillable. A notebook bound as a task inside a flow does execute, through the same engine core.
- The HTTP notebook surface is metadata, not a trigger. The routes create, version, branch, schedule, publish and read. The pipeline-run route records an executor's outcome rather than starting one, so a successful call there is a write-back, not a run. The agent tool family for compute refuses rather than fabricating a result until a deployment wires the relay from the node to the compute service.
- Built but not demonstrated: clean-room notebook execution is wired end to end and governed (approval-gated, M-of-N signed, alias-rewritten so the kernel never sees source table names, executed in a seccomp-confined sandbox), but no test has driven a real kernel end to end on a deployed cell.
What is measured
The compute plane publishes the four numbers that decide whether work will land: active kernels, warm pool size, job queue depth, and the two failure counters that distinguish the ways it can go wrong, kernel crashes and kernel bind failures.
Separating those last two is deliberate. A crash means a workload started and died; a bind failure means it never got a runtime at all. They have different causes and different fixes, and a single "failed jobs" counter hides the difference. Queue depth alongside warm pool size is what tells you whether a backlog is a capacity problem or a cold-start problem. See what is measured.
Across the deployment matrix
- Single-cell: the sandbox pool, kernels, and scheduler run locally.
- Multi-cell, distributed query execution and remote workflow dispatch fan work across cells; leader election keeps singleton work singular.
- Sovereign / air-gapped, the workbench needs no external compute service; secrets can resolve entirely from a local backend.
- Cloud-enabled, secrets can resolve through cloud backends, and libraries are CVE-scanned regardless.
Related
- The coding agent, the software-engineering surface built on this substrate.
- Notebooks, the notebook surface in the Data Platform, in depth.
- The AI system (core + edge), the intelligence engine that shares this sandbox.
- Compute service, the service that carries the workbench.