Platform replication and the system record
A workspace is replicated, cloned and recovered as a unit, and that is a thing a person or an agent provisions. This page is the other plane: how the platform replicates its own state, and how a node builds a view of itself out of the cells it owns.
The two are deliberately separate. One is a customer-provisionable resource; the other is the substrate keeping itself coherent.
Connections, not peers
Replication does not iterate raw peers. It iterates replication connections, and a connection is a declared object with its own shape:
| A connection declares | Values |
|---|---|
| Target peer | The cell or node on the other end |
| Direction | bidirectional, pull_only, or push_only |
| Databases | Which of the logical domains this connection carries |
| Schedule | When it runs |
| Scope | Resolved from topology, up to and including cross-organization |
Each tick the engine walks its connections and performs the inbound pull and the outbound push each one's direction permits. A peer that cannot receive a push, an outbound-only peer behind a one-way boundary, instead asks this cell for the events it has not seen, and the pull leg is guarded symmetrically with the push leg rather than being the softer of the two.
That symmetry is the design point. A mesh-pulled batch passes the same inbound guard chain as a gateway push: replay guard, cross-organization filter, allow-list, unseal, attestation and ledger. There is no direction of travel that is cheaper to trust.
What replicates, and at what scope
Replication is declared per table, not per database, and the declaration sits next to the table's own schema rather than in a separate configuration file. A table cannot be added to the platform without an explicit replication decision, because the field that carries it is required and has no default.
That single choice is what keeps "which data crosses which boundary" a property of the data model instead of a runtime setting someone can quietly widen, and it is the source the replication machinery is generated from rather than a document describing it.
The four things a table must say
| Declaration | What it answers |
|---|---|
| Movement | Does the row travel at all, and by what shape: stay put, converge by primary key, converge by content hash, or converge a metadata row while the bytes live in a data plane |
| Scope | How far: across the cells of a node, or across the nodes of an organization |
| Target | To whom: broadcast within that scope, or only to the owning identity's home and mirrors. A private message must not broadcast to every cell |
| Tenancy | Whose rows these are: a workspace, a clean room, an inherited parent, or an explicitly stated exemption |
Alongside these, a table may declare which columns are stripped at the emit and apply boundaries, so private material is removed on the way out rather than filtered on the way in, and a per-field merge rule where whole-row last-writer-wins is the wrong answer.
An internally inconsistent declaration is a hard error when the code loads, not a surprise in production. Declaring convergence without a primary key, or claiming a row has no owner while naming the column that identifies its owner, will not start.
Converging is the default, and staying local must be justified
The governing rule is what the design calls the node-view axiom: any cell of a node should have the same view of the node's core databases. Convergence is therefore the expected answer, and a table that stays local has to cite one of exactly four reasons:
| Reason a table may stay local | What it covers |
|---|---|
| Runtime | Genuinely cell-bound state: a running process, an in-flight queue or upload, a coordination buffer, anything pinned to this cell's hardware |
| Engine internal | The replication mechanism itself, which cannot replicate itself: replication logs, vector clocks, outboxes, per-peer cursors, watermarks, fencing tokens |
| Telemetry | Per-cell metering and usage streams, and per-cell hash-chained audit ledgers, where merging chains across cells would break the hash chain |
| Cache | Anything rebuildable from a convergent source: search and vector indexes, statistics, rollups, aggregates |
A table that fits none of the four is treated as a mistake to be corrected, not a preference to be respected. Making the justification a required, typed value is what turns "should this converge?" from prose nobody can check into a reviewable decision that a test can hold to.
Scope and the transports underneath
Scope is resolved from the same physical topology everything else resolves against, across nine levels: intra-node, intra-rack, intra-zone, cross-rack, cross-zone, cross-region, cross-node, nested-organization and cross-organization.
Convergence itself uses vector clocks for causal ordering, an apply pipeline that detects duplicates and staleness before resolving conflicts, anti-entropy repair, snapshot resync above a lag threshold, an outbox per database, replay guards and tombstone collection. Transports degrade from gRPC all the way to store-and-forward over a constrained radio link, with a bandwidth scheduler.
A worked example: why a session behaves the way it does
Sessions make the model concrete, because the answer is not uniform even inside one domain. Live sessions and session revocations converge across the cells of their node, so any cell can validate a session locally and a logout reaches every cell that could have honoured it. Password credentials converge further, across the organization's nodes, so a whole-node failure never costs a user the ability to sign in. The per-cell auth event stream converges not at all, under the telemetry exception, because each cell owns its own hash-chained audit ledger.
Three different answers, three different reasons, all declared beside the tables they govern. See Sessions across a node's cells for what that means in practice.
Metrics is the exception, on purpose
The metrics store does not replicate. It is the ninth database, an analytical store rather than a transactional one, and it is excluded from the replication plane by design rather than by omission.
Instead, cells query each other. A federated metrics query fans out to remote cells and merges what comes back, with three properties that keep the fan-out honest:
- A hard cap of 64 remote cells per federated request, so latency does not grow linearly and unboundedly with cluster size.
- A short per-cell timeout, with a persistent worker pool so the fan-out does not pay thread-startup cost on every query.
- Per-cell failures are returned as failures, not silently dropped, so a partial answer is legible as partial.
The reasoning generalises, and it is worth stating because it is the kind of decision a rival copies the shape of without the reason:
State that must converge is replicated. Observation is queried where it lives. Copying every metric sample to every cell would multiply the most voluminous and least valuable-to-duplicate data in the estate, to answer questions that are almost always about one place at one time.
Health synthesis, by contrast, does roll up: cell, service-group, zone and region health each synthesise from collected metrics into one view per level. Aggregates travel. Samples stay put.
Alignment rolls up by signed fan-in, not by replication
The metrics rule generalises, and alignment state is the other place it lands, in the opposite direction. Metrics are pulled where they live; alignment state is pushed as a signed summary. Neither is replicated.
The rule underneath both is the one the node-view axiom implies but does not say out loud: a cell is a component of its node, and cell-level state never crosses the node boundary. Convergence is an intra-node property. Something else has to carry what an organization needs to see.
For alignment, that something else is a fold and a push:
| Step | What happens |
|---|---|
| Observe | Each cell records its own alignment observations and the co-witnessed decisions it attests to |
| Converge, within the node | Both of those converge across the node's cells, and only across them. Their replication scope is intra-node |
| Fold | The node folds its cells' state into one node-level summary |
| Sign | The node signs that summary with its Ed25519 node identity, the same identity it signs metering bundles with, over the canonical bytes of the payload |
| Push | The node sends the summary to the organization proxy over the hub-and-spoke egress plane, under a system token, with a hard size cap on the body |
| Verify and fold again | The proxy verifies the signature against the sending node, stores the latest summary per node, and folds its own cells plus its verified peers into the organization view |
The mental model is worth stating plainly, because it is the part that makes the boundary legible: node-to-node is two people talking. A peer knows the node, by way of a summary that node chose to fold and sign. It never sees that node's cells, and there is no row it can read across the boundary to find them.
That the summary is signed is what makes the fan-in safe to accept. The proxy is not trusting the connection, it is trusting a signature it can check against the node's directory identity, which is why a peer's summary and the proxy's own cells can be folded into one view without the proxy having to treat a peer as an extension of itself.
A single or air-gapped node folds to itself. It is its own proxy, or it has none, so the push tick simply does not fire. There is no phone-home on this path, and an organization-level view on a disconnected node is that node's own fold, which is the honest answer rather than a degraded one.
The boundary is an encapsulation boundary
It is worth being precise about why the node boundary works this way, because it is a design decision rather than a limit waiting to be lifted.
Raw per-cell telemetry never crosses a node. A node presents node-level summaries to its peers, and anything else that crosses does so as a signed, self-sufficient document rather than as a row somebody reads out of a table. A peer cannot ask what a node's cells are doing, because that question has no surface to arrive on.
That has a direct consequence for cross-entity coupling, the measure of whether a relationship between two entities is symmetric or extractive. Its peers are cells, and cells are intra-node, so the coupling measure is node-scoped by the boundary. A cross-node version would require per-cell telemetry to cross, which is precisely what the encapsulation rule forbids. It is not an unfinished fold.
Cross-node cooperation is expressed a different way instead.
Cross-node quorum witnessing
When a cell takes a governed decision, it writes a self-witness, a signed record standing behind what it did. Those witnesses are what cross the boundary, and a decision's witnesses are unioned across the organization, so the question "was this decision witnessed by enough distinct peers?" is answered across nodes rather than inside one.
| Property | How it holds |
|---|---|
| Signed per row | Each witness is Ed25519-signed over its canonical claim, and is meaningful on its own rather than by virtue of the connection it arrived on |
| Re-verified on receipt | The proxy verifies every row against the witnessing cell's enrolled key. Forged, tampered, unsigned or wrong-key rows are dropped, never folded |
| Two authentication layers | A system token proves an organization cell is calling; the per-row signature proves who witnessed. The sender must also be an enrolled peer of that proxy, and only self-witnesses are accepted, so there is no attestation cascade |
| Idempotent | Rows dedupe on their attestation id, so a re-pushed batch is harmless and a dropped one self-heals on the next tick |
| Incremental | The spoke ships new rows by watermark, and a lost watermark simply re-pushes rows the proxy already holds |
| Off unless turned on | The producer is off by default, and an unwired quorum check answers unknown rather than a false "yes" |
The reason this matters beyond tidiness: a quorum that spans nodes is what lets a sovereign multi-node estate notice a defecting peer. A node whose witnesses stop agreeing with everyone else's is visible as such, without any node having to expose its cells to the others.
A single or air-gapped node is its own proxy, or has none, so the producer skips and the quorum is computed from local witnesses. That is the honest answer for an estate of one rather than a degraded one. The threshold itself is governed and deny-closed to a high bar, so a lone self-witness is never silently treated as a quorum.
Cross-organization traffic has its own key
The most interesting property of the replication design is what happens at an organization boundary.
A cross-organization or nested-organization batch is not authenticated with the cluster system token, because a foreign organization never holds it. It is authenticated with a key derived from the trust edge's shared secret, and both organizations must select the same edge to derive the same key. Trust edges are directory objects with per-organization sealing keys behind them.
Around that: a scope resolver, a peer manager, a cross-organization filter on what may traverse an edge, per-peer vector clocks, and an egress guard so denied tables never leave the node on a cross-organization connection.
So what: organization-to-organization data movement has its own cryptographic identity per relationship. Revoking a partner does not mean rotating the cluster. It means dropping an edge. That is the difference between a federation model and a shared secret with extra steps.
The system's own record
Each cell writes structured logs to its own local filesystem, and those rotate, as operational logs should. Above them, durable-worthy records land in a converged store that survives rotation and is replicated so every cell of the node sees the same view, served leaderless.
So a node can answer "what happened across my cells" without a designated log server to lose, and an operator reads the node-wide rollup under an explicit permission while a workspace member reads only their own tail.
On top of that, every cell attests what it is actually running. On each reconcile tick a cell emits a signed effective-configuration attestation, the policy versions it applied, its capability state, its worker set and health, any drift and any active pins, hash-chained into the audit. Distribution alone would be write-only; the attestation stream is what makes the fleet view symmetric, and it is what the drift dashboard renders from.
A disconnected cell keeps reconciling against last-known-good signed policy, and flushes queued attestations on reconnect so the fleet view back-fills its observed history rather than showing a hole.
Related
- Replication, DR & federation, the nine scopes and the catalog's four distribution modes.
- Workspace mobility, the customer-provisionable plane.
- Inside a cell, the nine logical databases this replicates.
- Cell network fabric, how cells find and trust each other.
- Monitoring and activity, the workspace-scoped view.