Replication, DR & federation
A single cell is complete on its own. When cells are grouped, the estate has to answer one question everywhere the same way: what data crosses which boundary, under whose authority, and how does it converge? System9 answers it from one shared topology rather than a global replication switch, each table declares its own scope, the catalog distributes across cells four distinct ways, and disaster recovery is a policy-and-drill discipline rather than an out-of-band script.
Replication is scope-resolved, and declared per table
Replication is resolved from topology, not broadcast. Each table declares its own replication policy alongside its schema, so what crosses which boundary is a property of the data model rather than a runtime setting someone can quietly widen. The engine classifies every path into one of nine widening scopes:
Figure B6: the nine replication scopes, widening from a single node out to a separate organization. The seven neutral scopes are topology boundaries within one trust cluster; the two green org-crossing scopes are authenticated with a key derived from a trust edge's shared secret, never the cluster system token. Ceiling: each table opts into its own scope. This is not uniform convergence, and the security domain has no replication adapter today, so do not read the ladder as "every store replicates everywhere."
Table equivalent
| Scope | Boundary crossed | Authenticated by |
|---|---|---|
| intra-node | within one node's cells | cluster identity |
| intra-rack | within a rack | cluster identity |
| intra-zone | within an availability zone | cluster identity |
| cross-rack | between racks | cluster identity |
| cross-zone | between zones | cluster identity |
| cross-region | between regions | cluster identity |
| cross-node | between nodes | cluster identity |
| nested-organization | into a nested org | trust-edge derived key |
| cross-organization | between separate orgs | trust-edge derived key (both orgs select the same edge) |
Cross-organization is cryptographically separate
A cross-org or nested-org batch is not authenticated with the cluster system token, a foreign organization never holds it. It is authenticated with a key derived from the trust edge's shared secret, and both organizations must select the same edge to derive the same key. Trust edges are directory objects with per-organization sealing keys behind them, so organization-to-organization movement has its own cryptographic identity per relationship. Revoking a partner is dropping an edge, not rotating the cluster, the difference between a federation model and a shared secret with extra steps.
Four ways the catalog distributes across cells
The catalog is not a single-cell registry. It reaches other cells in four distinct modes, each a different topology, and three of the four move no data at all:
Figure C5: the four catalog distribution modes. A filled box is the local cell; outlined boxes are peer cells or databases it reaches; the plain box is an external system outside the platform. Only replicated copies metadata between cells, grafted, federated and foreign leave the data where it is and reach it in place.
Table equivalent
| Mode | What crosses | Does data move? | What it is for |
|---|---|---|---|
| Replicated | Catalog change events; anti-entropy against peers | Metadata is copied to peers | A peer cell holds its own current copy of the catalog |
| Grafted | Query references only | No: transparent proxy | Mount a remote catalog subtree into the local namespace for cross-node queries without moving data |
| Federated | Decomposed sub-queries and their results | No bulk movement | One SQL query spanning several databases, executed per-database and joined |
| Foreign | Metadata registration and query pushdown | No: stays in the source | Register an external system (Postgres, MySQL, Iceberg REST, file collections) as foreign tables |
Two storage tiers, and placement that honors protection
Replication scope answers which boundary data may cross. Underneath it, a second question decides how many copies survive a failure and where they sit, and the answer differs by storage tier, because System9 keeps two distinct durability substrates rather than one store with hot and cold classes.
- Fast-disk structured state. The authoritative relational, graph, vector, and time-series state, the directory, catalog, compute, sessions, messaging, system, and storage metadata, lives in local database files fronted by the multi-model database engine. Its horizontal unit is a shard with one primary cell, kept current across cells by the log-based, vector-clock convergence engine below. Disaster recovery repoints a shard's primary to a surviving cell that already holds its replica.
- Object-storage bulk bytes. Object payloads are content-addressed blocks (large objects are split into blocks plus a manifest, so deduplication survives chunking). Each block is protected one of two ways: replicated as N whole copies on distinct cells, or erasure-coded into k data + m parity shards (Reed-Solomon) spread across cells. On write the pipeline is compress → encrypt → erasure-encode, in that fixed order, so a shard can be reconstructed without a key.
Figure C6: the two ways an object block is protected. Replicated keeps N
whole copies on distinct cells (a copy is modeled as k=1); erasure-coded
keeps k data + m parity shards so any k of the k+m rebuild the block.
Shipped erasure profiles are 4+2 and 8+4. Ceiling: which encoding an object
gets is topology- and size-aware, single-cell or single-rack deployments and
sub-block objects stay replicated; erasure is chosen only where topology and
object size justify it, and it is never retro-applied to bytes already written.
Table equivalent
| Protection | What is stored | Write succeeds when | Survives |
|---|---|---|---|
| Replicated (RF = N) | N whole-block copies on distinct cells | The local block is durable; a distribution shortfall is logged as under-width, not failed | Loss of up to N−1 copies |
| Erasure-coded (k + m) | k data + m parity shards across cells | Quorum is reached (k+1 with parity): otherwise the write refuses (no partial object) | Loss of up to m shards |
Placement refuses to collapse protection onto one cell
A single deterministic placement authority decides where each copy or shard goes. It is consistent-hashing based and resolves against the same region → zone → rack → cell topology every other placement question uses, so two cells with the same topology agree on placement without coordinating, which is what makes the protection invariants checkable across cells. It honors three constraints together:
- Replication factor / erasure width. N copies, or k+m shards, are placed on that many distinct cells.
- Anti-affinity. Copies and shards are spread across the failure-domain tree (cell, then rack, then zone, then region (rack by default)) so a single failure domain does not take out enough pieces to lose the object.
- No single-cell collapse. Placement refuses (rather than reusing a cell) when there are fewer distinct healthy, non-drained cells than pieces to place, and a single cell may never hold two shards of the same object, so losing one cell can never cost two shards. This is asserted as a pre-flight post-condition before any bytes move.
Residency composes on top: a hard require_zone / require_region on a
namespace is fail-closed, an unsatisfiable constraint refuses rather than
placing out of jurisdiction, and it is enforced on all four placement paths:
initial write, drain, repair-driven relocation, and rebalance.
The pre-flight placement post-condition is the weakest form of the spread
invariant: it runs before the object's true k is in hand, so it guarantees "more
than one domain holds shards" but cannot, at that moment, prove full survivability
for a wide erasure profile. The authoritative survivability check runs
post-hoc in the object durability audit, where the profile is known; threading
the true data-shard count into the pre-flight selector is owed refinement.
Pre-flight placement is not a complete survivability proof on its own; the
durability audit is where that proof is made.
This is the storage-tier expression of the same rule DR follows at cutover: standby election never invents a placement it only repoints at a cell that already holds the data, and it is the protection width above that guarantees such a cell exists.
The convergence machinery
Underneath every scope sits one convergence engine: a tick-based loop iterating connections in priority order, per-database adapters, vector clocks for causal ordering, an apply pipeline (dedup and staleness detection → conflict resolution → adapter apply), an outbox pattern per database, anti-entropy repair, snapshot resync above a lag threshold, replay guards, and tombstone garbage collection. Transports degrade all the way from gRPC to store-and-forward over constrained radio links with a bandwidth scheduler, which is what lets an air-gapped peer be modeled as an absent edge, not a failed one.
Convergence is not uniform across every store. Replication adapters exist per database domain, but the security domain has no replication adapter, so this is not a claim that every table in every domain converges the same way. Replication is opt-in per table, by design.
Disaster recovery and cloning
Beyond steady-state replication, resilience adds policy and rehearsal:
- Disaster recovery carries policy and observation, scheduled drills, cell transition and run rehoming, placement resolution, completeness digests, and clean-room-scoped recovery so a mutual-distrust room recovers under its own governance.
- Cloning covers database, workspace, and clean-room clones in copy-on-write, deep, and live modes, gated on net potential gain, metered, and carrying provenance, rather than being out-of-band scripts.
- Startup recovery reconciles in-flight compute work after a restart and scans for orphaned runs.
This holds across the deployment matrix: the same convergence and recovery machinery runs whether an estate is one cell or an elastic fleet, cloud-connected or air-gapped, with the degraded-transport and absent-peer cases treated as first-class states. The operational detail, the failover and failback paths, and copy-on-write cloning, is covered under Lifecycle & operations.
Related topics
- Deployment topology, the topology every scope is resolved from.
- Data residency, how residency constrains where replicas and failover may land.
- Core and edge, the store-and-forward path that carries replication to a disconnected peer.
- The cell model, per-table replication among coordinating cells.
- Disaster recovery, the failover and failback paths, readiness, and residency-aware standby election.
- Cloning, copy-on-write clones and the eight planes that move together.