Cell network fabric
A single cell is a complete platform. The moment there are two, a question appears that a single box never has to answer: how does one cell find, trust, and reach another, without a central master to ask, and without assuming the network is up. The Cell Network Fabric is the answer. It is the plane over which cells advertise themselves, elect the few cross-cell singletons that must be singular, prove they can actually reach a peer, and place authenticated calls to each other.
The fabric is designed to hold across the full deployment matrix: it must work with one cell and with a hundred, cloud-connected and air-gapped. That constraint is why it is built from peer coordination over shared local state, not from a coordinator every cell must phone.
"The fabric" spans two distinct subsystems that share the
word. One is the control plane: the endpoint registry, distributed leases and
leader election, the region/zone topology, the peering graph, and the mutual-TLS
identity snapshot. The other is the cell-to-cell transport path, the
resolve → channel → invoke call seam that carries an actual inter-cell RPC.
Both are real and both are named here as fabric behavior, but they are separate
components, not one object. Where a claim depends on which subsystem you mean,
this page says so.
The full term is Cell Network Fabric. "Fabric" on its own collides with an unrelated vendor product, so this documentation always qualifies it.
The fabric at a glance
Four mechanisms do the work, and every cell runs all of them:
- Endpoint registration. Each cell publishes its reachable endpoints, the protocol, host, port, TLS posture, priority, cost tier, and zone, into a fabric endpoint table. Publication is reconciled at boot and on every certificate rotation, and is funneled through a single writer so the table has one authority per cell rather than a race between services.
- Distributed leases and leader election. The handful of cross-cell singletons (a replication publish-leader, for instance) are held by time-to-live leases, so leadership is a claim that expires rather than a configured role that strands when its holder dies.
- Topology and peering. A region → zone → rack hierarchy classifies failure domains and cost, and a separate peering graph records which cell has been observed able to reach which.
- Attested reachability and authenticated calls. Reachability is proven, not assumed, and every inter-cell call is mutually authenticated, a cell's identity rides the TLS handshake, or, on a cleartext channel, a signed system token it refuses to omit.
Figure: the Cell Network Fabric. The control plane (top) is coordination over shared local state: a single-writer endpoint registry, TTL leases that carry a monotonic epoch, the region/zone topology, and the peering graph. The transport path (bottom) is one inter-cell call: resolve a dialable endpoint, open a mutually-authenticated channel, invoke a bounded RPC. The lease epoch is stamped onto published endpoints (dotted) so a stale leader can be fenced at read time. Ceiling: the "reachable peers only" filter on resolution is applied only when a reachability provider is injected; with none wired, endpoints resolve unfiltered, so treat reachability filtering as a composed behavior, not a property the resolver enforces on its own.
Table equivalent
| Mechanism | Subsystem | What it produces | Refuses / degrades when |
|---|---|---|---|
| Endpoint registration | Control plane | Reconciled endpoint rows (single-writer) | Stale rows are swept by age; publication is idempotent |
| Distributed leases | Control plane | One live holder per scope + a monotonic epoch | An expired lease is preemptible; a lower epoch is fenced |
| Topology & peering | Control plane | Failure-domain + cost classification; observed reach edges | Peering edges expire; an expired edge reads as absent |
| Resolve → channel → invoke | Transport path | One authenticated inter-cell RPC | Raises a typed error; refuses to send a blank token |
Fenced leases and the epoch that makes them safe
Leadership in the fabric is a lease, not a role. A lease is a row keyed by a scope, held for a short TTL (a low double-digit number of seconds by default) and renewed at roughly a third of that interval. When a holder stops renewing, it crashed, drained, or was partitioned away, the lease expires, and any other cell may preempt the expired row and become the new holder. Leadership therefore survives a cell going away without an operator in the loop.
What makes that safe under a partition is the lease epoch: a token that
strictly increases on every takeover. A new holder mints epoch + 1, and
that epoch is stamped onto the endpoints it publishes. A leader that lapsed
its lease but never withdrew its endpoint rows is caught at read time, the reader
sees a lower epoch and fences the stale leader rather than dispatching to it.
The epoch is the fencing token; the TTL is only what triggers the handover.
The TTL itself is adaptive: a healthy cell earns a longer lease (fewer needless handovers), a degraded one a shorter lease (faster failover). A graceful handover has its own protocol, drain, release, and a priority claim window for the intended successor, with a timeout that reopens the election to any cell if the successor never claims, so a planned transfer cannot deadlock.
Figure: a fenced lease takeover. Cell A holds the lease at epoch N and its published endpoints carry that epoch. When A stops renewing, the lease expires and Cell B preempts it, minting epoch N+1 and republishing. A's stale endpoint rows (red) still exist, but the dispatch reader fences them by epoch comparison, a lower epoch never wins. Ceiling: these are database-row leases guarded by a monotonic epoch, not a consensus log; safety rests on the atomic conditional update plus the fencing token, not on a quorum vote. Split-brain zone fencing is a separate, narrower slice. See the ceiling below.
Table equivalent
| Step | State | Fencing effect |
|---|---|---|
| A holds lease | epoch N; endpoints stamped N | A is the leader |
| A stops renewing | lease approaching expiry | none yet: TTL not elapsed |
| Lease expires | row preemptible | any cell may take over |
| B preempts | epoch N+1 minted | B is the leader |
| B republishes | endpoints stamped N+1 | readers prefer the higher epoch |
| A's stale rows linger | still stamped N | fenced at read time (N < N+1) |
The fabric includes a zone-fence decision, a pure function that flags the local zone as isolated when cross-zone heartbeats have gone quiet past a threshold, and returns "the local zone should fence" when it is both isolated and the minority by cell count. That is a decision function: it does not itself drop a cell to read-only, release leases, or invalidate routes. The reliably-wired fence in the fabric today is the lease-epoch mechanism above, which fences a stale leader at read time. So the zone-fence check is not automatic split-brain protection end to end: it provides the isolation-and-minority decision, and lease-epoch fencing is the mechanism actually on the serve path. The integration between the two is owed work.
Attested reachability: proven, not assumed
Being admitted to the fabric is not the same as being able to reach a peer. The fabric answers "can this cell reach that one?" with a layered reachability source that unions three inputs, cheapest and most authoritative first:
- Self. A cell always reaches itself: the peering graph refuses self-edges, so this is supplied directly.
- Attested peering. The peering graph holds edges that were observed and cryptographically countersigned (below). An edge whose observation lease has lapsed reads as absent, not stale, an unproven path is not a usable path.
- Static. Operator-declared reachability from configuration, an explicit directed edge set or an "all reachable" assertion, needing no network at all, which is what makes a fully air-gapped estate describable.
The attestation itself is a real signature exchange. To claim it can reach a peer, the observing cell signs a canonical reach claim with its per-cell identity key and asks the target to countersign it over a dedicated inter-cell RPC; it reports the path reachable only after verifying the target's countersignature against that target's enrolled key. The responder refuses to countersign unless the claim names it as the destination and the observer's signature verifies. The forgery boundary is deliberately the per-cell key, so a compromised node cannot mint reach claims for cells it does not control.
Reachability is a five-state classification, not a binary: a target is
DIRECT, RELAYED (reachable through a relay hop), PENDING (scheduled or
air-gap (a window will open), DENIED (policy), or PARTITIONED (physics) no
path). The first two are servable; the rest each explain why not rather than
collapsing to a bare failure, and a cut can be named as an air gap, a policy
boundary, or an unknown.
Figure: layered attested reachability. Three sources are unioned: self, attested peering (an observed edge whose two-way countersignature verified, and which reads as absent once its lease lapses), and static operator config that needs no network. The result is classified into five states: the two green ones are servable; PENDING (amber) is a window that will open; DENIED and PARTITIONED (red) each name why the path is unusable. Ceiling: the countersign RPC is a core-runtime mechanism. It is fully wired end to end at the core, and it does not exist on the edge (see parity below). At the core, whether a resolver actually filters by this classification depends on a reachability provider being injected; unwired, it does not filter.
Table equivalent
| Class | Servable? | Meaning |
|---|---|---|
DIRECT | yes | A dialable path to the peer exists now |
RELAYED | yes | Reachable through a relay hop within the hop budget |
PENDING | no (yet) | Scheduled or air-gapped: a window is expected to open |
DENIED | no | A policy boundary forbids the path |
PARTITIONED | no | No path exists: a physical cut |
Inter-cell calls: how a call actually resolves
When a cell places a call to a peer, the fabric turns a (scope, service) request
into an authenticated RPC through an ordered path: rate-limit → policy gate →
endpoint discovery → optional residency-region filter → rank → dial. The policy
gate is fail-closed, if the gate cannot even be consulted, the call is
denied rather than allowed, and it decides, per candidate network, whether
mutual TLS is required.
Discovery prefers a TLS-enabled endpoint, and the channel it opens carries the caller's identity one of two ways: on a TLS channel the identity rides the mTLS handshake (no token is attached); on a cleartext channel the caller attaches its signed system token so the peer's interceptor admits the call, and it raises rather than send a blank token, so a missing credential fails loudly instead of leaking an unauthenticated call. The call runs with a bounded timeout and returns a typed error (peer-unreachable versus policy-denied versus no-endpoints) so the caller can decide fallback rather than guessing from a generic failure.
Across the deployment matrix
| Corner | How the fabric behaves |
|---|---|
| Single-cell | No peers to coordinate. Reachability is self only; there are no leases to contend and no peering to attest: the fabric is inert coordination that costs nothing until a second cell exists. |
| Multi-cell, cloud | The full plane: endpoint registration and reconciliation, TTL leases with epoch fencing, the peering graph, and countersigned attested reachability over mutual TLS. |
| Air-gapped | Reachability leans on the static source (operator config, no network); absent peers are modelled as absent edges, not failed peers; leases and epochs still coordinate the cells that are present locally. |
| Edge | The Edge Agent is one device, not a cell with peers, so it deliberately disclaims cross-cell dispatch. Its lease analog is a leased conductor role won by a capability-scored election, and it holds a per-peer attested reachability map: each peer's reachability is an Ed25519-signed observation verified against that peer's own member identity, with a freshness window. See parity below. |
Two-runtime parity
The fabric is a place where the core and the edge implement the same idea and must be held to one contract rather than allowed to drift. Where they stand today:
- Leases and failover. The core holds cross-cell leases with epoch fencing; the edge holds a leased conductor role with TTL renewal and a capability-scored election (model tier × reachable transports × work-zone headroom) so a healthy edge device, not a degraded one, wins. Same shape, a role that expires and is re-won, sized to each runtime.
- Both runtimes attest reachability, by different mechanisms. The core proves a peer reachable with a countersignature over mutual TLS; that countersign RPC does not exist on the edge. The edge instead collects peer-signed reachability observations and admits one only when the signature verifies against that peer's member identity and the observation is still fresh. Both are deny-closed in the same direction: with no attestation, no peer counts as reachable, and an expired attestation reads as absent rather than as last-known-good. A member advertising no capability cannot outrank one with positive evidence, so the election promotes on evidence rather than on silence.
- Attesting the caller. Where the core proves a peer's reachability with a countersignature, the edge proves a caller's identity at its gateway: a connection is refused unless the client certificate attests the exact claimed cell, client authentication is mandatory (a certless client fails the handshake, it does not arrive anonymous), and a device that proves no fleet-signed posture admits nothing. This closed a real gap, the cell already dialed mutual TLS while the edge had answered in cleartext.
See Core and edge for the parity discipline and the owed-work ledger this draws on.
Related topics
- The shapes the fabric composes, and the region → zone → rack → cell topology it classifies against: Deployment topology.
- Admitting a new identity to the fabric, join authorization, cell peering, and the identity chain: Joining the fabric.
- The replication scopes and cross-cell singletons the fabric's leases elect: Replication, DR & federation.
- How a public front-door URL, an entity address, or a service scope resolves to a target: Directory router.
- One product, two runtimes, and the parity ledger this page's owed work sits in: Core and edge.