Disaster recovery
Losing a site should be a rehearsed transition, not an improvised one. System9 treats disaster recovery as policy plus practice: a workspace declares where it may recover to, the estate can assess on demand whether it is actually ready to fail over, a scheduled drill proves the plan without cutting over, and the real cutover runs as a fail-closed saga, with a matching path back. It is bounded by the same residency discipline as everything else: a failover stays within the pinned level, and a recovery that could only land off-jurisdiction is a refusal, not a degradation.
Policy: where a workspace may recover to
DR is declared per workspace, spanning its service-group planes. A policy names a primary region, an ordered list of standby regions (preference order), RPO and RTO targets, a failover mode (manual, automatic, or drill-only), and the standby mirrors that back it. The policy is strict data, malformed policy fails loudly rather than silently under-protecting a workspace.
Readiness: assessed on demand, not assumed
Before a failover is a good idea, the estate can ask whether it would actually succeed. Readiness is evaluated per standby, in preference order, first eligible wins, and a standby qualifies only with zero blockers:
| Check | A standby is blocked when… |
|---|---|
| Residency boundary (checked first, fail-closed) | The standby region is outside the workspace's pinned residency level |
| Mirror state | The standby mirror is not active |
| RPO | Replication lag exceeds the RPO target, or there is no lag data to judge |
| Region health | The standby region is not currently healthy |
| Capacity (N+1) | The standby region cannot absorb the load |
At the workspace level, readiness is a worst-plane gate: the workspace is failover-ready only when every plane is ready. "Ready" is a computed answer about the estate as it actually is, not a checkbox set at configuration time.
Drill: recovery you have practised
A DR drill derives the real failover plan and runs it dry, it never cuts over. It projects the RTO from per-step estimates and confirms the chosen target meets its RPO, and the drill passes only when the workspace is failover-ready, RPO-compliant, and within its RTO target. A policy set to drill-only permits rehearsal while forbidding a real cutover, so you can prove readiness on a production workspace without risking it.
The failover and failback paths
A real cutover runs as an ordered, fail-closed saga. The first step is a fence that must succeed, if the old primary cannot be fenced, the failover aborts rather than risk two primaries (split-brain). Only then does a standby get promoted, topology reconcile, and the primary pointer flip.
Figure: the failover and failback state flow. Failover is a fail-closed saga: fence → promote → reconcile → flip, and it aborts if the primary cannot be fenced (the split-brain guard). Failback is the symmetric return, gated on completeness at the return target. Ceiling: the promotion primitives are symmetric in both directions, but failback's estate-wide resolver convergence and run re-homing are owed. See the note below.
Table equivalent
| Step | Direction | What happens | On failure |
|---|---|---|---|
| Fence primary | Failover | Fence the old primary; a verified fence epoch is required | Abort: never proceed with an unfenced primary |
| Pre-cutover gates | Failover | Clean-room copy-safety and snapshot-completeness gates run fail-closed | Cutover refused |
| Promote standby | Failover | The elected standby is promoted to primary | Saga halts, marked partial |
| Reconcile topology | Failover | Topology is reconciled to the new primary | Saga halts, marked partial |
| Flip primary | Failover | Primary pointer flips; resolvers, observers, and in-flight runs re-home to the new primary | Best-effort; never re-raises into the completed promotion |
| Fence + promote back | Failback | Symmetric return to the original site, gated on completeness at the return target | Aborts fail-closed |
When a failover completes, the estate does not just promote a standby, it re-homes: resolvers repoint at the new primary, and in-flight workflow runs stranded on failed cells are re-homed to the current primary cell so work resumes rather than being lost. Completeness digests guard the cutover: a content digest (a deterministic hash over the workspace's converging tables) must match between the failed scope and the surviving peer, so a cutover cannot silently complete on an incomplete copy.
Failback exists and is reachable, and its promotion primitives are structurally symmetric with failover (the same net-gain guard and a completeness gate on the return target). But the estate-wide follow-through on failback is owed: the home-service-group restore and the reverse resolver/shard/observer/run re-homing are not yet wired, and the clean-room copy-safety gate runs on failover only. Failback is not a full mirror of failover today: the return promotes, but the surrounding convergence is in progress.
Standby election stays in-jurisdiction
Choosing where to fail over is a placement decision, and it obeys residency. The DR placement resolver elects a new primary only from cells that already hold the shard's replica, intersected with the live cells, it never invents a placement, only repoints at a cell that already has the data. When a residency boundary is in force, replicas outside the pinned level are filtered out; if a shard's only live replicas are all off-jurisdiction, it is skipped and recorded as a refused boundary crossing rather than redirected out of jurisdiction. This is the DR-layer expression of the platform-wide rule that failover stays within the pinned residency boundary.
The residency-aware refusal in DR standby election and readiness is built and tested, but the residency input is not yet passed through on the production DR wiring, so today the pinned boundary on a failover is held by the platform's general placement/egress residency enforcement, not yet by a DR-specific residency filter. The two agree by design, and the DR-layer filter is roadmap: residency on a failover is enforced today, by the general placement and egress path rather than inside the standby election.
Clean-room disaster recovery
A clean room adds its own fail-closed gate to failover: a standby is a valid DR target only if the target cell is cleared for the room's compartment, and the room's copy-safety suite, differential-privacy-budget continuity, output-review-posture continuity, and egress-on-replication consent, is satisfied at the promotion target. This gate is real and runs before any clean-room cutover.
A room runs on one cell at a time, so its recovery is a cutover between runs.
There is no mid-run re-placement executor. No path moves a room to another cell while a computation is in flight, carrying that run's epsilon spend, output-review posture and egress admission with it. A run in progress when a cell is lost is a run to restart on the recovered room, not one that migrates.
The clean-room admission gate applies on failover, not on failback. The return promotes under the net-gain guard and the completeness gate, and the room event is recorded, but the compartment, epsilon, output-review and egress checks that vet an outbound target are not re-applied to the return target. Where a room is compartmented, verify the home target's clearance before failing back rather than relying on the cutover to refuse.
Startup recovery after a restart
A cell restart is a smaller recovery of the same kind. On boot, startup recovery reconciles in-flight compute: a database health check, reaping stale kernels, marking crashed jobs failed with a restart-recovery reason, reloading the job queue, and warming the pool. An orphan scanner removes kernel working directories whose process is dead (guarded against PID reuse). Work that was in-flight when the cell went down is reconciled to a known state rather than left dangling.
Across the deployment matrix
| Corner | How DR behaves |
|---|---|
| Single-cell | Startup recovery reconciles in-flight work after a restart; there is no peer to fail over to |
| Multi-cell, cloud | Full failover/failback across cells and regions, bounded by residency and readiness |
| Air-gapped | The convergence and completeness machinery runs over store-and-forward transports; an absent peer is modeled as absent, not failed |
| Clean room | Fail-closed readiness and admission gating on cutover today; cross-cell mid-run recovery is in progress |
Related topics
- Data residency, the constraint that bounds where a failover may land.
- Replication, DR & federation, the convergence engine and completeness machinery underneath DR.
- Cloning, the mobility machinery DR shares for archive, restore, and standby copies.
- The cell model, distributed leases and peer coordination that survive a cell going away.