Skip to main content

The service orchestrator

A cell is not nine processes started by a script. It is nine services started, supervised, gated and drained by an orchestrator, and most of the properties the rest of the platform relies on are actually properties of this layer.

Each service declares itself​

A service does not get wired into a startup list. It ships a manifest naming itself, the entry point that runs it, and what it depends on.

name:        admin
runner:      the callable that runs the service
depends_on:  [security]

Dependency order is therefore derived from declarations rather than maintained by hand, which is why adding a service does not mean editing a boot sequence that someone will get wrong later.

Startup is gated on readiness, not on having been launched​

The distinction that matters: a dependency is satisfied when it is ready, not when its process has been created.

Readiness is an explicit contract. A service participates by exposing an out-of-process readiness adapter in one of two recognised forms: a blocking wait_ready that returns on success and raises on failure or timeout, or a non-blocking sample returning an ok flag with named checks, where the orchestrator owns the polling loop and the deadline.

There is exactly one authority on whether a service has an adapter, so "is this service gated" is not a per-caller judgement.

One deliberate exclusion

Every service also exposes a readiness snapshot, and it looks compatible with the adapter contract because it returns a similar shape. It is deliberately not accepted as a third form. A probe used for gating has to be an out-of-process check with a deadline the orchestrator controls; a snapshot is an in-process self-report. Treating them as interchangeable is how a service ends up gating on its own opinion of itself.

Phase gating: a half-provisioned cell runs only what it may​

A cell that is still being set up must not run everything a complete cell runs. The orchestrator resolves the setup phase, one of uninitialized, bootstrap, awaiting-admin or complete, and consults a per-service allow-list for that phase.

The phase reader is fail-closed: malformed state resolves to the most restrictive phase rather than being treated as complete.

This is what makes the setup gate real rather than cosmetic. A cell is never live but ungoverned, because the services that would make it live are not permitted to run until the phase permits them.

Failure handling: backoff and per-service circuits​

Figure: only a non-clean exit counts as a failure. Clean exits do not drive backoff, so a service that finishes its work is not treated as crashing. Repeated failures open a per-service circuit that suspends restarts until it expires, which is what stops a restart storm.

Table equivalent

EventTreatment
Clean exitNot a failure. Does not drive backoff or open a circuit
Non-clean exitCounted as a failure. Restart after a configured backoff
Repeated failuresA per-service circuit opens, with an expiry, suspending restarts
Circuit clearedExplicitly clearable per cell and per service, so an operator can retry without waiting

The clean-versus-crash distinction is the load-bearing one. A supervisor that treats every exit as a fault will fight with any service that legitimately completes, and a supervisor with no circuit will turn one failing dependency into an estate-wide restart storm that looks like a platform outage.

Supervision while running​

ComponentWhat it does
WatchdogReconciles desired state against actual, so drift from the intended service set is corrected rather than noticed
JanitorReaps what has been left behind
DiscoveryFinds the services present rather than relying on a static list
IPCThe control channel, including clearing a circuit
Policy and lifecycleThe rules for what may start, stop and restart, separate from the mechanics of doing it

Shutdown is ordered, not simultaneous​

Draining runs in a deliberate order, compute-heavy first and system last, so work in flight has somewhere to land while the services it depends on are still up. A cell drain is its own orchestrated operation rather than a signal broadcast to every process at once.

Startup recovery then reconciles in-flight work after a restart and scans for orphans left by a crashed or evicted worker, so a restart resumes rather than resetting.

Why this is in Platform Architecture​

Three claims made elsewhere in this documentation are actually claims about this layer, and it is worth being able to point at where they come from:

  • "A cell is never live but ungoverned" is phase gating with a fail-closed phase reader.
  • "A cell boots with no external dependency" requires a dependency graph resolved locally from declared manifests, with no coordinator to ask.
  • "A restart resumes rather than resets" is startup recovery plus the orphan scan.