Configurable workers

A worker is a scheduled or long-running background task owned by a service. It is not a request handler, nothing calls it over the wire, and it is not a separate process. It is a function the service's runtime imports and drives on a loop: the TTL scanner that expires rows, the replication engine that converges peers, the fleet roll-out driver, the metering flusher. Workers are what keeps a cell's state correct while no one is looking at it.

Every worker is configurable: it is declared in the service configuration with an enabled switch and an interval_sec cadence, and it is bound to a governed capability that decides whether it is allowed to run at all. This page explains that model; the worker catalog lists all 302 of them.

What a worker is

A worker is declared under orchestrator.services.<service>.workers.<name> and carries a small, fixed config shape:

Key	Meaning
enabled	Declare the worker. If false, the runtime ignores it entirely: it is not registered, not scheduled, and does not tick.
autostart	Start the worker when the service boots.
interval_sec	Periodic cadence in seconds. null (or absent) means the worker is one-shot or long-running: it runs to completion or loops internally rather than being ticked.
capability	The governed capability the worker serves (exactly one). The capability registry decides whether the worker may run.
restart_on_crash / backoff_sec / jitter_frac	Respawn only after an exception (never after a clean return), with a jittered backoff.
kwargs	Per-worker tuning: batch sizes, thresholds, timeouts, policy blocks.

A clean, intentional stop, the tick returning without an exception, or a quiesce signal, does not cause a respawn. Only a crash does.

How a worker is declared, registered, and scheduled

The path from a line of config to a running loop is short and each step is guarded by a test that fails the build rather than failing at 03:00 in a startup log.

Figure: the worker lifecycle. A declared worker (cyan) whose target imports and whose capability is allowed becomes a scheduled, ticking worker (green); a worker declared enabled:false, or one whose governed capability is turned off fleet-wide, is held apart (amber) and never runs.

Table equivalent

Stage	From	Condition	Result	State
Declared	config	enabled: true	proceeds to resolve	candidate
Declared	config	enabled: false	ignored: never registered	held (amber)
Target resolves	declared	module:function imports and is callable	proceeds to gate	neutral
Capability gate	resolved	capability allowed by registry + policy	registered	neutral
Capability gate	resolved	capability denied fleet-wide	held: does not run	held (amber)
Registered	gated	worker spec built	schedulable	neutral
Scheduled	registered	autostart + interval_sec	ticks on cadence	neutral
Runs	scheduled	tick performs real work	records a healthy tick	runs (green)

Three build-time guards hold this path honest, so a broken worker fails a test rather than degrading silently in production:

• Every declared target must resolve to an importable, callable module:function. A stale target used to throw during import and then fall silent, and its enabled switch could not work in either position.
• Every bound capability must exist in the registry. A worker binding an unregistered capability is invisible to worker-policy classification. It can never be treated as high-consequence.
• No enabled worker may be inert, a tick whose body performs no call except logging records a healthy tick forever while doing nothing. See declared-but-inert workers.

How a worker is gated

A worker does not decide for itself whether it should run. It serves a capability, and the capability registry (config/config.d/capabilities.yaml) governs the capability. Each registry entry carries a consequence (normal or high) and a governed_quantity that ties it to the Alignment threshold model. Turning a capability off disables every worker bound to it, fleet-wide, for a high-consequence capability, doing so requires a net-potential-gain demonstration rather than a silent flip.

This is why the catalog lists a capability beside every worker: the capability, not the worker, is the unit an operator governs. One capability commonly gathers several workers (for example every metering flusher serves observability.metering), and turning that capability off stops all of them at once.

Behavior across the deployment matrix

A worker is code that runs inside a cell, so it inherits the cell's corner of the deployment matrix. The model is designed so the same worker set holds in every corner; the differences are in what a worker finds to do, not in whether it exists.

Corner	How workers behave
Sovereign / air-gapped	Every worker runs locally against the cell's own SQLite stores and object store. Workers that would reach outside: otlp_push, ct_submit, the node:telemetry_* and exchange_* reporters: either ship off by default or no-op when no egress is configured, so an air-gapped cell is not left throwing on a network it does not have. Nothing on the critical path (replication, TTL, integrity, fleet, governance) needs a round trip.
Cloud-enabled	The same workers run; the egress-facing ones can be enabled and find an endpoint to report to.
Single-cell	Cross-cell workers (anti-entropy, gossip membership, rebalance, failover detection) still run; with one member they simply find nothing to reconcile and return cleanly.
Multi-cell (elastic)	Those same workers carry the real load: convergence, membership, rehoming. Because a cell is a whole platform, its worker set boots with it.

Per-cell worker cost multiplies

Ephemeral cells launch roughly a hundred at a time for a single job. Every one of them stands up its entire worker set, 300-odd loops, at boot. That is the case where a needless high-frequency worker, or an inert one recording healthy ticks, is most expensive: the cost is paid a hundred times over. It is also why the sub-second scanners (ttl_scanner, eviction_scanner at 0.1s; retransmit_due, apply_inbox at 0.25s) are written to a strict per-tick time budget rather than a naive full pass.

Fleet-common workers

Six workers recur across most services rather than belonging to one. They are described once here and omitted from the per-service tables in the catalog to avoid repeating them nine times.

Worker	Capability	Enabled	Interval	Runs in	What it does
metrics_publisher	metrics.federation	Yes	5–10s	admin, compute, data, messaging, metrics, node, security, storage, system	Publishes the service's metrics into the federation plane.
meter_flush	observability.metering	Yes	30–120s	admin, compute, data, messaging, metrics, node, security, storage, system	Flushes buffered metering entries (behind a WAL) into durable metering storage.
cost_flush	observability.metering	Yes	60s	admin, compute, data, messaging, metrics, node, security, system	Flushes buffered cost/usage entries; the drain interval bounds how much usage is at risk from an ungraceful stop.
lifecycle_listener	core.lifecycle	Yes	5–30s	compute, data, messaging, metrics, node, security, storage, system	Reacts to cell/workspace lifecycle events for the service.
crypto_poll	replication.crypto	Yes	2.5s	admin, data, messaging, metrics, node, storage, system	Polls for replicated crypto material, preferring HTTP transport.
worker_policy_reconcile	core.governance	Yes	60s	data, metrics, system	Reconciles the service's live worker set against the authored Worker Policy.

Declared-but-inert and off-by-default workers

Not every declared worker runs, and honesty about which is part of the reference. There are three distinct cases, and they are not the same thing.

Off by default (configurable, intentional)

21 workers ship enabled: false. These are legitimate opt-in features, not defects. They are off because they need egress, dedicated hardware, an operator mandate, or an authorization the default deployment should not assume. They include the local-model workers (gguf_warmup, gguf_idle_evictor, model_build_pipeline), external-egress reporters (otlp_push, ct_submit, ct_intelligence), the offense-capable Security workers (mtd_rotation, autonomous_red_team), the opt-in ML worker (drift_retrain), the event-driven session-revocation path (session_revocations, session_revocation_consumer), the forward-secret KEK ratchet (kek_forward_secret_ratchet), and the DLP scanner (dlp_scanner). Turning one on is a supported configuration change.

Disabled because inert (kept off so they cannot lie)

A sweep over the declared workers found a small number whose tick performs no work, the body computes nothing that escapes its own logging arguments. Five are pinned enabled: false so that re-enabling one without implementing it fails the build rather than silently resuming a no-op schedule that reports healthy ticks:

• data:meta_intel_index_sync, data:meta_intel_quality_compute, data:meta_intel_enrichment, data:meta_intel_usage_rollup, the four data.meta_intelligence workers; the capability had no live provider.
• data:replication_sync, a stub whose own docstring said it was disabled while the config had once enabled it.

(storage:service_group_migration was in this set and left it by being implemented, not by being re-enabled blindly, it now builds a transport and performs the migration.)

Declared, and gated rather than hollow

One worker is worth naming: security:device_attestation is declared enabled: true, but it performs no real firmware attestation. Rather than record healthy ticks while doing nothing, its tick refuses to run unless an operator explicitly acknowledges that stub state in configuration (security.tasks.device_attestation.stub_acknowledged), logging an error otherwise, so the scheduler sees a clean no-op instead of a tick that looks like attestation. Real firmware attestation is separate work.

The manager the worker resolves is not gated, deliberately: the IoT posture surface consumes it live, and gating the accessor would refuse a working surface to fix a hollow loop. The inert tick is the defect, so the tick is what fails closed.

A catalog is not an execution list

Separately, the Data service carries a typed ScaleOut worker registry that reads like an authoritative list of what runs in the background: entries with names, module paths, and intervals. Nothing in production imports it, and several of its entries are declared in no service configuration. It is evidence of intent, not of execution.

What production does publish is a different thing: at startup each service registers the workers it actually started, and the names it holds a lease on, into the cell runtime. The health surface derives each worker's status from that live registry, and reports unknown when the registry itself is absent, which is a different fact from "the worker is not running". Collapsing those two is what once let a node with no scheduler answer configured.

A reader should treat the catalog on this site, derived from the actual workers: declarations the runtime reads, as the source of truth, and the typed ScaleOut registry as a design artifact.

Two service shells with no workers

Finally, two services (awareness_engine and fabric_health) are declared enabled: true in config but carry no module and no workers; they are placeholders. The fabric-health work that a reader might expect there actually runs as the Admin service's fabric_health_agent (see the catalog).

In the console

An operator does not toggle these workers one at a time. They are governed from the Worker Policy console at /core/worker-policy, where a capability manifest collapses the per-worker toggles into a handful of operator-meaningful capability switches. From there an operator authors a signed policy at a chosen scope, previews which workers a change would start or stop, applies it under the governance gate, and watches the fleet attest what it actually ran. The full surface (including break-glass overrides and drift) is documented in Worker Policy.

Related topics

• Worker Policy, the console and signed engine that turn these workers' capabilities on and off across a fleet.
• The worker catalog, all 302 workers, grouped by service.
• The cell model, what a service and a cell are, and why per-cell cost multiplies for ephemeral cells.
• Governance, the capability registry and Worker Policy that gate every worker.
• Alignment, the net-potential-gain and governed-quantity model a high-consequence capability rides.