Worker Policy
Every cell runs a set of background workers, the loops that expire rows, converge peers, drive fleet roll-outs, flush metering. Worker Policy is how an operator governs that set as declarative desired state rather than by editing config files on individual cells: an authorized administrator authors a signed policy, previews its impact, applies it under the governance gate, and (when an incident demands it) opens a time-boxed break-glass override on a single cell. The engine behind the console then resolves that policy into each cell's effective worker set, holds it there against drift, and makes every cell attest what it actually ran.
This is the exemplar of the platform's operator model: a console surface that authors intent, a signed artifact that carries it, and a per-cell engine that turns it into running state and reports back. The workers reference describes what a worker is and how it is gated by a capability; this page is the control plane that decides which capabilities are on, for which scope, and proves it stuck.
The whole story: console to attestation
Figure: Worker Policy end to end. The operator authors intent in the console (neutral), sees a dry-run impact preview (cyan) before any write, and applies through the system-token-gated App-BFF. The governance gate refuses (red) an apply that would disable a high-consequence capability unless a net-potential-gain demonstration clears it; an accepted policy is signed, version-gated, and persisted (green), then resolved per cell, reconciled against drift, and attested back to a fleet drift dashboard. A break-glass pin (amber) is a time-boxed, signed override on one cell that the resolver applies at highest precedence and that heals away on expiry.
Table equivalent
| Stage | Where | What happens | Failure / guard |
|---|---|---|---|
| Author | Console → Worker Policy | Operator sets scope, capability switches, optional mandate | Form validation; scope ladder rules |
| Preview impact | Console (dry-run) | Resolve current vs proposed for a representative cell; diff which workers start/stop and which params change | No write performed |
| Apply | App-BFF → control plane | Classify consequence, run the value gate, verify + version-gate + persist | Disabling a high-consequence capability is hard-gated at enforce |
| Sign | Control plane | Bind the frozen policy spec to its issuing entity (domain-separated Ed25519) | An unsigned or mis-scoped spec is rejected |
| Resolve | Every cell | Merge the scope ladder into an EffectiveConfig with per-key provenance | Mandates clamp lower scopes; pins against mandated capabilities refused |
| Reconcile | Every service | Diff desired vs observed; re-assert managed workers; revert hand-edits | Per-worker apply error is logged and retried, never crashes the cell |
| Attest | Every cell | Sign a snapshot of resolved digest + observed state; ship cell-to-cell | Drift surfaces on the dashboard; attestation domain tag prevents replay |
In the console
Worker Policy has a dedicated operator surface in the Core System console at
the route /core/worker-policy (labelled "Worker Policy, worker execution
governance" in the console nav). It is a single page with five sections rather
than tabs:
- Live policies, every authored policy across the scope ladder, each showing
its scope (
org/node/service_group/zone/rack/cell_profile/cell), version, and issuer, with a mandate badge where set. Each policy can be retired. - Author a policy, a form for the policy id, version, scope and scope id, the target selector, an org-only Mandate toggle ("descendants may only tighten"), and a repeatable list of capability switches. Two actions: Preview impact (the dry-run) and Apply policy.
- Impact preview, the dry-run result: which workers start or stop and which params change. When the change is high-consequence, the preview warns that the net-potential-gain gate will hard-gate the apply at enforce mode.
- Fleet drift, per-cell attestation digests, a config-hash agreement badge, and a count of cells in drift (polled).
- Break-glass pins: active pins (cell, target, issuer, expiry), an Open break-glass pin action (cell, target kind, reason, ISO-8601 expiry), and a Release action per pin.
The Worker Policy console page performs no client-side permission check, it
does not read the caller's permissions and renders unconditionally. Its authority
guard lives entirely at the node App-BFF, which proxies to the system-private
control plane system-token-gated, and at the value gate on apply/break-glass.
That is a deliberate design (the sensitive decision is a system-service one), but
it means there is no frontend capability string to cite for this surface, unlike
the Fleet Releases
surfaces (which self-gate on system.fleet.* capabilities) and the unified
Config Policy surface (which self-gates on
system.cells.inspect / system.cells.manage, rendering a read-only banner
without write).
Intent, not one toggle per worker
An operator does not switch workers one by one. A capability manifest turns several hundred per-worker toggles into a handful of operator-meaningful capability switches, and a pure boolean evaluator derives each worker's enablement from the capability state. This is the same capability model the workers reference describes from the worker's side: a worker serves exactly one capability, and Worker Policy is where an operator turns that capability on or off for a scope.
A policy is a signed, frozen spec carried at one of seven topology scopes, organization, node, service group, zone, rack, cell profile, or a specific cell, and bound to the entity that issued it. An org-scope spec may be a mandate: it fixes the values it sets so lower scopes may only tighten, never loosen, them.
The scope ladder and mandates
Each cell resolves its own effective config by merging the ladder from low to high precedence, recording per-key provenance so the console can explain where every effective value came from:
| Precedence | Layer | Notes |
|---|---|---|
| 0 (lowest) | Service defaults | Manifest defaults + shipped worker.enabled |
| 1 | Org policy | Fleet-wide baseline; a mandate here becomes a bound |
| 2 | Node policy | Per-node override |
| 3 | Service-group policy | Per-role override |
| 4 | Zone policy | Per-availability-zone override |
| 5 | Rack policy | Per-rack override (including an isolated / air-gapped rack) |
| 6 | Cell-profile policy | Per-managed-profile (deployment-shape) override |
| 7 | Cell policy | One exact cell |
| 8 (highest) | Cell break-glass pin | Time-boxed, cell-scoped, audited |
The same seven topology scopes (org … cell) are the axis of the unified
Config Policy plane, worker enablement and typed config
resolve through the one engine, narrow-wins, with no coordinator.
Mandates are bounds, not just a layer. A lower layer that tries to cross a
mandated value is clamped back and the attempt is recorded as a mandate_clamp
provenance entry. A capability locked by mandate cannot be moved by any lower
scope, and a break-glass pin against it, or against a worker it governs, is
refused. You cannot break-glass your way out of an org mandate.
Apply is governed, drift is reverted
Applying a policy is not a raw write. The control plane classifies the action's
consequence: disabling a high-consequence capability, for example
audit.tamper_evident or replication.crypto, is classified as a
high-consequence fleet change and is hard-gated at enforce mode by the
net-potential-gain gate;
everything else is the ordinary apply. An accepted apply is verified,
version-gated (so a stale write cannot clobber a newer one), and persisted.
On every cell a per-service reconciler then makes the policy stick. Each tick
it resolves the desired state, observes the running workers, diffs the two, and
surgically re-asserts only the managed workers that changed, applying param and
interval changes live and restarting only on a target or code change. Re-asserting
the same state is a silent no-op; a hand-edit or out-of-band change is drift
and is reverted on the next tick. This reconcile loop is itself a fleet-common
worker, worker_policy_reconcile, serving core.governance on a 60-second
cadence, listed in the
workers reference.
Break-glass: a governed escape hatch
A break-glass pin is the first-class, audited, time-boxed way to let a single cell diverge legitimately (incident response, a one-off fix) instead of an untracked config edit. Issuing a pin runs an up-front mandate guard (a pin against an actively mandated capability, or a worker it governs, is refused at issue), then stores and audits it. The resolver applies active pins at the highest precedence; when a pin's clock runs out it is pruned and the cell heals back to governed state on the next reconcile tick. Issue, expiry, and release each emit an audit event.
Attestation: the fleet proves what it ran
Worker Policy is deliberately not write-only. Every cell both receives policy
and attests what it actually resolved and is running: a signed
EffectiveConfigAttestation snapshot carrying the resolved config digest (a
cheap equality check that two cells agree on desired state for the same policy
version), the observed worker state versus desired, and an overall drift
flag. Attestations are shipped cell-to-cell by an HTTP pull-poll, not
replication, because the audit chain is not replicated, and they feed the
console's Fleet drift dashboard. Attestation signing uses a distinct domain
tag from policy signing, so a policy signature can never be replayed as an
attestation.
Across the deployment matrix
| Corner | How Worker Policy behaves |
|---|---|
| Single-cell | The cell authors (if it is the control-plane node), resolves, reconciles, and attests locally; with one member there is nothing to disagree with, so drift is trivially none. |
| Multi-cell (elastic) | Each cell resolves its own effective config from the shared scope ladder and attests back; the pull-poll aggregates the fleet's real state. Ephemeral cells resolve policy and stand up their worker set at boot, so a hundred-cell burst inherits the same governed set. |
| Sovereign / air-gapped | Signing and verification are offline (Ed25519 against material the cell holds); resolution and reconcile need no network; attestations move cell-to-cell with no coordinator and no phone-home. |
| Cloud-enabled | The same engine and the same governance; connectivity only changes how quickly attestations propagate, not whether policy is enforced. |
Related topics
- Configurable workers, what a worker is, how it is declared, and how a capability gates it.
- Config Policy, the same signed engine governing typed configuration for the nine sections and services, with tiered delegated authz and a config-change audit.
- Config deployment, the declarative fleet-config cycle, and how applying a Deployment preset authors Worker Policy for you.
- The Net Potential Gain gate, the value gate that hard-gates a high-consequence policy change.
- Capability & approval, the capability and consequence model policy switches ride on.