Skip to main content

Edge operator runbook

This is the day-two guide: how to operate an already-enrolled Edge Agent as a recurring discipline, organised by the task you are doing rather than by the command. Where the Enroll a device and operate it air-gapped tutorial walks a device through its first sovereign lifecycle once, and the rest of this command reference documents each command in isolation, this page is the operator's manual for keeping a device healthy over its life. Every task links to the exact s9 command that carries it out.

Prerequisite

The device is already enrolled (or running fully sovereign) and s9 status and s9 doctor succeed. If not, start with the air-gapped tutorial.

The operating cycle​

Figure: the edge operating cycle. Steady state is a supervised daemon (edge, outlined) you check (health), drain (the signed journal and metering spool, green), and maintain (identity, cyan). Couriered bundles are admitted offline (amber, dashed, "no network"); an upgrade applies and returns the device to the cycle; decommission (red) ends it. The drain and admit steps are where an air-gapped device exchanges state with the fabric, on a courier's schedule, not a live link.

Table equivalent

TaskCommandsWhen
Keep it runnings9 run, s9 fleet-releases serviceContinuous (supervised)
Check healths9 status, s9 doctor, s9 probeRoutine / on symptom
Drain journal + spools9 audit export, s9 meteringRoutine / when a link returns
Maintain identitys9 refresh-cert, s9 rotate-delegation, s9 check-revocationOn cadence / before expiry
Admit couriered bundless9 revocation apply-bundle, s9 fleet-releases apply-bundle, s9 briefingOn courier arrival (offline)
Apply an upgrades9 fleet-releases, s9 migrateOn release
End of lifes9 decommissionOnce, irreversible

Keep the daemon running​

A permanently-deployed device runs the resident daemon under the host's init system, not by hand. The daemon sends heartbeats when linked, serves models locally, watches inboxes, and drains spooled work.

s9 fleet-releases service render     # preview the unit file first
s9 fleet-releases service install    # install the OS service that supervises s9 run
s9 fleet-releases service status     # confirm it is up

The service keeps s9 run supervised across reboots (systemd, launchd, or Windows SCM), and s9 fleet-releases service is where you render, install, and inspect it. To run a single heartbeat by hand, for a test, or a device that only checks in occasionally, use s9 heartbeat.

Check health​

s9 status                            # posture, identity, bindings, link state
s9 doctor                            # diagnose common problems (s9 doctor --fix to repair)
s9 probe https://node-a.system9.local --json   # reachability to a specific node

s9 status and s9 doctor are the two you run first on any symptom; s9 probe isolates a reachability problem to one endpoint. If a peer rejects the device for version skew, compare wire-schema hashes with s9 schema-hash (see Releases & versioning).

Drain the journal and the spool​

Work done while dark is spooled locally and rides home later. Two things accumulate and should be drained on a schedule, or whenever a link returns.

s9 audit export --sign --out /media/usb/run-record.json   # the signed reasoning/action journal
s9 audit verify /media/usb/run-record.json --device-key ./device.pub.pem
s9 metering status --json                                  # local metering spool state

s9 audit export turns the device's hash-chained journal into a signed, verifiable run-record, the proof of what the device did, valid offline against its enrolled public key. Metering records drain to a cell automatically when the daemon links; s9 metering shows the spool's state so you can confirm it is not backing up. On an air-gapped device, both travel by courier rather than by a live link.

Maintain identity​

A device's certificate and its delegation both have a lifetime; refresh them before they lapse rather than after.

s9 refresh-cert                      # new key + CSR, new certificate (private key stays on device)
s9 rotate-delegation --ttl-seconds 86400   # swap the delegation token, cert unchanged
s9 check-revocation                  # has this device's delegation been revoked?

s9 refresh-cert rolls the device certificate (the private key is regenerated on-device and never leaves it); s9 rotate-delegation swaps the delegation token without touching the certificate. Run s9 check-revocation as a routine check when the device is linked, it asks the operational endpoint whether the delegation still stands.

Operate air-gapped: admit couriered bundles​

An air-gapped device receives governed updates only through verified, offline routes, a signed bundle on removable media, checked against the trust roots the device already holds, with no network in the loop.

# Certificate revocations (revoke a compromised peer without a network)
s9 revocation apply-bundle --bundle-path /media/usb/revocation.bundle --dry-run
s9 revocation apply-bundle --bundle-path /media/usb/revocation.bundle

# Standing orders / field-manual briefings
s9 briefing            # see the discovery, catalog & field-manual reference

Use s9 revocation to apply a couriered revocation bundle (always --dry-run first), and the offline apply-bundle route under s9 fleet-releases for releases and model approvals. Set the transmit posture to silent on a sovereign device with s9 telemetry set off under it, the daemon has no heartbeats to send and bounds its local ticks instead. Peer status messages queue and drain through s9 inbox.

Apply an upgrade​

Upgrades are signed releases, applied ring-by-ring, atomic, and rollback-able. The Releases & versioning page is the model; the day-to-day commands are:

s9 fleet-releases check-update                 # is a signed release waiting?
s9 fleet-releases apply-update --require-signature   # verify + apply (atomic, rolls back on failure)
s9 migrate --dry-run                           # what store passes would run?
s9 migrate                                     # apply the outstanding store data passes

On an air-gapped device, replace apply-update with the couriered s9 fleet-releases apply-bundle. Always run s9 migrate --dry-run before s9 migrate, new code is designed to read an older database, but the store passes are what bring the local schema forward.

Decommission at end of life​

s9 decommission --confirm-device-id device:abc123 --receipt /media/usb/decom-receipt.json

s9 decommission destroys the device's identity, the inverse of enrollment, and irreversible. It writes a signed receipt so the retirement is provable off the device.