Fleet Releases
Getting software onto one device is easy. Getting a signed, verified version onto thousands of devices across contested links, without breaking the ones that matter, is the problem Fleet Releases solves.
The shape of the answer is familiar to anyone who has run a serious fleet: signed artifacts, cohorts, staged rings, health gates, automatic rollback. What is different here is what the signatures are made of, what proves a build was not swapped, and how a cohort is defined.
This page is the mechanism. For where the Edge Agent sits in the product, see core and edge; for the transports a release travels over, see edge and fleet management; for the console surface and its capability map, see config deployment.
The objects
| Object | What it is |
|---|---|
| Package | The release artifact, with a manifest describing it |
| Track | The channel a package is published on, with version pins |
| Bay | A named cohort of devices, defined as a query |
| Rollout | A planned advance of a track's pin across bays, ring by ring |
| Ring | One stage of a rollout, with its own soak period and health gates |
| Install token | A single-use-bounded credential that enrolls a new device |
A device's target version is resolved by walking one chain: device to bay to track to pin. There is one answer, computed the same way everywhere.
Signing: post-quantum, and hostile to downgrades
A release signature is the root of the whole trust story, so it is worth being exact about what is signed and with what.
Three signature algorithms are supported, and the choice is explicit in the payload:
| Algorithm | Behaviour |
|---|---|
| Hybrid Ed25519 + ML-DSA-65 | Both signatures must verify. Classical security today, post-quantum security against harvest-now-decrypt-later |
| Ed25519 only | Classical only. A populated post-quantum field on this payload is rejected as a downgrade attempt |
| ML-DSA-65 only | Post-quantum only, for deployments held to a pure post-quantum standard |
Two details make this stronger than a version string in a header.
A missing post-quantum library is a refusal, not a fallback. If a verifier cannot check the post-quantum half of a hybrid signature, the result is invalid, with a reason saying exactly that. It does not quietly verify the classical half and call it good. That silent degradation is the single most common way a hybrid scheme becomes a classical scheme in production.
Signatures are domain-separated. A fixed, version-tagged separator is prepended before hashing, so a manifest signature cannot be replayed as a package signature even by an attacker who controls the input bytes. The separator carries a version, so a future scheme change is a new separator rather than an ambiguity.
Verification itself never raises on an invalid signature. It returns an explicit invalid result with a reason. Only a malformed payload is an error, because "this signature is wrong" and "this is not a signature" warrant different handling.
The transparency log: proving a build was not swapped
Signing proves who produced a package. It does not prove that the package a device received is the same one everybody else saw. That is a different property, and it is the one that catches a targeted attack.
Every package and signature pair is recorded in a tamper-evident, append-only log, so a device, an auditor, or the reconciler can prove a build was witnessed and has not been silently substituted for one particular target. The log leaves are built through the same canonical encoding used to submit and to verify, so the bytes cannot drift between the two operations.
The transparency client and the certificate-transparency-backed path are present, and a durable leaf store exists: it persists each append and replays the ordered leaves on a cold start, so the signed tree head stays monotonic across restarts. The release-transparency path does not construct it. The durable store is wired on the key-compromise path, while a release log built without one keeps its leaves in memory and resets its tree head on restart.
So the mechanism is real and proven elsewhere in the product; what is owed is its use here. Treat release transparency as partially wired rather than as an operating guarantee to put in a contract today.
Bays are queries, not lists
This is the design decision that pays off most over a fleet's life.
A cohort is not a maintained list of device identifiers. It is a query over device facts, expressed as a set of clauses that must all match, any of which may match, and which must match none:
all: tag in [env=prod, site=fra1]
profile in [agent, operator]
os = linux
arch in [x86_64, aarch64]
none: tag in [maintenance]
A device enrolled tomorrow that matches the query is in the cohort tomorrow, with nobody editing anything. A device tagged for maintenance leaves every cohort that excludes that tag, immediately and everywhere, which is the behaviour you want at three in the morning.
Static membership lists rot. They are correct on the day they are written and progressively wrong afterwards, and the drift is invisible until a rollout misses the devices that needed it most.
Rollout: rings, soak, gates, rollback
Figure FR1: the rollout lifecycle. The transition worth noticing is RUNNING to ROLLING_BACK, which is automatic on a critical health-gate failure rather than a decision a human has to be awake to make. The legal transitions are enforced in two places, the durable store and a pure restatement of the same graph that guards a transition before any write is attempted, so an illegal advance is refused rather than half-applied.
Each ring moves through its own states: waiting, entering, soaking, and then promoted, failed, or skipped. Soaking is where the gates run.
The health gates
While a ring soaks, gates are evaluated on every worker tick. Three are built in:
| Gate | Fails when |
|---|---|
| Failure ratio | The proportion of successful applies falls below the configured threshold |
| Consecutive failures | Too many failed applies land in a row, in time order |
| No critical drift | Any critical-severity drift event is recorded for the ring's devices since the ring started |
Beyond these, a rollout can carry custom gates as expressions evaluated against the ring's own data, each with its own identifier and severity, so a team can gate on a condition the platform's authors never anticipated.
A single critical gate failure fails the ring and rolls the rollout back. Not an alert, not a pause awaiting acknowledgement. The default is to undo.
An operator can override a gate, and doing so requires a signed reason and is recorded. The override exists because sometimes the gate is wrong; requiring a signature and an audit entry exists because sometimes the operator is.
Manual control
Start, pause, resume, cancel and roll back are all explicit operations on a running rollout. Pausing leaves the fleet where it is rather than reverting, which is the right behaviour when you want to investigate before deciding.
Holds: freeze, thaw, and blackout windows
Two independent mechanisms stop a device receiving an update, and they exist for different reasons.
A freeze is a hold on a target. It applies to a single device or to an entire bay, and it lasts until an administrator thaws it or until its expiry passes. The device-level freeze is checked before the bay-level one, so a single device can be held without touching its cohort. Expiry is handled on read, so a stale freeze is never observed. Freezes are how you say not this one, not yet, without editing a cohort or cancelling a rollout.
A blackout window is a hold on time. It is a recurring interval, expressed as a schedule plus a duration, during which the rollout worker refuses to enter new rings and the desired-state resolver tells devices to hold. This is the change freeze every regulated and operational environment already runs informally: quarter end, a maintenance weekend, the hours around a known-critical operation.
The difference matters. A freeze protects a device from a change. A blackout window protects a time period from all changes.
Freeze and thaw are served from an in-memory registry today, behind an interface that a durable store can replace without touching the resolver. A freeze does not currently survive a restart. Plan around that.
Desired state, and what a device asks for
A device does not receive a push it cannot refuse. On its own update tick, the agent asks the cell what its desired state is, and gets one of four answers:
| Answer | Meaning |
|---|---|
| No-op | Already on the target version |
| Update | Fetch and apply the named target package |
| Hold | Frozen by an administrator, by a blackout window, or by the rollout engine |
| Decommission | This device is retired |
The resolver is pure: it reads the device to bay to track to pin chain and computes an answer with no side effects, so it is safe to call concurrently and produces the same answer for the same facts. A device that asks twice gets the same answer twice.
Package bytes are then streamed to the agent in chunks, so a large artifact crossing a slow link is a resumable stream rather than one enormous request.
Firmware updates ride the same command queue the agent already polls, seeded as a command rather than as a special path, so the agent runs its standard download, verify, swap flow.
Version skew between a device and its cell
A rollout is not instantaneous, so for a window of time a fleet runs mixed versions against one cell. The platform admits that skew only where it is provably safe, and the rule is narrow on purpose.
A cell accepts a device whose wire schema is its own, or one the cell's checked-in compatibility window names, or one whose own advertised window contains the cell's. Entries reach that window through a tool that compares descriptors before an additive change and refuses any change that removes, renumbers, retypes or renames a field, or alters a call signature. Additive only, checked mechanically rather than asserted.
No prior schema is currently accepted. The first entry appears at the next additive change to the wire format. Until then a device and its cell must match exactly, which is the correct default for a product that has not yet shipped an incompatible pair.
Three limits, because they decide rollout order rather than being footnotes:
| Transport | Skew behaviour |
|---|---|
| gRPC, HTTPS, DoH, WebSocket | The device advertises its window in a header. Skew works in both directions |
| Messaging bridge | Carries no such header, so an older device is admitted and a newer one is not. Upgrade the cell first, or accept that newly-upgraded devices on this path wait |
| Radio | Carries a short tag rather than a full advertisement, so skew is one-directional |
And one the tooling cannot catch: a descriptor comparison sees shape, not meaning. A field that keeps its number and type but changes what it means passes the check and breaks the peer. That is a review matter, not a mechanical one, and no gate will catch it for you.
Drift: severity that means something
Drift is the continuous comparison of what a device is running against what it should be running. The classification is where the value is, because "1,400 devices have drifted" is not actionable and three critical ones are.
| Severity | What produces it |
|---|---|
| Info | Behind on a non-mandatory bump, a patch within the same major and minor version, with no policy violation and healthy attestation |
| Warn | Behind on a mandatory bump across a major or minor boundary, a configuration-hash mismatch, or stale attestation |
| Critical | Attestation is invalid on a track that requires attestation, or the device is running a yanked package |
That last clause is the one to notice. When a package is yanked, every device still running it becomes a critical finding automatically, with no separate recall process to run and no list to reconcile. The yank is the recall.
Every device also carries a timeline: install and update attempts, drift detections, and commands issued, merged chronologically into one feed, so diagnosing a single misbehaving device does not mean correlating four views by hand. (The audit-record source in that merge is a placeholder pending a later wiring phase.)
Enrollment
A new device joins with an install token: bounded by a maximum number of uses and a required future expiry, stored only as a hash, and returned in plaintext exactly once, at issue. There is no path to read it back, which is the property that makes it safe to treat as a secret.
Redemption is atomic with the device record and enrollment, so a token cannot be half-spent.
Declarative operation, and GitOps
The whole fleet configuration, bays, tracks, pins and rollouts, can be expressed as YAML manifests and applied from a command line or driven from a repository webhook.
The reconciler has one property worth calling out: diff is pure read. Nothing is materialised until apply runs, so a plan can be produced and reviewed with no risk of partial application. On apply, bays and tracks are created or updated in place, and new rollouts are planned.
Mutating an in-flight rollout is refused, and reported. A running rollout is not a declarative object, because rewriting a rollout that is halfway through its rings has no safe meaning. The entry is reported back rather than silently applied or silently dropped.
What is wired, and what is owed
Stated plainly, because a fleet-management claim is easy to over-sell:
| Area | Status |
|---|---|
| Signing and verification, hybrid and post-quantum, domain-separated | Present, with downgrade rejection |
| Bays as queries, tracks, pins, desired-state resolution | Present |
| Rollout state machine, rings, soak, built-in and custom gates, automatic rollback | Present |
| Blackout windows, drift classification, device timeline, install tokens | Present |
| GitOps diff and apply, with in-flight rollouts refused | Present |
| Transparency log | Client and certificate-transparency path present. A durable leaf store exists and is used on the key-compromise path; the release log does not construct one, so its leaves are in memory |
| Freeze and thaw | In-memory; does not survive a restart |
| Signing-key rotation | Planning surface, plus trust-root publication. The rotator materialises the new keypair and returns a transition plan without mutating anything, and a publisher pushes a trust root to devices on the lane the edge reads. Re-signing active manifests and retiring the old key from the verify-only set after the grace window are owed |
| Package object storage | Local reader ships; the cloud object-store reader is an interface awaiting wiring |
Related topics
- The transports a release crosses, including air-gapped bundles: edge and fleet management.
- What an Edge Agent is, and what it is not: core and edge.
- The console surface, its routes and its capability map: config deployment.
- The signing and key-custody plane these release keys live in: signing and cryptography.
- Every command the Edge Agent exposes on a host: edge command reference.