When the meter cannot report
Most platforms have one answer to "what if it cannot phone home?", and the answer is usually either it stops working or it silently stops counting. Both are bad, in opposite directions: the first punishes a customer for a network they do not control, the second is revenue loss dressed up as resilience.
System9 is built for deployments where the link is a state, not a requirement, so this case is designed rather than handled. This page walks the scenarios an operator will actually hit.
First, three different outbound paths
These get conflated constantly, and conflating them makes every answer wrong. There are three separate reasons a cell might contact corporate infrastructure, with three different postures:
| Path | What it carries | Default | Can an operator stop it? | Effect of it never succeeding |
|---|---|---|---|---|
| License refresh | A check for a renewed or updated key | On in connected mode, skipped entirely when disconnected | Yes, by running disconnected | None directly. The installed license keeps governing until its own term runs out |
| Metering uplink | Signed usage bundles for billing | Always on | No. It is the billing plane | Usage accrues locally and queues. The offline ladder eventually degrades the workload |
| Proactive Support | Bounded health snapshots and event streams | Opt-out, consent-gated, off and fail-closed on an air-gapped posture | Yes | None. Support telemetry is not enforcement |
They are different planes with different governance. The support-telemetry gate does not govern the metering uplink, and turning off phone-home support does not stop metered usage from being recorded or reported. Anyone who tells you otherwise is describing a different product.
The rest of this page is about the middle row.
The path a metered event takes
Figure M1: durability before every risky step. The event is written to the write-ahead log before the database insert, and the bundle is persisted to the queue before any network attempt, so neither a crash nor a dead link can lose usage. The gate that matters is the last one: an acknowledgement is verified, never assumed. Anything wrong with it, a bad signature, a mismatched bundle, a malformed envelope, is treated exactly like no acknowledgement at all, and the bundle stays queued.
Three properties in that diagram deserve to be stated as commitments.
Nothing is counted as sent until it is proven received. The acknowledgement from corporate infrastructure is signed, and the signature is checked over the acknowledgement's own fields. A failure to verify is treated identically to no response, never as success. The bundle stays in the queue and the offline state machine does not advance.
A window that is skipped is never billed, so windows are not skipped. The boundary between reporting windows is a durable watermark, not a variable in a running process. A restart resumes from where it left off rather than re-anchoring to the current time and forgiving everything since the last tick.
A fault is not an absence. If the meter cannot read its own records, that is an error, not an empty window. Only a genuine absence of usage advances the watermark. This is the same distinction the license refresh draws between the server said this is gone and the network did not work, and it exists for the same reason: the two look alike at the call site and mean opposite things.
Scenario by scenario
The link drops for an hour
Nothing happens that anyone notices. Bundles build and queue; when the link returns, the queue drains oldest first and each bundle is acknowledged. Bundle identifiers are sortable and each one references the previous, so the backlog replays in chronological order and the chain stays continuous.
No enforcement change. No warning. This is the ordinary case.
The link is out for days
Usage keeps accruing locally and the queue keeps growing. The workload is unaffected until the deployment crosses the offline window it contracted for, which is covered in the offline ladder below.
The queue is bounded, not infinite. It holds roughly a month of hourly bundles. Past that, the oldest bundle is evicted and the eviction is audited, so a backlog that exceeds the design window is visible as a recorded event rather than as a quiet gap in the billing record.
That bound is a deliberate trade. An unbounded queue eventually consumes the disk of a cell that is doing real work, and a cell that fills its disk to preserve billing data has its priorities backwards.
The cell crashes mid-flush
The write-ahead log covers this. Entries are written before the database insert, and pending log files are replayed at the next start, so usage that was in flight during a crash is recovered rather than lost.
A crash during a flush is also the one legitimate source of a gap in the hash-chained record, which is why gaps are detected and reported rather than ignored: see what a gap means.
The flush itself falls behind
If the write-ahead log accumulates beyond its limit, new writes are rejected rather than silently dropped or allowed to consume the disk without bound.
This is worth understanding as a design stance. The failure is loud and bounded. A metering subsystem that quietly discards what it cannot keep up with would be the worst possible failure mode, because the resulting bill looks plausible and is wrong.
A cell has no route home, but a peer does
A cell that cannot reach corporate infrastructure can send its bundles to a peer that can, which forwards them upward.
The forwarding model is careful about attribution and about trust:
- The producing node's bytes travel verbatim, carrying the identity of the node that produced them, so records stay attributed to the cell that did the work rather than to the cell that relayed it.
- The peer's own signatures cover the hop, not the whole journey. The forwarding cell applies its own envelope upstream, and the fact that it verified the peer's signature is recorded durably rather than re-derived later. The forwarder vouches for what it checked.
- Forwarding reuses the existing outbound queue rather than adding a second sender, so the retry, backpressure and acknowledgement handling are the same code that handles a cell's own bundles.
The failure this design exists to prevent is a specific and nasty one: an endpoint that accepts a peer's bundles and never forwards them is worse than one that refuses, because the peer sees success, stops retrying, and its usage is never billed. Both ends read a silent one-sided loss as working correctly.
The deployment is permanently air-gapped
This is a supported posture, not a failure. A license can carry an air-gap addendum that declares a negotiated maximum offline period, and the ladder below grades against that window instead of the standard one.
Usage is still metered, still hash-chained, and still bundled locally. What changes is only where the bundles can go, and the terms under which the estate keeps operating while they cannot go anywhere.
Someone turns the clock back
Backdating the host clock is the cheap way to defeat any offline timer, so it is treated as a hard signal rather than as an input to be averaged in.
At start, the cell compares the wall clock against its last recorded time anchor. A retreat beyond the allowed drift budget:
- Emits a clock-tamper event to a durable record, and
- Advances a persisted floor so the elapsed measure never loses the missing time, and
- Forces the suspended state pending operator review, regardless of what the other graders say.
The elapsed measure itself is the maximum of monotonic time, wall-clock time and that persisted floor, so it is resistant to any single one of them being wrong.
The elapsed measure falls back to the install baseline, never to the zero epoch. A freshly installed cell that has not yet reported does not read as decades overdue and does not degrade on its first day. This sounds obvious and is a classic bug in systems that measure "time since last contact."
The offline ladder
Time offline is graded into the same four states as license expiry, so there is one ladder for the whole platform rather than a parallel vocabulary.
The thresholds are fractions of the offline window the license itself declares, and there are two ramps:
| Warn | Grace | Read-only | Suspended | |
|---|---|---|---|---|
| Standard sync window | 75% | 100% | 150% | 200% |
| Negotiated air-gap window | 90% | 100% | 110% | 120% |
The air-gap ramp is tighter after the boundary and the reason is not punitive. An air-gap window is already long and was negotiated deliberately, so there is far less justification for drifting well past it. A standard window is short, and a customer with a genuinely broken link deserves a much longer runway before anything degrades.
The first threshold in both rows is a warning boundary, not an enforcement change. The state stays fully active; an operator is simply told, with a reason string naming how many days offline out of how many the license allows. Enforcement begins at 100%, the moment the deployment is past the window it contracted for.
A license that declares no offline window is not graded on this axis at all. The code refuses to invent a window, and the expiry ladder's verdict stands untouched.
Three graders, one ladder, most severe wins
Days offline is one of three independent inputs, and they answer three different questions:
| Grader | The question |
|---|---|
| License expiry | Has the customer's term run out? |
| Offline duration | How long since this cell was last verifiably heard from? |
| Chain integrity | Is this cell's own metering record intact? |
All three grade into the same four states, and the most severe wins. The merge is the whole reason the platform has one degradation ladder instead of three subsystems each with their own idea of what "read-only" permits.
Two safeguards keep the extra graders honest, and both run in the customer's favour:
- A grader can only make the verdict worse, never better. Nothing on this path can promote a deployment into a state it has not earned.
- A grader that cannot produce a trustworthy reading is skipped entirely. It does not guess, and it does not degrade anything on the strength of a measurement it could not take.
The practical consequence: a recently-synced deployment with an intact record gets back exactly the verdict the expiry ladder produced. Both escalations are provably inert for a healthy estate.
What a gap in the record means
The usage record is hash-chained with a monotonic sequence per cell and chain. A gap is a missing or out-of-order entry.
A healthy chain never has one. The legitimate cause is a crash during a flush, which is rare and is accompanied by its own audit event. A gap with no accompanying event is the tamper signature, and it is exactly what this design exists to make visible.
Gaps are reported two ways: as ranges included in the outbound bundle, so the record that reaches corporate infrastructure is honest about its own holes, and as a flag that feeds an anomaly rule.
A scan that could not read a chain is reported separately from a scan that read it and found it clean. Collapsing the two into one empty result would mean an integrity scan over a broken database came back green, so they travel in different fields and every consumer has to handle both, because a finding is evidence about the chain, and a fault is evidence about nothing. A fault must never render as reassurance.
What never stops, in any state
Whatever the ladder says, a fixed floor holds:
Degrade the workload, never the senses.
Reads and export are allowed in every state. So are the system's own essential writes: audit, telemetry, attestation, and metering itself. These are not tunable, and no state turns them off.
That last inclusion is deliberate and slightly counter-intuitive. A deployment that has degraded because it could not report keeps metering, so that when the link returns the record of what happened during the outage is complete. A system that stopped counting when it degraded would be unable to bill honestly for the period it was degraded, and unable to show the customer what actually happened.
What an operator should watch
| Signal | Where | Why |
|---|---|---|
| Days since last verified sync, per cell | The license fleet view | The direct input to the offline ladder |
| Pending bundle backlog | Metering state | Growth means bundles are not being acknowledged |
| Bundle eviction events | The audit record | The backlog exceeded its bound and the oldest data was dropped |
| Write-ahead log rejections | Service logs | Flush is falling behind; this is the loud failure, treat it as urgent |
| Clock-tamper events | The audit record | Forces suspension, and warrants investigation on its own merits |
| Chain gaps, and scan faults | Anomaly rules | A gap without an accompanying event is the signature this design exists to surface |
Related topics
- The four states these graders feed, and what each permits: the licensing lifecycle.
- Installing or renewing the license that declares the offline window: applying a license.
- How a metered event becomes a verifiable bill: metering, FinOps and tokenomics.
- The separate, opt-out support channel this page is repeatedly not about: proactive support.