What is measured
Monitoring and logging describes the pipeline. This page answers the other question, the one an operator actually asks during an evaluation: what does it measure, and can I see the metric I care about without instrumenting it myself?
The short answer is that instrumentation is structural. Every service gets a set of cross-cutting instruments for free by virtue of being a service, and each one adds a domain publisher for the things only it can see. There is no service in the platform that is dark.
The three layers of instrumentation
Figure F5: the three layers. The universal layer is the one that makes the coverage claim hold, because a service does not have to remember to instrument its request path, its database calls or its workers. The domain layer is where each service publishes the handful of numbers only it knows. Note the ingest property: publishing backs off when the collector is unreachable and never blocks the work it is measuring. Observability that can stall the system it observes is a liability, not an asset.
Layer 1: what every service reports without being asked
| Instrument | Type | Answers |
|---|---|---|
| HTTP requests, and request duration | Counter, histogram | Traffic and latency on every HTTP surface, by route and outcome |
| gRPC requests, and request duration | Counter, histogram | The same for the internal backplane, which is where cross-cell calls actually live |
| Database queries, and query duration | Counter, histogram | Which service is loading the data layer, and how hard |
| Worker status | Gauge | Whether each background worker is alive, which is the difference between "configured" and "running" |
| Errors | Counter | Error volume by service and kind |
| Service info | Gauge | Version and build identity of the running service |
| Security abuse indicators | Counter | Abuse signals raised on a service's own surface |
| Topology: zone cells, region cells, cross-region round trip | Gauge | The measured shape and latency of the estate, not the configured one |
That last row is worth pausing on. Cross-region round-trip time is measured and published as a metric, so placement and replication decisions are informed by observed latency rather than by a number someone typed into a configuration file once.
Layer 2: the host and the machine underneath
A host collector reports the classic resource surface as first-class metrics: CPU (usage, core count, one, five and fifteen minute load), memory (used, available, total, percent), swap, disk (read and write bytes, free, total, percent), network (bytes and packets each way, active connections, errors in and out), process count, and uptime.
A per-cell record additionally carries what a bare host collector cannot know: request latency percentiles, throughput, active, queued and failed job counts, job failure rate, a system health score, error and warning counts, fatal crash count, and an energy and thermal group (joules, CPU temperature, fan speed). It also carries an anomaly score, a predicted-failure flag and an estimated time to next failure, which is where monitoring stops being a dashboard and starts feeding predictive maintenance.
Energy and temperature are not exotic extras at the edge, they are operational constraints. A cell in a vehicle, a cabinet or a remote site has a power and thermal budget, and a platform that cannot report against it cannot be run there.
Layer 3: what each service publishes about its own domain
Every service area carries its own publisher. This is the part that makes the catalog useful, because these are the numbers that tell you whether the product is healthy rather than whether the machine is:
| Area | Representative signals | The question it answers |
|---|---|---|
| System | Per-database file sizes (system, directory, sessions), fabric endpoints and active leases, replication engine registered and running, replication adapter count, workers alive and dead, crypto readiness, setup completed | Is the cell actually assembled and running, or merely started? |
| Storage | Active volumes, capacity and used bytes, fragmented volumes, active and orphaned blocks, active objects, active namespaces, degraded shards | Is durability intact, and is capacity where it was expected to be? |
| Storage read and write paths | Distributed read totals, checksum mismatches, shard fetch failures, degraded reads, encoding mismatches, superseded-layout serves, transform failures; distributed write quorum failures and dedup hits | Is the data coming back correct, and is it coming back the fast way or the degraded way? |
| Messaging | Subscription lag (maximum, sum, and how many subscriptions are behind), outbox pending and due, inbox seen and oldest message age | Is anything falling behind, and by how long? |
| Data | Convergence gauges (unmerged keys, peers tracked, oldest unmerged sample age), catalog leaf provisioning, registered commands, command audit emit failures, structure load factor and false-positive ratios, memory statistics | Has state converged, and is the data plane keeping its own commitments? |
| Compute | Active kernels, warm pool size, job queue depth, kernel crashes, kernel bind failures | Is there capacity, and are workloads landing or failing to land? |
| Security (crypto) | X.509 and SSH certificate totals, PGP key totals, crypto snapshot version, certificate expiry in seconds | Is the identity plane populated, current, and not about to expire? |
| Security (threat) | Threat level, blocked addresses and domains, adaptive rate multiplier | What is the gateway currently doing about hostile traffic? |
| Replication | Applied, published, rejected, skipped, deferred, conflicted, unsupported, degraded, caught-up, log pruned, tombstones collected, cross-region lag in seconds, bandwidth bytes, apply errors | Is convergence healthy, and when it is not, which of the nine failure shapes is it? |
| Fabric | Routing lookups, cache hits and misses, routing denials, rate-limited routes, leases acquired, renewed, expired and preempted, health probes, failures and flaps, endpoints registered, deregistered and swept | Is service discovery and leadership stable, or is it churning? |
| Node gateway | TLS certificate expiry, uptime | Is the front door about to stop trusting itself? |
| Metrics itself | Collection ticks, total samples stored, uptime | Is the observer observing? |
| Admin | Per-database sizes, uptime | Out-of-band management plane health |
Two of these deserve to be called out as the ones a competitor's dashboard usually lacks.
Replication is measured by outcome shape, not by a single lag number. Applied, rejected, skipped, deferred, conflicted, unsupported and degraded are separate counters. "Replication lag is 4 seconds" and "replication is silently rejecting a class of rows" look identical on a single lag gauge and completely different here.
Fabric health counts flaps, not just failures. A service that fails cleanly and a service that oscillates between healthy and unhealthy produce very different operational pain, and only the second one is invisible to a point-in-time health check.
The crypto plane has its own instrument set
Because certificate and key operations are the thing everything else depends on, the crypto plane is instrumented separately and in Prometheus-native form: issuance count and duration, unseal duration, rotation count, key revocation list propagation time, attestation count, audit chain verification count, snapshot tick duration, active certificate count, hardware security module availability, per-certificate expiry, key-use count and duration, audit event count, and SIEM forwarding forwarded and dropped counts.
Labels on these carry algorithm, purpose, outcome, storage type, backend, trust level and transport, so "issuance is slow" can be resolved down to which algorithm, on which backend, for which purpose. Key-revocation-list propagation time in particular is the metric that tells you how long a revoked key stays dangerous, which is a security property, not an operational one.
The naming and dimensioning model
A sample is more than a name and a number:
| Part | Rule |
|---|---|
| Name | Dotted and hierarchical, for example storage.shards.degraded or messaging.subscriptions.lag_max |
| Type | Counter, gauge, histogram, or summary |
| Unit | Carried on the sample, so a chart does not have to guess |
| Universal dimensions | Cell, node, and service are dedicated columns, because every sample has all three |
| Everything else | A free-form label set |
The split between columns and labels is a deliberate design decision rather than an accident of schema evolution. Cell, node and service are columns because every sample carries them. A dimension that only some metrics carry, a clean-room identifier being the clearest case, rides as a label, because a column would be empty on almost every row and would fight the store's layout.
Crucially, the rollup engine groups by the whole label set, so a label-borne dimension survives aggregation. A per-clean-room operational view is a label-filtered query over rolled-up data, not a separate pipeline.
Histograms and summaries are stored with their structure intact: bucket boundaries, sum and count for histograms; quantiles, sum and count for summaries. Rollups carry average, minimum, maximum, sum, count and the p50, p95 and p99 percentiles, so latency questions survive aggregation instead of collapsing into a mean that hides the tail.
The operator plane and the tenant plane
This is the governance property that separates a metrics store from a governed metrics store, and it is the part most platforms get wrong by pooling everything into one namespace and sorting it out in the dashboard.
A sample with no workspace label is node telemetry. The metric store is the node workspace's store: host resources, per-service health, convergence gauges, everything that describes the node and its cells rather than anyone's work. Tenancy, when it appears, arrives as a label.
The classification rule has three properties worth stating:
- It is made in exactly one place. An operator surface and a tenant surface cannot drift into two different answers about the same sample, because there is only one function that decides.
- It is fail-closed. A sample whose labels cannot be read is treated as tenant-scoped and withheld. The safe failure is an operator missing a number, never a tenant's signal appearing on an operator dashboard.
- It does not depend on cluster size. A sample carries its own cell and node, so classification is per-sample. A merged multi-cell result classifies exactly as a single-cell one, and scaling in or out changes the volume, never the rule.
Access to the metrics surface is itself guarded. Both reading across the node and modifying metrics state (ingest, alert rules) sit behind an authenticated boundary, and if that authentication is unavailable the request is denied rather than allowed through.
Retention: five tiers, not one dial
Raw samples and rolled-up buckets age out on separate schedules, so a long history costs a fraction of what keeping raw samples would:
| Tier | Default retention |
|---|---|
| Raw samples | 7 days |
| 1-minute rollups | 30 days |
| 5-minute rollups | 90 days |
| 1-hour rollups | 1 year |
| 1-day rollups | 2 years |
A retention pass reports exactly how many rows it removed from each tier and whether it reclaimed space, so retention is observable rather than a silent background truncation. The resolutions available for rollup are one minute, five minutes, one hour, one day and one month.
Metrics do not replicate, and that is the point
The metric store is the one core database excluded from replication by design. Copying every sample to every cell would multiply the most voluminous and least duplication-worthy data in the estate to answer questions that are almost always about one place at one time.
Instead cells query each other, with a bounded fan-out: a hard cap on remote cells per federated request so latency cannot grow without limit as the estate grows, a short per-cell timeout, a persistent worker pool so the fan-out does not pay thread-startup cost on every query, and per-cell failures returned as failures so a partial answer is legible as partial rather than quietly short.
Aggregates, by contrast, do travel: cell, service-group, zone and region health each synthesise into one view per level. See platform replication for the general rule this is an instance of.
The alert rules that ship
The platform arrives with working rules rather than an empty rule engine. Each carries a threshold, a duration the condition must hold (so a spike is not an incident), a severity, and where to send it:
| Rule | Condition | Severity |
|---|---|---|
| High CPU | Host CPU above 90% for 5 minutes | Warning |
| Critical CPU | Host CPU above 95% for 1 minute | Critical |
| High memory | Host memory above 90% for 5 minutes | Warning |
| Disk space low | Disk above 85% for 10 minutes | Warning |
| Disk space critical | Disk above 95% for 1 minute | Critical |
| Storage degraded shards | Any degraded shard, immediately | Critical |
| Messaging lag high | Maximum subscription lag above threshold | Warning |
| Cell health degraded | Health score below 70 | Warning |
Warnings log; critical alerts log and publish to a messaging topic, so they can drive Automations rather than only a human's inbox.
Conditions available to a custom rule are above, below, equals, and
absent. That last one matters more than it looks: absent fires when a metric
stops arriving at all, which is the failure mode a threshold rule can never
catch, because a dead collector reports nothing rather than reporting something
bad.
Alongside threshold rules, an anomaly detector runs over the same samples, so a condition nobody wrote a rule for can still surface.
Getting the metrics out
| Path | Direction | Notes |
|---|---|---|
| Prometheus-compatible query and scrape | Out | Point an existing Prometheus or Grafana stack at a cell |
| OTLP export | Out | Push to an existing OpenTelemetry collector |
| OTLP receive | In | Feed the platform's own store from existing instrumentation, so one pane covers both |
| Federated query | Internal | Fan out across cells and merge, with failures surfaced |
The receive direction is the one that is easy to overlook. Because the pipeline accepts OTLP as well as emitting it, an existing estate does not have to be migrated to be seen; it can report in alongside the platform's own telemetry.
In a sovereign or air-gapped deployment every stage of this runs locally, and the export paths are available to an in-estate stack with no dependency on an external monitoring service.
Related topics
- The pipeline these metrics flow through: Monitoring and logging.
- What metered events cost and who is charged for them: Metering, FinOps and tokenomics.
- Acting on a metric rather than reading it: Automations.
- The record of who did what, which is a different plane from telemetry: Activity Ledger.
- Turning condition indicators into a bounded failure estimate: Predictive maintenance.