Monitoring & logging
Monitoring answers "what is happening, and is it healthy?" System9 runs a full telemetry and logging pipeline on the same governed substrate as everything else: typed metric samples flow through batched ingest into a metric store and query engine; alerting rules with anomaly detection raise notifications; health synthesis turns raw samples into a status for cells, service groups, zones, and regions; and structured logs carry a correlation ID from edge to core. The distinction that matters most in a distributed estate is per-cell versus the estate view, each cell observes itself, and those observations federate up into one picture without a central collector every cell must reach.
The metrics and telemetry pipeline
Figure F2: the monitoring pipeline. Typed samples from per-cell collectors are ingested in batches into a metric store and a Prometheus-compatible query engine. A rollup engine and cross-cell federation lift the per-cell view into an estate view (green); health synthesis derives status for cells, service groups, zones, and regions; alerting rules with anomaly detection dispatch notifications (amber). Telemetry interoperates through OTLP export and receive and Prometheus compatibility.
Table equivalent
| Stage | What it does |
|---|---|
| Collectors | Emit typed samples (counter, gauge, histogram, summary) with labels, cell, node, service, unit |
| Batched ingest | Accept samples in batches into the metric store |
| Metric store + retention | Persist samples; manage retention |
| Query engine | Serve queries, Prometheus-compatible |
| Rollup engine | Aggregate across time and topology |
| Health synthesis | Derive status for cells, service groups, zones, regions |
| Alerting | Evaluate rules with anomaly detection; dispatch notifications |
| Federation | Combine per-cell views into an estate view |
| OTLP | Export and receive OTLP; interoperate with existing stacks |
This page covers the pipeline. What is measured is the catalog: every service's own signals, the universal instruments each one gets automatically, the host and per-cell resource record, retention tiers, and the alert rules that ship by default.
Interoperable by construction
Telemetry is not a walled garden. The query engine is Prometheus-compatible, and the pipeline both exports and receives OTLP, so an existing observability stack can scrape System9 or feed it. A collector registry and metadata registry keep the set of collectors and the shape of what they emit discoverable rather than hard-coded.
Per-cell logging versus the estate view
Logging is structured, with correlation-ID propagation via HTTP middleware and a gRPC interceptor, so a single request can be followed as it crosses from the edge to the core. Durable log capture is fail-open and re-entrancy-guarded: a lost durable copy never breaks the caller, logging, like metering, must never interfere with the operation it observes.
Two properties are specific to a distributed, classified estate:
- Per-workspace and per-clean-room log isolation. Logs are scoped to the work boundary, not pooled into one undifferentiated stream.
- Code-name redaction on emitted records. Internal code names are stripped from log records before they land, so a log a customer reads never leaks one.
A correlation registry maps external SIEM event IDs to platform correlation IDs under classification-aware controls that stop correlation context crossing from a classified system to an unclassified one, the estate view never becomes a side channel that defeats the classification ladder.
Per-cell structured logging is operational telemetry, high-volume, retention- managed, for diagnosis. The Activity Ledger is a different plane: an append-only, hash-chained, per-workspace record of what was done, tamper-evident and retained as an account of activity, not as debug output.
Across the deployment matrix
- Single-cell: the pipeline runs entirely on the one cell; health synthesis and alerting cover that cell's own scope.
- Multi-cell: cross-cell metric federation combines each cell's self- observation into an estate view; health synthesis spans service groups, zones, and regions without a single point every cell must reach.
- Sovereign / air-gapped: collection, storage, query, health synthesis, and alerting run locally; OTLP and Prometheus interop are available to an in-estate stack, with no dependency on an external monitoring service.
- Cloud-enabled: the same pipeline; OTLP export can additionally feed an external stack where policy allows it.
Related topics
- What is measured, the catalog of signals this pipeline carries.
- Metering, FinOps & tokenomics, the cost intelligence layer reads over this telemetry.
- Threat detection, how the SIEM consumes logs and the correlation registry.
- Edge and fleet, how edge devices report into the estate view.