Messaging
A message is not a second-class citizen here. It carries the same classification, the same workspace scope, the same audit chain, and the same cost attribution that a table does, because it is the same substrate.
That is the whole argument, and it is easy to state and hard to retrofit: most platforms have a data governance model and a message bus, and the bus is where the label gets lost.
Is it a broker, or a client?
This question comes up immediately, so here is the direct answer.
System9 runs a real broker. It is its own broker, speaking its own protocols: gRPC and REST with server-sent events and WebSockets, over mutual TLS, persisted in FluxDB.
It is not an AMQP broker, not an ActiveMQ broker, and not an MQTT broker. It does not listen on those ports. Where those protocols appear, System9 is the client connecting out to someone else's broker.
That distinction matters for planning, so it is worth being precise about each side.
What listens
The messaging service binds four sockets: HTTP and gRPC on loopback for in-cell callers, and HTTPS and gRPC over mutual TLS for cell-to-cell traffic.
| Surface | What it serves |
|---|---|
| gRPC | Publish, batch publish, fetch, server-streaming subscribe, acknowledge, cursor commit, subject and subscription management, consumer-group join, leave, heartbeat and member fetch, and the cross-cell delivery pair |
| REST | The same operations over JSON, plus send-to-entities, dead letters, streaming queries and pipelines |
| Server-sent events and WebSocket | Live event tailing |
A client library and command-line tool ship for the gRPC surface.
What connects out
| Protocol | Role |
|---|---|
| MQTT | Client. The edge gateway connects to an operator-run broker. The broker is mosquitto, EMQX, HiveMQ or similar, and is not shipped |
| AMQP 1.0 | Client, for Service Bus and compatible endpoints |
| Kafka | Client, consuming |
| AWS, Azure and Google messaging | Outbound producers for queues, topics, streams and event buses |
The messaging service contains an AMQP bridge and an MQTT bridge module. Both are design-stage adapters with no wire-protocol implementation: concept mappings, topic translation and quality-of-service tables, with no frame codec, no socket, and no callers. Their configuration keys are never read.
Separately, the edge gateway has a file called an MQTT listener that binds a default port. Its own documentation is clear that it does not open a broker socket; it wraps an injected client. If you are trying to work out whether System9 brokers MQTT, that filename is the most likely thing to mislead you.
Treat both as roadmap. An external system that needs to speak AMQP or MQTT to System9 needs a broker in between today.
Messaging is a data service
The reason the governance is identical is that messaging is not a neighbour of the data platform. It is part of it.
| Shared with the data platform | How |
|---|---|
| FluxDB | The bus persists through FluxDB, into its own logical database, through the same access layer every other service uses |
| Tiered storage | The same hot, warm and cold pattern: payloads live inline in the hot tier and migrate to warm and cold on the storage service's object store |
| Convergence | The same conflict-free primitives the data plane converges with, over a changeset log with catch-up, snapshots and peer sync |
| Connectors | The stream connector layer is shared with the data platform's ingestion rather than duplicated |
So "a message carries the same classification as a table" is not an integration that was built. It is a consequence of the message and the table being stored, tiered, replicated and governed by the same machinery.
The tiering has a safety property worth naming
Messaging's hot-to-warm migration copies a payload to the warm backend but deliberately leaves the canonical hot-tier bytes in place. A separate sweeper later walks those rows, verifies the tier backend's read path is actually observable for that specific reference, and reclaims the inline copy only if the verification succeeds.
The rule it encodes is one the whole platform follows: no destructive delete until the replacement read path is observable. A migration that deletes the original on the strength of a successful write is a migration that discovers its own bugs as data loss.
The tier backend also carries a bandwidth budget, so a long-running recall cannot monopolise the object store's read and write capacity and starve everything else using it.
The message model
An envelope carries a subject, headers, crypto metadata, content, and a routing path. Around it sit fragments, routes with a TTL and a maximum hop count, acknowledgements, durable cursors, backoff policies, retention policies, subscription policies, and per-message policy overrides.
The vocabulary is explicit rather than implied:
| Dimension | Values |
|---|---|
| Kind | data, control, system |
| Priority | low, normal, high, urgent |
| Delivery semantics | fire-and-forget, at-least-once, exactly-once |
| Cursor policy | auto, manual, none |
| Codec | bytes, JSON, Arrow, Parquet, MessagePack |
| Compression | gzip, zlib, zstd, lz4, snappy |
| Encryption | none, AES-GCM, PGP |
| Transport | gRPC, HTTP, MQTT, PGP |
| Routing mode | local, broadcast, multicast, anycast |
| Rebalance strategy | range, round-robin, sticky, cooperative sticky |
The publish path
Publishing builds the envelope, optionally fragments it MTU-aware with a per-piece SHA-256 and CRC32, stores payloads in a content-addressed blob store, and inserts idempotently, so a duplicate publish is a success that reads back the stored row.
Along the way:
- Subject and partition management with watermarks and sequence allocation.
- Quota checks on message count and total bytes, per subject or per entity.
- Token-bucket rate limiting scoped per cell, per entity, per subject, or globally, returning explicit retry guidance rather than silently dropping.
- Schema validation against a registry, with strict, warn, or off enforcement per subject.
Large payloads offload to the object store above a threshold, with SHA-256 deduplication, compression, and TTL-aware garbage collection, retrieved transparently on fetch.
Delivery, consumption, and reliability
| Mechanism | Behavior |
|---|---|
| Delivery | To local subscribers and to peers by routing hint, enforcing inflight caps globally, per subject, and per subscription |
| Consumer groups | Four rebalance strategies including cooperative incremental rebalance, member heartbeats, and optional persisted assignments so a coordinator crash does not lose the group |
| Durable cursors | Per subscription, subject, and partition, with monotonic advance, expansion as partitions grow, per-partition and total lag computation, and a batcher that coalesces updates to cut write amplification |
| Acknowledgements | Five states (received, applied, failed, nack, replicated) with deterministic idempotent IDs, consumer-side dedupe, and ack-timeout detection returning redelivery candidates |
| Transactional outbox | Due-message selection, atomic claim by pushing a visibility timeout, configurable backoff and dispatch budgets |
| Transactions | Atomic multi-subject publish with epoch-based producer fencing, a prepare, commit, abort lifecycle, and automatic abort of stale preparing transactions |
| Retention | A maintenance sweep enforcing per-subject retention by age, count, or bytes, and conservatively reclaiming blobs only when unreferenced by every table |
Cross-cell replication and cloning
A unified changeset log over messages, outbox, and inbox with per-table watermarks and batched feeds. Snapshot export and import with headers and optional payload inclusion. And a catchup orchestrator that compares the local watermark to the remote high-water mark, requests a full snapshot when lag exceeds a threshold, then switches to incremental. Convergence uses conflict-free primitives: a positive-negative counter and an observed-remove set.
Cloning supports shallow copy-on-write. A clone's partitions reference the source's sequence ranges, reads fall through for shared ranges, writes always go to the clone, and anything past the clone high-water mark is automatically independent. A background worker promotes shallow to deep by copying shared segments in batches.
Transports and routing
The cell-to-cell transport is gRPC with a retry policy, per-endpoint circuit breaking and hedged requests.
Routing selects the next hop by health (latency and error rate), locality (region and availability zone), weight and transport-security requirement, through a thread-safe hint cache.
Outbound producers exist for the AWS, Azure and Google messaging services, each created behind an egress guard, with the ordering control each service actually offers: a partition key, an ordering key, or a session identifier. A producer whose service has no equivalent ignores the setting rather than pretending.
For what is a bridge and what is not, see the broker question above.
Encrypted messages, and multiple recipients
Messages between cells travel in a signed and encrypted envelope, and one envelope can address many recipients at once.
How a multi-recipient envelope is built
The payload is encrypted once, under a single session key, and that key is wrapped separately for each recipient. One standard OpenPGP body, one small key packet per recipient.
Any recipient holding a matching key decrypts the shared body independently, and the envelope carries one signature over the whole thing.
Two properties are worth calling out:
Size does not grow with the audience. The body is encrypted once regardless of how many recipients there are, so a broadcast to a large workspace or clean room costs one payload plus a small key packet each, rather than a full copy per recipient.
A recipient whose key cannot be resolved aborts the whole send. There is no partially-encrypted envelope and no quiet omission of someone the sender believed was included.
Enforcement is per principal kind. Cell-to-cell and node-to-node traffic requires encryption; user traffic is optional; and the policy is resolved rather than assumed.
Older envelope versions are still decoded on receipt, so a cell that has not yet been updated can still be understood. Only the sending path moved.
Crypto provenance: who signed it, and which keys could open it
Every cross-cell encrypted delivery now records its own crypto provenance, so for any message you can answer two questions after the fact:
| Question | What is recorded |
|---|---|
| Who signed this? | The signer's public key fingerprint, and the scheme |
| Which keys could decrypt it? | Each recipient's resolved key fingerprint |
Only public fingerprints and SHA-256 references are stored. No key material, no plaintext, no signature bytes. The record is a set of pointers to material held elsewhere, which is what makes it safe to keep and to query.
Two design choices worth naming:
Recording is fail-soft. If the provenance write fails, whether because there is no messages database on that node or from a transient fault, the failure is logged and swallowed, and the message is still delivered. Audit enrichment is not allowed to become a delivery dependency. The trade is explicit: provenance can have holes, and delivery does not have outages. What that costs an auditor is set out below.
It gives blob retention something to hold on to. The retention sweep only reclaims a stored blob when no table references it. Now that provenance rows exist, crypto material referenced by them is protected from reclamation, so tidying up old messages cannot quietly destroy the material their provenance points at.
Two rows that mean different things
The ledger carries two kinds of record, and conflating them is the mistake this section exists to prevent.
| Row | Written | What it asserts |
|---|---|---|
| Unverified | By the sender, at send | These fingerprints were named as signer and recipients. A claim about who the envelope was built for |
| Verified | By the receiver, on arrival | This signature was cryptographically checked, and it passed. A record of a check that actually ran |
A receiver writes its row after the message row lands, on its own cell's database, carrying the verified signer's fingerprint and the time of the check. It is that cell's attestation, not a global fact, which is consistent with the rest of the per-cell provenance model: a second cell that also received the message attests separately.
For an auditor the distinction is the whole point. Send-attribution is a claim; receiver-verification is a check. A query that treats them as one field with two values will read claims as proof.
Two different failure postures
These are easy to run together and they are opposites.
Verification is fail-closed. If the signature does not verify, or the envelope cannot be decrypted, the delivery raises and is not ingested. A message with a bad signature does not land and then get flagged; it does not land.
Recording the verification is fail-soft. The provenance write is wrapped, and a failure is logged and swallowed so it can never block ingest.
Put together, they produce one consequence worth stating outright:
A missing verified row does not mean the signature went unchecked. It can equally mean the check passed and the record did not land, or that the payload was never PGP-framed, or that encryption enforcement is disabled for that sender class. None of those write a row.
Read a present verified row as positive evidence that a check ran and passed. Do not read an absent one as evidence of anything. The fail-soft write buys delivery availability at exactly this cost, and an audit built on inferring from absence would be drawing conclusions the ledger does not support.
Two honest limits
The recipient list is not confidential. Everything outside the encrypted body is cleartext: the subject, the headers, and the routing entries naming each recipient and its cell. Content is protected; who is talking to whom, on what subject, at what time, is not. Sealed-sender style metadata confidentiality is a separate piece of work.
It is transport protection, not end-to-end to the reader. The receiving cell decrypts on arrival and stores the message in its log. This protects the message in flight between cells; it does not keep it opaque to the cell that delivers it.
Governance on the message path
| Control | How it applies |
|---|---|
| Classification | A message carries the same classification as a table |
| Workspace scope | Every subject and every inbox is scoped, and provisioning gives each workspace its own inbox |
| Subject-level grants | With guards on the publish and consume paths |
| A per-workspace audit chain | With retention and peer audit consumption, so peers consume each other's audit |
| Payload encryption | Per the envelope's declared scheme |
| Clean-room scoping | Streams can be scoped to a room and inherit its egress policy |
| Bandwidth budgeting | So a noisy subject cannot starve the fabric |
The feed and publishing layer
Sitting over the bus is a governed entity feed that is unusual enough to name separately: composition, fan-out, and ranked reading with reshares, mentions, notifications, and preference gates; entitlement and clean-room guards on every item; cross-organization trust scoring; per-entity direct-message inboxes with threads; and content publishing that cryptographically signs publications and versions, enforces per-region content residency, verifies signatures and AI-disclosure provenance, and scores trust-weighted candidates.
It federates over ActivityPub and ActivityStreams with WebFinger actor discovery and HTTP-signature signing and verification against sealed actor keys.
A distributed directory of cryptographic entities that can also publish to and subscribe from each other across organizations, with provenance, is a capability set no other data platform carries.
What is measured
Messaging is instrumented around the question that actually matters for a broker, is anything falling behind, and by how long? Published signals cover subscription lag as a maximum and a sum, how many subscriptions are behind, outbox pending and due counts, inbox messages seen, and the age of the oldest message in the inbox.
Oldest-age is the one to watch. A lag count tells you how many messages are outstanding; an age in seconds tells you how stale the worst one is, which is the number a service-level objective is actually written against. High maximum lag raises a warning alert out of the box. See what is measured.
Entity inboxes
Every entity, a person, an agent, an organization or a device, has an inbox addressed by its identity, and the messenger routes purely by resolved home cells rather than by what kind of principal the recipient is.
Two properties are load-bearing:
The sender cannot be forged. The acting entity is stamped by the server after the caller's own headers, so a caller-supplied value is overwritten. The attestation rides the envelope to remote cells, so it survives the hop.
A partial delivery is visible. Sending to several recipients returns a result per recipient: delivered, denied, no home, or failed. A multicast that reached four of five is reported as such rather than as success.
Where a recipient's home is dark, the send reports no home and the caller decides whether to queue, skip or surface it, rather than the system delivering blindly.
Workspaces are provisioned with default streams by kind: an inbox and activity for a person, governance and billing streams for an organization, alerts for a device. A device is a headless principal, not a page in the social fabric, and its stream set says so.
Recall
A published message can be recalled. Consumers subsequently see a tombstone rather than the payload.
Recall is a persistent job, not a fire-and-forget call: it survives a restart by reloading from its store, it can be previewed before it runs, and it can be cancelled. It carries its own audit chain.
The scope is the one that matters operationally: a recall fans out to peer cells in the same service group, because the meaningful answer is every matching message in the group, not just the ones on whichever cell the operator happened to reach.
Where the log lives, and how far it travels
The durable log converges across the cells of a node, so losing a cell does not lose messages. Operational machinery, the outbox, the inbox, dead letters, acknowledgements and sequence blocks, stays local to its cell and is rebuilt rather than replicated.
Cross-node movement is different: it is point-to-point delivery of a signed envelope, or a deliberate clone, rather than log replication. Cross-cell delivery is off by default, so a single-cell deployment is unaffected by any of it.
Partition ownership is fenced. A recorded assignment, not a recomputed hash ring, decides which cell serves a partition, and a write arriving at a deposed owner is rejected and rolled back rather than splitting the log.
Under a partition: writes continue locally, the outbox retries with backoff that collapses when the error class clears, gossip marks peers down, and catch-up repairs the gap on heal, taking a snapshot when the lag is large and incremental changes when it is not. Retention refuses to delete messages a peer has not yet received, so the hole stays closable.
What to know before you design against it
| Delivery | At-least-once is the guarantee. Exactly-once is approximated by idempotent publish, inbound deduplication and producer fencing, not by a distributed transaction with an external sink |
| At-most-once | Not expressible. The schema permits only the two above |
| Ordering | Per partition, not global |
| Classification | Subjects carry labels and metadata; there is no enforced message-level classification label today. Scope and access control are enforced; a per-message label is not |
| Messaging clean rooms | Modelled with participants and egress rules, without a backing store. The surface refuses rather than returning empty |
| Speaking AMQP or MQTT to System9 | Needs a broker in between. See the broker question |
Across the deployment matrix
- Single cell: the bus is fully functional locally and replication listeners are idle.
- Multi-cell: subjects replicate and clone across cells over mutual TLS, with conflict-free convergence.
- Sovereign and air-gapped: no external broker is required, an unreachable peer is an absent edge, and store-and-forward with custody transfer carries traffic when a link returns.
- Cloud-enabled: cloud messaging services can be registered as catalog connections behind an egress guard.
Related
- FluxDB, the database system the bus persists through.
- Streaming, the real-time query surfaces over these streams.
- Resources you provision, subjects, subscriptions, and inboxes in the inventory.
- Stream hardening, chain of custody and the cross-domain guard.
- Replication, DR & federation, the replication scopes messaging resolves against.
- The messaging service, the operational service view.