Object storage
System9's object store is not an S3 implementation with extras. S3 is one of five front doors onto a native engine, and it is deliberately the least capable of them.
The distinction matters commercially, so here it is up front: the things that make this store different are content addressing as the primary key, a namespace tree rather than a flat bucket list, classification enforced in the data path with a database constraint behind it, placement that understands your physical topology, and a durability claim you can measure rather than read.
In one paragraph
Objects are stored as content-addressed blocks in append-only volumes, chunked under a manifest so a range read touches only the covering chunks, compressed, encrypted with a per-block key, and then either replicated or erasure-coded across cells chosen by a deterministic placement function that respects rack, zone and region boundaries. Every read verifies twice. Every namespace carries a classification that the data path enforces. Replication and erasure are the same mechanism with different parameters, so the durability arithmetic has no special cases.
The engine
Content addressing is the primary key
A block's hash is its primary key, not a derived identifier and not an entity tag calculated afterwards.
That one decision gives deduplication structurally rather than as a feature: writing bytes that already exist is a hash lookup that finds them. A reference count on each block is what garbage collection acts on when the last reference goes away.
The hash is taken over the original bytes, before compression, so two writes made under different compression settings still deduplicate against each other.
Objects above a threshold become manifests
A large object is split into ordered blocks under a manifest hash, and each chunk records its logical offset.
Storing the offset is what makes a range read cheap: the reader can work out which chunks cover the requested range without reading blocks to discover their lengths. A partial read of a large object touches only the part it needs.
Shards describe themselves
The shard record carries far more than a location. Each one records the encoding generation and the data and parity widths that produced it, plus its own compression algorithm, original size, encryption mode, nonce, tag and wrapped key.
Two consequences, and both are load-bearing:
A shard is interpretable on a peer that holds no object metadata. Repair and anti-entropy depend on that, because the cell doing the rebuilding may never have seen the object.
Two encoding generations cannot be silently mixed. Without the generation stamp, re-protecting an object can leave two shard sets at the same index, and a reader that mixes them decodes garbage while every individual checksum still verifies. That failure is invisible precisely where you would look for it, which is why the generation is recorded rather than inferred.
Replication and erasure are one mechanism
A replica set is recorded as erasure with a data width of one. There is no separate replication path and no second durability calculation.
The benefit shows up everywhere downstream: placement, repair, the audit and the conformance report all reason about one shape.
Namespaces are a tree
Most object stores give you a flat list of buckets. Here a namespace is a node in a tree with a parent, and a bucket is simply one node type alongside directories and mount points.
Directories are real rows with their own child counts and size accounting, not a prefix convention over flat keys.
Every namespace carries a workspace and a classification, and the classification is a database constraint rather than a label applied by convention. That constraint is the source of truth that every downstream copy inherits, which is what stops a replica of a secret namespace defaulting to something weaker.
Grafting one namespace into another
A mount point splices one namespace subtree into another, in one of three modes:
| Mode | Behaviour |
|---|---|
| Read-only | The source appears in place, writes refused |
| Read-write | The source appears in place and accepts writes |
| Overlay | The source shadows what is beneath it, by priority |
Listing a grafted path returns a priority-ordered union: mount sources expand inline, higher priority shadows lower, and the namespace's own children rank lowest. Resolution is deliberately single-hop, which is what makes a graft cycle harmless rather than something to detect and handle.
A graft can also point at another node, in which case it resolves to a reference on that node and is treated as a custody set that happens to live elsewhere rather than as a special remote case.
The effect is that one physical namespace can appear at several logical paths, a workspace can mount another's data read-only, and an overlay can shadow a base without copying it.
Mounts are created, listed and removed through the namespace surface. A local or remote graft goes through the graft manager; a graft onto an external store goes through the external-mount path, discriminated by whether the mount names a connection.
Addressing
An object reference is a valid HTTPS URL, deliberately not a custom scheme, and resolves three ways: by path, by object identifier, or by namespace identifier plus a path. A version qualifier selects a specific version.
The same reference has a compact binary form so it can be embedded in a scoped access token.
Placement understands your topology
Placement is a deterministic function of the object, the shard index and the candidate cell, so every cell computes the same placement for the same object.
That property is not decoration. If two cells disagreed about where a shard belongs, the rebalancer would oscillate, moving a shard off and back forever, and repair would rebuild into a location another cell believed was already occupied.
Failure domains are ordered cell, rack, zone, region, from a taxonomy shared with the rest of the platform rather than one storage invented for itself. Cloud provider and labels are separate axes rather than levels in the tree.
Placement returns exactly the requested number of targets on distinct healthy cells, and refuses rather than reusing a cell, because collapsing a failure domain destroys the guarantee the profile promised.
Residency is enforced, and an impossible one refuses
A namespace can require a zone, region, rack, cloud provider or label, or prefer one, or spread across them.
A requirement that cannot be satisfied refuses the write, with an error distinct from "not enough cells". The distinction matters because the remedies are opposite: adding cells anywhere fixes a durability shortfall and may well make a residency violation worse.
This is the failure mode the design is most pointed about. A residency rule that is recorded, persisted and never consulted produces an object that is readable, whose metadata is correct, and whose bytes are in the wrong jurisdiction. Nothing in the system looks wrong.
Global placement, local reachability, never mixed
Placement topology is global and agreed. Reachability is local and measured. They are deliberately different inputs and never blended.
Reachability is not a boolean. It has five states:
| State | Meaning |
|---|---|
| Direct | Reachable now |
| Relayed | Reachable through a boundary cell |
| Pending | A scheduled path, such as a physical-media transfer. Not an error |
| Denied | Policy. An operator can change it |
| Partitioned | Physics |
An air gap is an absent edge, not a special case.
Peering is measured over a bounded neighbourhood rather than a full mesh, because probing everything becomes the congestion it is trying to observe, and because a full mesh would reveal the whole topology across boundaries that exist to prevent exactly that. Observations are leased: past its lease an observation is absent, never stale-but-usable.
Reads are ranked by time, not by preference
Unreachable candidates are excluded rather than deprioritised, which is where most of the recoverable latency lives. The remainder are ordered by estimated time for that shard's size, so a local parity shard beats a cross-region data shard. Unmeasured links get pessimistic defaults so they are never ranked fastest by accident.
Hedged requests are tiered: near candidates hedge immediately, far ones only after a missed median, relayed ones never. Parity shards are free durability; they are not free bandwidth.
Relay is read-only, deliberately
A read can be delegated to a boundary cell that can reach custody directly, which returns one reconstructed stream rather than fanning out individual blocks.
Writes are never relayed. A relayed write cannot verify placement or confirm the spread invariants, so it would report success while violating the durability it claims. The design calls that durability theatre and refuses it.
Durability you can check
Profiles
Two copies is the default, not one. Erasure profiles state the exact number of distinct cells they require, validated when configuration loads rather than discovered when a write fails.
Profile selection can be automatic, and it is guarded by domain diversity: a six-cell deployment in a single rack is given replication rather than erasure, because four-plus-two across one rack survives no rack failure at all.
Small objects have an erasure floor derived from the block size, not chosen. Below it, splitting an atomic object into six sub-block shards on six cells costs six placement decisions and a cross-cell fan-out on every read, to protect something that fits in one block.
Five invariants
| Invariant | |
|---|---|
| I1 | No two shards of one object share a cell |
| I2 | Shards span as many failure domains as are available |
| I3 | Losing any one failure domain still leaves enough shards to rebuild |
| I4 | An object holds its full shard width, not merely the minimum |
| I5 | An object always holds at least the minimum. Critical |
Only the last is critical, and separating it is the point: it is the only one where the object may already be unrecoverable. Grouping them would bury the one that warrants waking someone.
Distinct cells is not durability. Six shards on six cells in one rack survives zero rack failures. Counting cells feels like checking spread and is not.
One definition of these predicates serves three consumers: the placement post-condition, the repair loop, and the periodic audit.
The write quorum
A write acknowledges at one more than the minimum where the width allows.
Acknowledging at the bare minimum returns success the instant the object becomes readable, which is the moment it has zero redundancy. It deliberately does not demand full width either, which would fail every write while any peer was down; the gap is caught by the full-width invariant so repair closes it.
Verified twice on every read
Every read checks the stored bytes against their checksum, then checks the final plaintext, after decryption and decompression, against the content address.
Alongside that: per-volume Merkle trees, a rate-limited and resumable bitrot scanner, cross-cell anti-entropy comparing Merkle roots, and quarantine. A quarantined block raises on read rather than returning corrupt bytes.
A checksum failure on the hot path marks the shard degraded and queues repair. It does not delete it, because it may still be the only copy of that stripe.
The audit is sampled, and the report says so
The durability audit samples rather than sweeps, because a full scan every interval is the kind of thorough that gets switched off the first time it causes an incident, and a disabled audit detects nothing.
The protection report returns the declared profile alongside measured conformance over a stated sample, with three properties worth quoting:
- The sample size is reported, never implied.
- Unevaluated is not conforming. It is counted separately.
- Conformance against a declaration that cannot be resolved returns zero with its reason, not a reassuring zero.
One implementation serves both the storage service and the console, so a panel cannot quietly disagree with the alert.
Replication across cells
Metadata replicates through a write-ahead log with explicit gap detection, while blocks transfer in chunks with a content-addressed existence probe first, so a block the peer already holds is never shipped.
A primary is elected per service group on a lease, and the holder is a cell rather than a node, because cells own storage. Each takeover bumps a monotonic epoch that the write path checks, so a deposed primary fences itself before its lease lapses. There is no consensus protocol here; safety comes from the fence.
Conflicts are largely designed away: identical bytes hash identically, so a concurrent duplicate write deduplicates instead of conflicting. Metadata converges by primary key. An event that cannot be applied is deferred with a reason rather than dropped or retried blindly, and anti-entropy is the backstop for anything the log missed.
One conditional primitive exists, an atomic create-if-absent, and where a transport lacks it the layer above refuses rather than degrading to check-then-write. Two writers both seeing absence and both writing is how the second silently replaces the first while both receive the answer only a race winner should get.
Service groups
A service group is a set of cells that collectively own a slice of state. For storage it is both the placement boundary and the election scope.
A namespace maps to groups by role, primary, replica, archive, compute or overflow, with priority, and the mapping inherits down the namespace tree unless a descendant pins its own. That is what lets one namespace keep its primary in one group and its archive in another.
One cell is a complete object store
Single-cell is a first-class deployment shape, not a degraded one, and the code is explicit about it: no peer means no replication, no election, and anti-entropy correctly does nothing. Those are correct states, and the health surface names them rather than reporting a fault.
What you still get on one cell is nearly everything:
- Content-addressed deduplication with reference-counted collection
- Compression and envelope encryption with per-block keys
- Append-only volumes and a checkpointed write-ahead log
- Chunk manifests with true range reads
- Full versioning, delete markers, time travel and point-in-time restore
- Merkle trees, bitrot scanning, quarantine, read repair, double verification
- The namespace tree with inherited policy, access control and classification enforcement
- Snapshots, lifecycle rules, tiering, multipart, change streams, S3 Select
- Tamper-evident audit and object provenance chains
- Quota, metering and cost attribution per workspace
- All five access methods
- Lakehouse tables over the same store
The honest framing against a small single-node store: this is not the same thing plus governance. It is the full substrate with the distribution layer quiescent, and adding a second cell turns on erasure, relay, anti-entropy and replication with no migration, because the shard model already describes both.
Five ways in
| Method | What it is for |
|---|---|
| S3 with SigV4 | Existing S3 tooling, including presigned URLs |
| Native REST | The full surface: namespace tree, protection conformance, key lifecycle, change streams, hooks, audit and provenance |
| gRPC | Streaming, and the only atomic create-if-absent primitive |
| WebHDFS | Hadoop-ecosystem clients |
| SFTP | Workspace-path-shaped access, governed by the same path grants |
The native surface is the richest. S3 is the most familiar, and the most constrained.
What S3 refuses, and why that is the point
Several S3 operations return not implemented on purpose, because the platform will not store a control it cannot enforce:
| Operation | Why |
|---|---|
| Bucket policy, bucket CORS | No evaluation on this surface |
| Enabling bucket versioning | Versioning enforcement is not complete on this surface, so accepting it would be a lie |
| Directory rename | Needs tree resolution the call site does not do |
| WebHDFS append | The store is content-addressed and immutable per version. Emulating append as read-modify-write is a different operation wearing the same name |
Object Lock is enforced, and the history is worth knowing because it explains the shape.
For a long window the configuration endpoint returned not implemented, deliberately. Before that it had done something worse: it accepted a mode and a retention period, stored them, and echoed them back, so a customer's own console confirmed the setting. Nothing read that configuration when objects were written. A seven-year compliance lock could be set, verified, and the objects deleted a minute later.
Three things had to exist before the refusal could be lifted, and all three now do:
- Default retention attaches at the write, to every new object version, on upload, copy and multipart completion. A failure to attach refuses the write rather than acknowledging an unprotected object.
- Per-object retention and legal hold can be set and read directly.
- Governance and compliance differ on delete. A compliance lock cannot be shortened or downgraded before it expires. A governance lock can be bypassed, but only by a caller authorised to do so, and only over a governance lock.
A legal hold with no retention is carried as an already-expired row whose hold is the only thing blocking deletion, which is the correct shape for an indefinite hold.
A WORM hold across cells
Enforcing a lock on one cell is the easy half. In a replicated store a hold has two more ways to fail, and both are closed deliberately.
Convergence can only strengthen a hold. When two cells have edited the same lock row, they are merged by a join that cannot weaken the result: retention takes the later of the two expiries, legal hold is ORed, and compliance dominates governance. The join is commutative, associative and idempotent, so cells that apply the same updates in different orders reach the same answer. A replication conflict can reorder events; it cannot shorten a retention period or drop a hold.
One cross-field rule finishes it: a compliance lock is never left bypassed. If a merge would combine a compliance mode with a soft-delete tombstone from the other side, the tombstone is cleared, because a bypassed lock would be skipped on read and the object would become deletable. Compliance dominance would be decorative without it.
A delete cannot slip through the convergence window. The risk is narrow and real: a peer has already received a compliance or legal-hold lock that this cell has not yet seen, so this cell's local view says the delete is fine. To close it:
| Situation | What happens |
|---|---|
| The local view already denies | Authoritative, answered immediately, no fan-out |
| The local view allows, and the object is under a WORM regime | The cell asks its peers before deleting |
| Any peer holds the lock | Denied |
| A peer is unreachable, under compliance or legal hold | Denied, fail-closed |
| No WORM signal at all | The local view is authoritative |
The reasoning behind the fail-closed row is the part worth keeping: a wrongly permitted compliance delete is irreversible, and a wrongly refused delete is retryable. When those are the two errors available, the choice makes itself.
Across the matrix: on a single cell the peer set is empty, so the check is a no-op and the local view stands. On a multi-cell node the window is closed by asking every peer in the node. Sovereign and air-gapped deployments ride the intra-node mesh over mutual TLS, with no coordinator to reach.
Tiering, lifecycle and retention
Objects move between hot, warm, cold and archive tiers on access patterns and age, with per-bucket lifecycle rules in the familiar shape.
Two governance bounds apply, and both only ever prevent demotion, because a promotion never reduces availability:
A classification floor. Each classification has a coldest tier an automatic rule may move it to, and for the most sensitive the answer is no automatic demotion at all. Confidential data stops short of archive because archive trades retrieval time, and a rule reading "nobody touched this in a year" cannot know about an obligation to produce it quickly.
Retention and legal hold. A hold blocks demotion, and a hold that cannot be read is treated as held.
Security
Encryption runs a three-level key hierarchy: a node key wraps a per-namespace key, which wraps a per-block data key. Where a deployment requires encryption at rest, a block that would land as plaintext fails the write. A silent fallback would produce a system that reports encryption at rest while not providing it, and a plaintext block is indistinguishable from an encrypted one to every layer above.
Classification is enforced at three points: the database constraint on the namespace, the tiering floor, and the lifetime of any temporary remote copy.
That last one is worth stating precisely. A copy materialised on a remote cell to serve a read is a lease, not a cache:
| Classification | Maximum lease |
|---|---|
| Public | One hour |
| Internal | Fifteen minutes |
| Confidential | Five minutes |
| Secret and above | Zero. Nothing is materialised |
An unrecognised classification resolves to zero, fail-closed. A cache is something you tune; a lease expires whether or not anyone is watching, and its maximum lifetime is derived from what it holds rather than configured by whoever deployed the cell.
Authorization layers coarse permissions, role-based access with inheritance down the namespace tree, per-namespace access lists that can grant to an entity, a cell, a service group or an agent, and fine-grained path-prefix deny carve-outs that apply identically across S3, native, WebHDFS and SFTP.
Scoped access tokens are time-limited, permission-scoped, use-counted, revocable and bound to a specific reference, which is strictly more expressive than a presigned URL.
Auditing is unconditional. The switches that would have disabled read, write or administrative logging were removed rather than wired, on the grounds that a setting stating a posture that is false, in the direction an operator would not expect, is worse than no setting.
Tables on top of objects
The lakehouse writes Iceberg and Delta tables onto this same object store, which makes object storage the cross-cell substrate: a table written on one cell is visible on all of them because the store is shared.
The comparison worth drawing is to managed table buckets elsewhere. Those put table files into a purpose-built bucket. Here tables sit on top of the general store, so a table file inherits content addressing, deduplication, compression, encryption, erasure coding, topology-aware placement, classification enforcement and audit for free, and the same bytes remain reachable as objects through every access method.
Both formats are peers rather than one being privileged, and a table can be published in the other format over the same files without copying them.
Managed tables register as first-class catalog objects with their columns projected, so column-level policy has rows to bind to.
Related topics
- The database engine alongside this one: FluxDB.
- Tables over this store: lakehouse tables.
- How placement resolves against physical topology: deployment topology.
- Cross-cell replication in general: platform replication.
- What is measured here: what is measured.