Structures
Alongside the SQL and analytical engines, the Data Platform runs a data-structure engine, a Redis-class engine of memory-speed structures, governed exactly like a table. It is the engine a caller reaches when the natural shape of the data is a counter, a queue, a set, a sorted leaderboard, a probabilistic sketch, or a stream, not a relation. Because it lives inside the platform, a customer keeps their existing Redis clients and gains classification, workspace scoping, authorization, audit, and cross-cell replication that a cache does not give them. On the customer console this surface is Structures.
Structures and FluxDB are both components of the data service, and you provision, govern, meter and audit them the same way. They are not separate products.
They are, however, separate engines. FluxDB is the SQL multi-model database over SQLite and DuckDB; Structures is a memory-speed command engine with its own store, command processor, write-ahead log and sharding model. Structures does not run on FluxDB and does not use its manager, and a guard test fails the build if it tries.
That distinction matters when you reason about durability and latency, because the two have genuinely different storage models. It does not mean you are buying two things.
In one paragraph
A native RESP2/3 server speaks to standard Redis clients over a raw socket, and behind it sits a governed engine of ~34 structure types, including a full CRDT family, that carries the same classification, scope, authorization, and audit as a table, and replicates across cells. The wire is not an authorization bypass.
Four things it is, at once
| Facet | What it is |
|---|---|
| The type surface | 34 structure types (33 customer-visible; one is internal bookkeeping): strings, hashes, lists, sets, sorted sets, streams, JSON documents, vectors, tensors, dataframes, time-series, geospatial, trees, intervals, bitmaps, roaring bitmaps, Bloom and Cuckoo filters, HyperLogLog, HyperMinHash, Count-Min sketch, t-digest, DDSketch, top-K, Merkle logs, bitemporal records, and a CRDT family (counter, register, flag, set, map). |
| The wire | A native RESP2/RESP3 protocol server: redis-cli and the Python, Node, Go, and Java client libraries connect over a raw TCP socket. Registration is per connection, not per command, so transactions, watches, and protocol state persist across a session. |
| The engine | A 16,384-slot hash ring with hash tags for co-locating related keys; ten eviction policies; MULTI/EXEC transactions with MVCC, snapshot isolation, and WATCH; hierarchical-timing-wheel TTL down to key, field, member, path, row, and column level; Python-native scripting with SHA-cached functions; a slow log and per-command metrics; a cost-based plan optimizer; and time-travel snapshots with namespace diffing. |
| The distribution | Slot-based routing to the owning cell, minimal-movement slot migration on rebalance, and write-ahead-log replication between cells with CRDT merge strategies: last-writer-wins register, positive-negative counter, observed-remove set: that converge deterministically. |
The wire is not a bypass
The most important property of the structure engine is what happens when a standard Redis client connects to it directly. It does not find an ungoverned back door. The connection is deny-by-default until it authenticates, and every command re-enters the same engine-boundary authorizer the HTTP surface uses, bound to a verified directory entity.
*Figure: the Redis wire, governed. A standard client connects over RESP2/3, but the connection is deny-by-default until authenticated, and every command re-enters the same engine-boundary authorizer (amber) as the HTTP surface, bound to a verified directory entity. Only then does it reach the governed engine, whose keys route by slot to the owning cell and replicate cross-cell with CRDT merge.
Table equivalent
| Step | What happens | Governance |
|---|---|---|
| Connect | RESP2/3 over raw TCP, per-connection state | Deny-by-default until authenticated |
| Authenticate | Connection bound to a verified directory entity | Same identity model as everything else |
| Authorize | Every command re-enters the engine-boundary authorizer | Same path as the HTTP surface: the wire is not a bypass |
| Execute | Structure operation on a workspace-scoped key | Classification, scope, audit apply |
| Replicate | WAL shipping to peer cells | CRDT merge converges deterministically |
Two runtimes, one contract
The structure engine is one of the capabilities held to two-runtime parity. The core runs the full Python engine; the Edge Agent carries a matching Rust runtime so a disconnected device has the same memory and coordination primitives it would reach at the core, over the same command contract. The edge slice ships a subset, strings, hashes, lists, sets, sorted sets, and streams (including consumer groups), plus the offline-convergent CRDTs (PN-counter, OR-set, LWW, MV-register), persisted sealed at rest so a captured device yields no structure plaintext. A shared RESP transcript verifies that the core and the edge route and answer a command the same way.
Why it is different
- A cache does not carry policy. The same set or sorted set that a Redis client would keep in an ungoverned cache is here a workspace-scoped, classified, audited asset that replicates across cells.
- CRDTs converge without a coordinator. The counter, register, flag, set, and map types merge deterministically, which is what lets a disconnected edge device and a core cell reconcile after a partition without a central arbiter.
- It is a distinct engine, not the SQL engine wearing a hat. Structures sit beside the SQL and analytical engines on the same substrate, one governance plane, three engine families.
Across the deployment matrix
- Single-cell: the engine, the RESP server, and TTL run locally.
- Multi-cell, keys route by slot to the owning cell; WAL replication with CRDT merge keeps peers convergent.
- Sovereign / air-gapped: no external cache or coordinator is required; the engine is part of the cell.
- Edge, the Rust structure runtime gives a disconnected device the same primitives over the same contract, sealed at rest.
In this section
| Page | What it covers |
|---|---|
| The wire and Redis compatibility | RESP2 and RESP3, the Redis and KeyDB command surface, cluster semantics, and why the wire is not an authorization bypass |
| Concurrency and expiry | Snapshot isolation without fork(), fine-grained locking down to a frame cell, transactions, and TTL that reaches a field |
| Distribution and protection | Slot topology across cells, per-type replica policy, CRDT convergence, spill beyond RAM, and the fence that stops a partitioned primary serving |
Related
- The database substrate, the per-cell stores the structure engine sits among.
- In-engine enforcement, the policy plane the Redis wire re-enters.
- Data lifecycle: TTL, tombstone purge, and compaction that govern how structures age out.
- Core and edge, the two-runtime discipline this engine is held to.