Skip to main content

Storage and performance

FluxDB keeps two storage planes apart on purpose, because a live transactional shard and a durable analytical archive want opposite things from storage.

PlaneHoldsOptimised for
Low-latency local storageThe live shard files every access method executes against: SQL, document, graph, vector and time seriesLatency and IOPS. The engine reads these in-process, and mmap means reads come from the page cache without a syscall
Object storageLakehouse tables in open format, plus shard backupsDurability, capacity and cost. Erasure-coded, content-addressed, deduplicating
The live path does not touch object storage

This is the distinction most often misread, so it is worth stating flatly. A FluxDB SQL query, a document lookup, a graph traversal, a vector search and a time-series read all execute against the local low-latency path. Object storage is reached through the lakehouse, where open-format tables are written for engine-neutral access and cross-cell visibility, and for shard backups.

Running live transactional shards on object storage would put a network round trip inside the query path and give up the latency floor that made an embedded engine worth choosing in the first place. That is exactly the trade this split refuses to make.

The inverse mistake is equally real: keeping bulk analytical history on local disk gives up erasure coding, deduplication and cheap capacity, which is why the lakehouse lives where it lives.

What lives where​

The lakehouse namespace carries a deliberate layout per database:

scaleout-{database_id}/
    data/{table_name}/{shard_id}/{flush_id}.parquet
    metadata/{table_name}/{filename}
    backups/{shard_id}/{backup_id}.sqlite

Three things to read off it:

  • Flushed data is Parquet, an open columnar format. Your data at rest is not a proprietary blob, and the analytical engine reads Parquet natively.
  • Flushes are per shard, so a flush is a bounded operation rather than a whole-table rewrite.
  • Backups are the shard file itself, so a restore is a file, not a replay.

Security on both planes​

Neither plane is the trusted one. Both carry enforcement, and they carry different enforcement because the threats differ.

On the query path, row filters and column masks are compiled into the plan of the engine that executes, so a caller reaching a shard gets exactly the rows and columns their grants allow. Workspace scope resolves beneath the query, in the resolution of which physical database a logical workspace database maps to, so scoping is not a predicate someone can omit.

On the object plane, classification is a property of the namespace with a database constraint behind it, enforced in the data path. Objects are content-addressed and verified twice on every read (a CRC against the stored checksum, and a content hash of the final plaintext). Encryption can be made mandatory, where the block store refuses a plaintext write rather than warning and proceeding. Tiering is bounded by a classification floor, so secret and above are never moved automatically and an unrecognised label fails closed to no movement.

The result is that a shard's live pages and its flushed Parquet are both governed, by mechanisms appropriate to each, rather than one being the soft underbelly of the other.

Reasoning about capacity​

Here is how to size the system.

Start from the engine​

A shard is a full embedded engine executing in-process against a local file, so its behaviour is the behaviour of SQLite or DuckDB on your hardware. Both are among the most heavily benchmarked engines in the industry, continuously, by independent parties. Published third-party work is a sound first estimate before you measure anything yourself.

The embedded property is the part that is easy to undervalue. There is no network hop between the caller and the engine, no connection handshake on the hot path, and no serialization to a wire format and back. That is a floor a client-server database structurally cannot reach, regardless of how well it is tuned or how much hardware sits behind it.

Then multiply by the shard plan​

Capacity scales with how many shards you run and where they sit. Because a shard is a whole engine rather than a partition of one server's resources, adding shards adds engines, and adding cells adds their compute, networking and storage to the same pool.

The router pushes predicates and projections down into each shard, so a scatter-gather returns the smallest useful result rather than raw rows for the coordinator to filter. That is the difference between distribution that helps and distribution that just moves the cost.

Then account for the plane split​

Working-set queries hit the low-latency local path. Historical and analytical scans read Parquet through the analytical engine. Sizing them separately is the point of keeping them separate: you buy latency where you need latency and capacity where you need capacity, instead of paying for one profile across the whole estate.

What usually turns out to matter

Raw single-query latency is rarely what constrains a real deployment. The engines underneath are already fast, and most estates run on more hardware than their workload needs.

The constraint that shows up in practice is what happens as the data gets larger, and as it spreads across sites and jurisdictions. That is a question about throughput, placement and durability, and it is answered by shard count, cell count and protection policy rather than by a faster query.

Across the deployment matrix​

  • Single cell. Local shards on local storage, with the cell's own object store behind them. Everything on this page works with nothing else installed.
  • Multi-cell. Shards place across cells by topology, replicas span failure domains, and object storage erasure-codes across them.
  • Sovereign and air-gapped. Both planes are local to the estate. No external storage service is required for either.
  • Cloud-enabled. External S3, Azure or GCS can back the object plane where a deployment prefers it, as an option rather than a requirement.