The data service
The data service is the query and governance plane of a cell. It runs the
database engines, holds the catalog, presents the lakehouse, executes pipelines
and streams, resolves the ontology, scales queries out across cells, and speaks
the external database protocols. It is also where clean rooms and the
in-engine Data Security enforcement plane live, so it feeds both the data
platform and the intelligence platform.
Feeds: Data Platform · Intelligence Platform.
What it owns
- Database engines: a multi-model engine, a time-series engine, and a Redis-compatible data-structure engine.
- Catalog: replicated, distributed, federated, and grafted; see Lakehouse & catalog.
- Lakehouse. Iceberg and Delta at parity: on managed tables.
- Streams and pipelines: streaming queries and ETL, plus web ingestion.
- Ontology: federated semantics over the catalog.
- Scale-out: distributed query execution across cells.
- External protocol surfaces: the database wire protocols external clients connect through.
- Clean rooms: the multi-party work boundary.
- Data Security: one in-engine row/column enforcement plane across every engine type; see Governance.
What the data service carries
It is the largest service in the platform by a wide margin, and the clearest way to read the list below is as the whole data plane of a cell, not a component of it.
Engines and stores
| Area | What it does |
|---|---|
| The database engine | A resilient data access layer over a transactional core and an analytical core, with connection pooling, read and write lane separation, per-database write locking, and workspace database resolution beneath the query |
| Model-specific providers | Document collections, a property graph, vector spaces and time series, over the same substrate rather than as separate products |
| Structures | 34 structure types including a full CRDT family, over a native Redis-compatible wire |
| Time series | A tiered engine: raw samples hot, rollups warm, Parquet cold, behind a write-ahead log |
| Managed tables | Iceberg and Delta as co-equal formats, reading zero-copy over object storage |
| Scale-out | Sharding, topology-aware placement, two-phase commit across cells, scatter-gather federation, and workload-classified routing between the transactional and analytical engines |
Catalog, semantics and modeling
| Area | What it does |
|---|---|
| The catalog | The namespace over every engine, the metadata planes, contracts enforced on write, quality scoring, and four co-equal distribution modes across cells |
| Lineage | Column-level, deny-by-default capture, written to a per-cell tamper-evident chain and read through a viewer-aware filter |
| The ontology | Object, link and action types with staged changes, verified evolution chains and SDK emitters |
| Federated semantics | The cross-cell semantic layer, exchanging signed chart projections rather than raw vectors |
| Data modeling | Transformation projects compiled into a governed dependency graph, plus dimensional modeling with conformed dimensions and slowly changing dimensions |
| Metadata and discovery | Search indexing, faceted and hybrid search, and the background workers that keep the index current |
Movement and arrival
| Area | What it does |
|---|---|
| Data pipelines | The distributed ETL fabric: connector drain, transform, sink write, spread across cells under leasing, staging and governed admission |
| Ingestion | Stream connectors, file and document ingestion with format detection and chunking, and web crawling with robots policy and per-domain politeness |
| Extraction | Turning arriving bytes into governed, typed observations |
| Streams | Complex event processing, streaming SQL, KQL and PromQL, live queries and a reflex engine |
| Real-time intelligence | The live query and reaction surfaces over arriving data |
| Features | Feature groups and materialisations for the ML path |
| Replication | The data domain's own replication adapter, with CRDT convergence |
Governance and boundaries
| Area | What it does |
|---|---|
| Data security | One enforcement plane: row filters and column masks compiled into each engine as native predicates and projections, with ten masking strategies and predicate protection |
| Governance | Classification, tag policy, schema governance and data lifecycle |
| Clean rooms | The multi-trust-domain boundary, with aliasing, signed notebooks and the egress gate |
| The Activity Ledger | The per-workspace and per-room hash-chained activity record |
| Trust | Workspace trust edges and cross-organization validation |
| The Data Exchange | Signed cross-organization listings, offline verification and the accrual ledger |
| Sharing | Delta Sharing as server and client, workspace shares, and public datasets |
Workspace-facing surfaces
Overview, endpoints, secrets, health, catalog bindings, automations, and lens registration. These are the surfaces a workspace console reads, and they are part of this service because the data plane is what a workspace mostly is.
Several of the areas above have internal names that are deliberately not product names, so they are described by what they do rather than branded. Where a capability does have a cleared public name, it is used and linked.
APIs and surfaces
The data service speaks the standard database wire protocols so that
existing clients connect unchanged:
- the PostgreSQL wire protocol for SQL clients (JDBC / ODBC /
psql), with in-band SSL negotiation; - Arrow Flight SQL (gRPC / Arrow) for columnar clients (DuckDB, Pandas, Polars);
- a Redis / Valkey-compatible surface for the data-structure engine.
Inside the cell these bind loopback for intra-cell callers; the multi-cell mesh binding uses mutual TLS. External clients never connect to the data service directly, they go through the node service's protocol gateways, which proxy to the data engine. See the Ports & protocols reference.
The eight external faces
Beyond the wire protocols, the data service exposes eight external HTTP faces, one per engine family, so a caller can reach a store directly rather than only through SQL.
| Face | Reaches |
|---|---|
| Document | Document collections |
| Graph | The property graph |
| Vector | Vector spaces and similarity search |
| Search | Full-text and faceted search |
| Structures | The structure engine over HTTP, alongside its native Redis-compatible wire |
| Time series | Registered series, range and latest queries, aggregation |
| Iceberg REST catalog | External compute engines reading tables in place |
| OpenMetadata | Existing metadata tooling reading the catalog |
Each is self-authenticating: the caller presents a per-database credential that is verified in-handler and resolves to a real directory entity, because an external BI tool or JDBC client has no operator session. The credential is the principal, deny-closed on any failure, and the request then re-enters the same data-security evaluation as any other read.
Two planes, outward and inward
The symmetry here is deliberate and is worth understanding, because it is why the same policy applies to a BI tool and to an agent running inside the cell.
The outward plane exposes every engine to external consumers with external credentials, locations, mounts and network rules, and a per-database credential to directory entity to wire-authentication path.
The inward plane is the mirror image: one polyglot client that co-located callers, the query executor, background workers and agents, use to reach every engine through a single security-enforced surface, rather than each subsystem inventing its own route to the data.
Neither is a shortcut around the other. They are two doors into one enforcement plane.
Workers
Background workers cover clone TTL expiry, cross-cell logical-drift detection for managed Delta/Iceberg tables, structure shard snapshot and WAL checkpointing, catalog replication, and pipeline scheduling. Each runs per cell.
The query and governance plane
Figure: the data service's plane. Every external client passes the node gateway (amber, deny-by-default) before reaching the engines; Data Security is drawn amber because it enforces row/column policy in-engine and can refuse a read; scale-out is outlined because it fans a query across cells. Ceiling: enforcement is applied in-engine across engine types; the diagram shows the governance path, not a per-query plan.
Table equivalent
| From | Relationship | To | State |
|---|---|---|---|
| Client | reaches engines only through | node gateway | deny-by-default |
| Database engines | catalog and resolve semantics via | Catalog + Federated Semantics | neutral |
| Database engines | are filtered by | Data Security (row/column) | refusal-capable |
| Clean rooms | enforce sharing through | Data Security | refusal-capable |
| Database engines | fan out across cells via | Scale-out | edge (cross-cell) |
In the deployment matrix
data runs identically in every corner. Single-cell, scale-out and mesh
bindings are idle and every engine is local. Multi-cell, the catalog replicates
and grafts across cells and scale-out fans queries across them. Sovereign /
air-gapped, all engines, the catalog, and Data Security run with no external
dependency; the Data Exchange
moves data between organizations through signed offline bundles.
Related topics
- Data Platform · Lakehouse & catalog · Governance · Data Exchange.
- The two work boundaries, workspaces and clean rooms.
- Ports & protocols reference.