Data Pipelines

A pipeline is a graph of steps that drains a connector, transforms records, and writes a sink, spread across cells and governed on the way. It is the platform's distributed, multi-cell ETL fabric, and the thing that makes it different from a scheduler is that every stage of it sits inside the same policy, lineage and metering plane as a query.

One word, two subsystems

Do not conflate them. Data Pipelines is a data-movement fabric; Model Routing is a model-selection ladder. They live in different platforms, solve different problems, and share nothing but an internal engineering word. When a reader says "cascade," establish which one they mean.

The distributed ETL fabric

The unit of a pipeline is a partition, and one partition runs on one cell as a single flow: connector drain → transform → sink write, over a normalised record that carries a row from any source to any target in one shape. Decode is total, a record is never silently dropped on the way in.

Around that core, the fabric spreads one logical flow across cells and governs every run.

*Figure: one logical flow, many cells. A splitter fans a connector across partitions; the dispatcher clears a governed admission gate (amber), selects a cell, leases a slot and fails over on error; a remote partition ships to the cell where the data lives over a TLS-gated, fail-closed transport, and is staged with preview/commit/discard (cyan) before it is committed to a sink (green). Ceiling: distribution across cells is realized as cell selection + leasing + failover, not an active rebalancer. A connector reporting running does not by itself prove live data is flowing. See the connector ceiling below.

Table equivalent

Stage	Role	State
Connector drain	Poll a source into normalised records (decode never drops)	neutral
Splitter	Fan one connector across N partitions by stable hash	neutral
Dispatcher	Net-potential-gain go/no-go, cell selection, lease, failover	refusal-capable
Transform	Apply transforms, including AI transforms and Functions	neutral
Remote partition	Ship to the cell where the data lives, TLS-gated, fail-closed	refusal-capable transport
Staged buffer	Durable preview → commit / discard before the sink	candidate
Sink write	Materialise to the target	verified

Dispatch, placement, leasing and staging

• Dispatch and placement. A dispatcher, a cell selector that excludes degraded work-zones, and a splitter place partitions; a routing dispatcher ships a remote partition to a peer cell, and a peer transport resolves the peer over service discovery and posts under a TLS-gated system token that fails closed.
• Remote partitions are staged. A remote partition is offered with preview → commit → discard semantics, backed by a staged buffer store and a peer-side manifest registry whose garbage collection is fail-closed, an expired buffer is discarded or quarantined, never auto-committed.
• Leasing and draining. Partitions run under TTL-bounded leases (a counting-semaphore over per-cell lease scopes) with renew and reclaim, and a bounded drain loop turns a connector into batches. Cell-health input from live topology and heartbeat feeds selection. Recovery across cells is failover plus lease renewal, not a standalone rebalancing routine.

Control tables: the operational spine

A pipeline's operational state lives in control tables: a mutable control row plus a hash-chained revisions ledger, so every enable, disable, edit or run-now appends a tamper-evident revision (prev_hash → entry_hash) that a verify-chain endpoint can recompute. Each entry carries per-entry run history and a watermark that advances as runs complete.

Flow definition and declarative pipelines

A declarative pipeline is compiled into an executable flow. The flow layer carries contracts, models with a topological-order validator, a planner, a ${param} resolver, a dry-run preview, JSON serialisation, a control runner, and pipeline-to-flow conversion that turns a declarative pipeline specification into an executable graph.

A decorator-based declarative DSL is also offered, declare tables and views as methods and attach quality expectations that warn, drop, or fail, with a DAG-resolving executor that topologically sorts dependencies, skips the dependents of a failed table, reports per-table records and duration, records per-run lineage with a quality score, and evolves schema under type-widening rules.

Ceilings on the flow graph and the declarative DSL

• Not every node kind compiles yet. The flow planner compiles ingest, materialise, for-each and notebook nodes; transform, expectation, branch, merge and sub-flow nodes (and conditional edges) are not yet compiled. Treat the flow graph as covering the compiled node kinds, not the whole vocabulary.
• The declarative DSL executes table bodies through an injected seam. The DAG, quality-expectation and lineage machinery is real, but the executor runs a table's body only through a caller-supplied function, absent that, a run resolves the graph and reports zero records. The orchestration is built; the body-execution wiring is the caller's to supply.

Transforms: including the Functions call site

Transforms come from a catalog: ordinary transforms, AI transforms (a first-class category that calls the governed inference bridge once per batch, fail-closed), industry-pack transforms, and custom functions. A custom function transform is exactly the Functions call site #1, the ETL transform step runs a user-defined function through the same governed, tiered sandbox as every other function call site.

Materialisation, orchestration, drift and admission

• Materialisation targets. Engine tables, streaming targets, external targets, fan-out to several targets at once, catalog registration with column inference, and a knowledge target that lands a flow's output as knowledge rather than rows (gate-aware and off by default).
• Orchestration. Admission control with a per-workspace cap, idempotent backfill, triggers (cron, upstream, event), a dead-letter queue with replayable remediation, and per-run metering.
• Schema drift is handled two ways, and they are distinct. A pipeline applies a drift policy (none, additive, full, or rescue) that decides whether to add a column, widen a type, rescue an unexpected value into a sidecar, or reject. Separately, the streaming ingestion path chains drift observations by content hash and gates automatic heal behind high confidence and full field coverage, and never auto-applies a person-data heal.
• Governed admission. A net-potential-gain gate clears the whole job at run start, and again per cell and per row; a value-level personal-data scanner observes the actual record values (not just column names) on the staged path.

Edge ETL: plan where the data is

An edge push-down planner decides, given a task and the edge device's advertised capabilities, which volume-reducing transforms push down to the edge and which stay at the core, so a flow can ship findings, not bytes, rather than pulling raw data to the core first. This is how the pipeline fabric respects data gravity across the core/edge boundary.

danger

Connector ceiling: running is not receiving

Most connectors are push-mode and open no transport of their own: a push-mode connector that nothing has pushed into is started, healthy by its own check, and carrying nothing. A lifecycle status of running therefore does not imply live data. The truthful signal is a separate ingress state (not-started / awaiting / receiving) derived from measured records ingested. Quote the connector catalog as a catalog; verify an individual connector's ingress before demonstrating it.

Not to be confused with model routing

Confidence-gated model routing, where a request is answered by the cheapest sufficient model tier and escalated only when confidence is low, is an unrelated capability that happens to share an internal word with this one. It is documented with the model tiers it belongs to, under models and sovereign AI.

In the deployment matrix

Data Pipelines is multi-cell by design: single-cell, a partition drains, transforms and sinks locally; multi-cell, the splitter, dispatcher, remote partitions and peer transport spread one flow across cells, and the edge push-down planner keeps work near the data. Sovereign and air-gapped, the same fabric runs with no external call, and the governed admission gate and value-level scanner apply identically. Model Routing honours the same corners: it defers to the placement ladder for cost and locality, and in air-gapped profiles the frontier rung is simply unreachable, so the cascade terminates locally.

Related topics

• AI functions, the function object whose ETL-transform call site runs as a pipeline step.
• Notebooks, a notebook binds into a pipeline as a task node (execution reachability is stated precisely there).
• The AI system (core + edge). Model Routing and the confidence-gated cascade in the context of the whole AI stack.
• Lakehouse & Catalog, the tables a pipeline materialises into.
• Data Governance, the enforcement plane the admission gate and value-level scanner belong to.
• compute service, the execution plane that runs pipeline transforms and functions.