Skip to main content

Data engineering

The pages in Concepts describe what the platform holds. This area is about moving and transforming it, and the through-line is that none of these execution surfaces is an unpoliced build path. A transform reads through the same enforcement plane as a query, emits the same lineage, and meters the same way.

In this area​

PageWhat it covers
Connectors and ingestionWhat you can connect to: 106 real-time protocol drivers across industrial, tactical, healthcare, finance and security, federated providers that read a source in place, and a bounded web crawler
Unstructured and mediaDocuments, audio, video and imagery: local transcription, grounded entity extraction, and the gate that decides whether extraction runs at all
AI functionsFunctions as the user-code plane, one object reachable from five call sites, with an isolation tier computed from the classification of the data it touches
NotebooksImmutable server-hashed versions on named branches, pipeline binding, scheduling, and publication as an API endpoint
Data PipelinesThe distributed ETL fabric: connector drain, transform, sink write, spread across cells with leasing, staging, and governed admission

The through-line​

  • A transform is a governed read plus a governed write. The same predicates and masks apply, the same lineage is captured, the same cost is attributed.
  • Governed admission is on the run, not just the row. A pipeline run crosses an admission gate before it starts, and a value-level personal-data scanner runs on the record path.
  • The isolation is computed. A function's sandbox tier is resolved from code trust, data classification, and the cell's regime. Classification raises the tier, and an author can demand more isolation but never less.
  • It spreads across cells. One pipeline partition runs on one cell, with leasing, draining, and rebalancing rather than static assignment, so a flow is not pinned to where it was authored.