Streaming intelligence & streaming SQL
The Data Platform does not stop at tables at rest. It carries a real-time query surface over the same governed substrate, so a pattern in a live event stream is matched, joined, and acted on under the same classification, row filters, and lineage that govern a batch read. Streaming is not a bolt-on engine beside the lakehouse; it is the same substrate, queried as events arrive.
In one paragraph
We match patterns over live event sequences, run streaming SQL with temporal joins and watermarks, and speak three query dialects. SQL, KQL, and PromQL, against streams that carry the same classification and governance as the tables next to them.
The surfaces
Six capabilities make up the streaming plane. Each is a distinct query surface, and each runs on named, workspace-scoped streams rather than an ungoverned side channel.
| Surface | What it does |
|---|---|
| Complex event processing (CEP) | NFA-style pattern matching over event sequences, with quantifiers (once, one or more, zero or more), configurable windows, and partitioning: "A then B then not C within 5 minutes." |
| Streaming SQL | A UDF registry (workspace-scoped or system-wide), streaming DDL (CREATE STREAM, derived streams, alter, drop), temporal joins between two streams with a time bound, and emit semantics (watermark, processing-time, per-record). |
| Two dialects beyond SQL | A KQL tokenizer, parser, function registry and translator, and a PromQL parser and engine including rate, increase, and histogram quantiles. |
| Event streams | Named, workspace-scoped streams with Redis-Streams semantics (append with generated IDs, blocking consumer reads, consumer groups with pending tracking), plus deduplication windows, schema enforcement, and trimming. |
| Live queries | Push-based subscriptions with diff detection, so a callback fires only when the result set changes, at a configurable refresh interval. |
| Reflex engine | Threshold and absence conditions that fire an alert or a write action, with suppression windows and per-reflex state, replicated across cells. |
| Streaming tables & materialized views | Streaming-table DDL, a table consumer, a rollup pipeline, a router that splits stream reads from table reads, and incremental refresh for sum/count/average/min/max: full recompute only where correctness requires it. |
How a live event is served
An event does not take a shortcut around governance. It enters a named stream, is matched or aggregated by the streaming engine, and is served under the same policy plane that governs the tables, the streaming path re-enters the shared in-engine enforcement, it does not sit beside it.
*Figure: one event, one governed path. An event lands in a named, workspace-scoped stream (schema-enforced, deduplicated, trimmed); CEP, streaming SQL / KQL / PromQL, and materialized views query it; and the result passes back through the same in-engine enforcement (amber) that governs the tables before it reaches a live-query subscriber or fires the reflex engine.
Table equivalent
| Stage | Element | Governance touch |
|---|---|---|
| Ingest | Named stream (Redis-Streams semantics) | Schema enforcement, dedup window, workspace scope |
| Query | CEP / streaming SQL / KQL / PromQL | Runs against the scoped stream |
| Enforce | In-engine enforcement plane | Classification, row filters, column masks re-applied |
| Serve | Live query (diff-detected push) | Only changed results delivered |
| Act | Reflex engine (threshold / absence) | Alert or write action; per-reflex state replicated across cells |
Why it is different
- One substrate, not a second product. The streams sit inside the Data Platform, so a stream is a catalog asset with a classification and a lineage record, not an external message bus a governance layer never sees.
- Three dialects, one plane. SQL, KQL, and PromQL are parser front-ends over the same governed streams, so a security analyst's KQL and an SRE's PromQL run against data governed exactly like an analyst's SQL.
- Absence is a first-class condition. The reflex engine fires on what did not happen (a heartbeat that stopped) not only on thresholds crossed.
Across the deployment matrix
- Single-cell, streams, CEP, and reflexes run locally against the cell's own event sources.
- Multi-cell: reflex state is replicated across cells, and streaming tables compose with the scale-out plane for cross-cell reads.
- Sovereign / air-gapped: the streaming engine needs no external broker; event sources are the cell's own, and nothing in the path phones home.
- Cloud-enabled: cloud messaging sources (SQS, Kinesis, Event Hubs, Pub/Sub, and others) can be registered as catalog connections behind an egress guard, see Third-party interoperability.
Related
- Stream hardening for classified & regulated work: the chain of custody, classification guard, and cross-domain guard on the streaming path.
- In-engine enforcement, the shared policy plane the streaming path re-enters.
- Scale-out & distributed execution, how streaming tables and queries fan out across cells.
- Messaging & Streaming (service), the message bus the streams ride on.