The catalog and lineage
The catalog is the single place to answer "what data do we have, who owns it, what is it made of, where did it come from, and what will break if I change it", with the answer scoped to a workspace and carried across cells.
The namespace
The catalog is addressed with a consistent three-level model,
catalog → schema → asset, using one catalog.schema.table scheme across every
engine. An asset is not only a SQL table. It can be a view, volume, function,
connection, model, stream, structure, graph, vector space, or document
collection, so the catalog is complete across engines rather than SQL-only.
Assets are organized into domains with owners, and a domain registry resolves effective policy over the hierarchy, so governance can be delegated to a business unit without losing the estate-level view.
What the catalog holds
| Object | What it is |
|---|---|
| Assets and domains | The catalogued tables, objects, streams, and every other asset kind, organized into domains with owners |
| Data products | Datasets treated as products with typed input and output ports and a contract, so a consumer knows the shape and the guarantees, not just the location |
| Semantic and dimensional models | A modeling layer, including a transformation-project layer using the vocabulary analysts already work in: projects, models, seeds |
| Provenance | A provenance envelope on a product and a provenance record on the read side, so a downstream user can see what a product was derived from |
| Policy | Tag and classification policy over catalogued assets, evaluated as part of the governed read |
The metadata over every asset
Figure: every asset carries four metadata planes. The contract enforcer is consulted on catalog writes and blocks an operation that would violate an active contract (amber, held). Lineage is written to a per-cell tamper-evident chain and read through a viewer-aware visibility filter. Ceiling: tag taxonomies and classification policy are describable; automatic tag propagation along lineage is not a claim, see the note below.
Table equivalent
| From | Relationship | To | State (meaning) |
|---|---|---|---|
| Asset | carries | Governance metadata (owner, glossary, tags, classification) | neutral |
| Asset | is bound by | Data contract (enforced on write) | refusal / held (amber) |
| Data contract | blocks | a write that would violate it | refusal (amber, dotted) |
| Asset | is scored by | Quality score (7 dimensions, A+…F) | neutral |
| Asset | records | Lineage on a per-cell tamper-evident chain | verified (green) |
Governance metadata
A business glossary with an approval workflow, hierarchical tag taxonomies, stewardship assignment, and the five-rung classification ladder (public, internal, confidential, secret, top secret) with rules that block export, share, or download at configured levels and require recorded justification to downgrade. Classification is evaluated as part of the governed read.
Contracts enforced on write
Data contracts cover schema, freshness, and quality. The distinguishing move is that an enforcer is consulted on catalog writes: an operation that would violate an active contract is blocked, rather than the violation being discovered later in a report.
Quality and metadata intelligence
- Quality scoring across seven weighted dimensions (completeness, freshness, documentation, governance, usage, quality, lineage), graded A+ through F.
- Auto-enrichment with personal-data detection across nine pattern families and domain inference across nine domains.
- A metadata event bus, usage analytics, recommendations, and breaking-change detection.
Lineage as a by-product of the read
Lineage and provenance are captured as a by-product of the read, not produced as a separate documentation exercise.
- Column-level lineage with a SQL lineage collector that extracts relationships by parsing queries, plus impact analysis and coverage measurement, so "what breaks if I change this" is answerable after the fact rather than reconstructed by hand.
- A capture guard at the engine write boundary, running before the mutation commits, written into a per-cell tamper-evident chain and read through a viewer-aware visibility filter so the provenance graph does not reveal the existence of data a viewer cannot read.
The guard has three modes, and the default is observe
Every mutation arriving at the write boundary is classified as exempt, allowed, gap-recorded or rejected:
| Mode | An uncontexted mutation |
|---|---|
| Observe (the shipped default) | Durably records a coverage gap and allows the write |
| Simulate | Records the gap and flags what enforce would have done, still allowing the write |
| Enforce | Rejects the write |
A replicated write is exempt, because the edge was already captured on the origin cell and re-capturing would duplicate it.
Be precise about this: deny-by-default describes the guard's classification, not the shipped behaviour. Out of the box a mutation that cannot be captured is recorded as a gap and allowed. Enforce is opt-in, and simulate exists so an estate can see what enforce would reject before turning it on.
The property that is true by default and worth the claim: a non-covered tool cannot silently evade capture. It trips the guard and leaves a durable gap, so coverage is measurable rather than assumed.
- Ingest tokens let external emitters contribute lineage without broader access, and a propagation notifier tells downstream owners when an upstream change affects them.
Lineage and provenance capture, and classification policy, are what this page covers. Two neighbours are worth scoping separately: automatic propagation of classification tags along lineage, and enforcement of data-product service levels, are on the roadmap. SLA definitions exist and are carried on the contract today.
Lineage goes out as well as in
Capturing lineage is only half of what a data platform owes an estate that
already runs DataHub, Marquez, Atlan, Collibra or Purview. Captured lineage
publishes outward as OpenLineage RunEvent documents to an operator-configured
catalog, so System9 joins an existing lineage graph instead of asking to replace
it.
Two properties make this safe to turn on, and one makes it useful.
It is off unless someone with authority turns it on. Sending lineage outside the estate is egress, and it is treated as such. The endpoint, namespace and producer identity are ordinary configuration, but the enable flip is carried by a signed capability, not a settings toggle. With no signed enable and no endpoint, which is the sovereign and air-gapped default, no transport is built and no network call is ever attempted. This is not a feature that has to be remembered and disabled; it is one that has to be granted.
Publishing cannot wedge a write. The transport posts through the same egress guard the platform uses for outbound webhooks, which refuses non-public and rebinding targets and keeps TLS identity pinned to the hostname. Every failure, whether a refusal by that guard, a timeout, a network error or a non-success response, is swallowed and logged. An unreachable external catalog degrades lineage publishing, never the lineage write itself, and never the transaction that produced it.
The graph is node-scoped, so a sharded node publishes one coherent picture. This is the detail that is easy to get wrong. A node's cells each capture their own shard's lineage, so if each cell published under its own identity, an external tool would see a node's lineage arrive as several disconnected graphs. Instead the OpenLineage namespace is derived from the node identity, and specifically from the same canonical node identifier the audit layer stamps on every operation. Two consequences follow:
- All of a node's cells publish into one graph.
- That graph joins to the audit record on one identity, so a lineage edge seen in an external tool and the audited operation that produced it are the same node, not two things that have to be correlated by hand.
Export configuration itself resolves through the platform's configuration policy plane rather than per-cell local files, and it converges across a node's cells with narrower scopes winning. A node is uniform by default; a site or zone beneath it can deliberately differ. That is what stops a half-configured node from publishing a silently incomplete graph.
Events are published unsigned over server-verified TLS. A third-party catalog is not a mutual-TLS peer in the platform's trust fabric, and no chain, uplink or mutual-TLS material is touched on this path. Treat outbound lineage as interoperability, not as an extension of the estate's cryptographic trust boundary.
Four co-equal distribution modes
The catalog is not a single central service, and it does not distribute across cells in only one way. There are four modes, and they are co-equal. A deployment picks the one that fits each source, and the same governed namespace presents all four. Only the first copies metadata between cells; the other three leave the data where it lives and reach it in place.
Figure: four co-equal ways the catalog spans cells and reaches outside. Only replicated (green) copies catalog metadata between peers. Grafted, federated, and foreign leave the data in place and reach it: a mounted subtree, a decomposed cross-database query, or an external system registered as foreign tables.
Table equivalent
| Mode | What crosses the wire | Copies data? | Use it for |
|---|---|---|---|
| Replicated | Catalog change events; anti-entropy against peers with per-peer vector clocks and a replication log | Yes: metadata | A peer cell holds its own current copy of the catalog |
| Grafted | Query references only | No: transparent proxy | Mount a remote catalog subtree into the local namespace for cross-node queries without moving data |
| Federated | Decomposed sub-queries and their results | No bulk movement | One SQL query spanning several databases, executed per-database and joined |
| Foreign | Metadata registration and query pushdown | No: stays in the source | Register an external system (PostgreSQL, MySQL, SQL Server, Iceberg REST, file collections) as foreign tables |
Grafting adds an enable, disable, and detach lifecycle, configurable cache TTL, per-graft security checks, and a graft topology view. Foreign integration adds auto-discovery, incremental sync with change detection, and bulk metadata sync. Above these four modes, a federated-semantics layer can glue models across cells covering one domain by exchanging signed projections rather than raw data. The deeper topology treatment, the nine replication scopes and how a mode is resolved from topology, lives in Replication, DR & federation.
Domains: delegating governance without losing the estate view
Assets organize into domains with owners, and a domain registry resolves effective policy over the hierarchy: ancestors, descendants and subtree rollup. Domains carry their own administrators, workspace templates, and foreign-domain registration.
The point is delegation without fragmentation. A business unit governs its own domain, and the estate-level question is still answerable in one place, because the domain is a node in one hierarchy rather than a separate catalog.
Health, quotas and operations
A catalog that spans cells needs to be operable, and these are the surfaces that make it so rather than leaving it to grow unbounded.
| Surface | What it does |
|---|---|
| Catalog health aggregation | Across engines, so "is the catalog healthy" has one answer rather than one per engine |
| Quota enforcement | Checked before asset creation, not discovered afterward |
| Usage events and access-pattern observability | What is actually being read, and how |
| Credential lifecycle | Rotation for the credentials catalog connections hold |
| Deletion audit with tombstones | A removed asset leaves a record that it existed and was removed, so a reader distinguishes deleted from never present |
| Background workers | Reindex search, and run catalog and workspace anti-entropy |
The catalog and the directory replicate together
The catalog answers what data exists and who may reach it. The directory answers who exists: organizations, nodes, users, agents, devices, service accounts, groups, roles, delegations and crypto profiles.
They are separate domains with separate stores, deliberately, and neither can query across into the other. But they are the two that replicate most widely, and they replicate together, because an access decision needs both halves present locally:
- across the cells of a node, so any cell can answer without a round trip;
- across the nodes of an organization, and into nested organizations.
That co-replication is what makes a local answer possible. A cell resolves a three-part name, resolves the calling entity, resolves its grants, and compiles the policy into the query plan, without reaching for a remote directory service or a central catalog. It is also why a disconnected cell can still enforce policy: both halves of the decision are already on the box.
So what: an entity minted in one place is resolvable, verifiable and addressable elsewhere, and the asset it is asking for is catalogued there too. That is the difference between a distributed catalog and a catalog with a cache in front of it.
Interoperability
The catalog is readable by existing metadata tooling: OpenMetadata-compatible REST endpoints for tables, topics, databases, tags, glossary terms, lineage, and full-text search, so a customer's existing catalog and lineage tools can read this catalog rather than being replaced by it. See catalog interoperability.
Catalog export renders summary JSON, JSON, DDL, CSV, Parquet and OpenLineage, and the advertised format list and the produced format list are the same list by construction, so asking for a format that is not offered tells you so rather than returning something shaped like it. There is no Apache Atlas projection: if it is a requirement, treat it as not present. See catalog interoperability.
Across the deployment matrix
- Single-cell: the catalog is local and complete for that cell's assets.
- Multi-cell: replication and federation give each cell a local view of a shared catalog, and a workspace's catalog spans cells.
- Sovereign and air-gapped: the catalog answers from the cell, with no external directory in the path.
- Cloud-enabled: federation to foreign and cloud catalogs is available as an option.
Related
- Lakehouse tables, the open formats the catalog registers.
- Open sharing, how a catalogued asset leaves the estate.
- Data products, a first-class catalog object with a contract.
- Classification and lineage, the enforcement side of these tags.
- Governed reads, where classification policy is evaluated.