Connect a data source
This is usually the first real thing anyone does. The question behind it is whether your data has to move, and for most sources the answer is no.
Pick the shape first
| You have | Use | Does data move? |
|---|---|---|
| A warehouse or lakehouse (Databricks, Snowflake, Fabric, BigQuery) | A federated connection | No. Read in place, with query pushdown |
| An object store (S3, Azure, Google Cloud Storage) | A federated connection | No. Read in place, in open formats |
| A database (PostgreSQL, MySQL, SQL Server) | A federated connection | No |
| A live feed (Kafka, MQTT, a sensor protocol, a webhook) | A stream connector | Yes. It is a stream; it lands |
| Files, documents, mailboxes, media | An object or file connection | Catalogued always; parsed only if you turn extraction on |
| A website | The web crawler | Yes, bounded by depth, page count and robots |
Federate first. A connection you do not have to copy is a connection you do not have to keep in sync, re-secure, or explain to an auditor.
What you need
| A workspace | The connection is scoped to it, not to the node |
| Credentials stored as a secret | You supply a reference. See below |
| Permission | To create a connection in that workspace |
A connection stores a secret reference, not a secret value. The resolution happens inside the service that owns the data path, and the resolved value never comes back.
Anything credential-shaped in a scan descriptor is rejected at any depth: password, secret, token, API key, access key, private key, client secret. If you find yourself pasting a password into a connection body, you are on the wrong path.
Connect a warehouse or lakehouse
Create the connection with its provider, endpoint and a secret reference, then sync metadata. Schemas, tables and columns land in the catalog. The data does not.
Per-platform specifics, including what each one reads and the caveats worth knowing before you promise anything, are in scanning external platforms. The short version:
- Databricks takes a personal access token or an OAuth service principal, and reads Unity Catalog. Without Unity Catalog it falls back to information-schema queries rather than reporting an empty estate. Full SQL pushdown.
- Snowflake takes account, user, warehouse and role. It also supports a nine-source security-posture extraction, which no other connector does.
- Fabric takes an access token. There is no token exchange, so mint the token yourself, and plan the query path deliberately.
Confirm the connection is real
Do this before you build anything on it:
- Health-checks on create. A connection that fails here is telling you now rather than at query time.
- List schemas. Proves authentication and read scope.
- Run an actual query. This is the step to insist on. A healthy connection is a handshake; only a query proves the data path.
Connect an object store
Same shape: provider, bucket or container, and a secret reference. The store is read in place, with format detection for Parquet, CSV, JSON, Avro, ORC, and Iceberg and Delta metadata.
An object store connection is also what a discovery scan runs against today.
Connect a stream
A stream connector is a running driver, not a query. Configure it, and a worker polls or receives on a schedule.
Over a hundred protocol drivers ship, from Kafka and MQTT through industrial protocols to healthcare, finance and tactical formats. See connectors and ingestion for the inventory.
Two things to set deliberately:
- Push or pull. A pull connector polls; a push connector is verified per connector at ingress rather than accepting whatever arrives.
- Where it lands. A stream becomes a governed subject, so classification and workspace scope apply from the first message.
Crawl a website
Give it seed URLs, a maximum depth and a maximum page count. Both bounds apply, so a site's own link graph cannot run away with your budget.
What you cannot turn off: robots directives are honoured with no override path. A disallow stops the crawl. Politeness is enforced per workspace and per host across the whole estate, so one workspace cannot exhaust another's budget for a domain.
Fetched pages enter the same funnel as any other document, and every fetch lands on an append-only, hash-chained ledger you can read back: pages fetched, bytes, robots skips, rate-limit hits, per domain.
Crawled content is labelled at ingest, and that label escalates but never downgrades. A source marked external trust is refused entry to a clean room, fail-closed. Plan a crawl into a workspace, not a room.
Turn on extraction, deliberately
Discovering a file always catalogues it. Understanding it is a separate decision, because cataloguing costs storage and extraction costs inference.
With extraction off, documents, audio and video land as catalog assets and nothing is inferred. With it on, transcripts and entities become signed observations. See unstructured and media ingestion.
Verify it worked
| # | Check | Why this one |
|---|---|---|
| 1 | The connection health-checks | Proves credentials and reachability |
| 2 | Schemas and tables appear in the catalog | Proves read scope |
| 3 | A real query returns rows | Proves the data path. Nothing before this does |
| 4 | Policy applies | Query as a less-privileged principal and confirm you see less |
| 5 | Lineage captured | The read appears on the asset rather than needing reconstruction |
| 6 | For a stream, messages are arriving | Check subscription lag and the age of the oldest message, not just that the connector is running |
When it does not work
| Symptom | Most likely cause | What to do |
|---|---|---|
| Create is rejected for a credential | You supplied a value, not a reference | Store the secret and reference it. The guard rejects credential-shaped fields at any depth |
| Health check passes, queries fail | Reachable but not authorised for the objects | Check the role or grant on the source side, not the connection |
| A Databricks query is refused by name | The connection has no SQL warehouse | It refuses rather than posting a statement that cannot succeed. Add a warehouse |
| Fabric columns come back as an error | Deliberate. It refuses rather than returning an empty column list | An empty schema would read as a valid table with zero columns and sync a wrong schema silently |
| A connector family answers "unavailable" | No backend for that kind on this cell | It names the kind rather than reporting an empty estate. That distinction is the point |
| A stream connector runs but nothing lands | Wrong subject, or the push ingress rejected it | Check the oldest-message age. A lag count says how many; the age says how stale |
| A crawl stops early | Robots, a bound, or backpressure | All three are visible in the fetch ledger. Robots has no override |
| Documents catalogue but produce no entities | Extraction is off | That is the default. It is a separate switch from cataloguing |
Related
- What each platform reads, in detail: scanning external platforms.
- The full connector inventory: connectors and ingestion.
- Scanning an estate rather than ingesting it: scan an estate.
- Where connections appear in the console: workspace tools.