Skip to main content

Discovery & Mapping

Discovery & Mapping is how System9 learns a data estate: it discovers and profiles sources where they live, normalises what it finds into one canonical semantic model, and keeps that picture current from the edge leaves that do the scanning back to a trunk that holds the merged view. It runs on the same cell, identity and audit model as the rest of the platform, so a discovered object carries its classification and its provenance from the moment it is catalogued.

Scan in place. Move nothing.​

The guardrail is a contract, not a comment, and it is printed on the deliverable the client receives.

Every scan submission and every scan detail carries the data-movement answer, which is none, together with a per-depth explanation of exactly what touched the estate, an immutable snapshot of the scope that was authorized, and a record of how the scan authenticated.

DepthWhat it reads
Shallow (the default)Catalog metadata only. No rows are read
StandardPer-column sample statistics
DeepFull profiling

Higher depths require an explicit budget uplift. The default is the one that reads nothing.

What it actually connects to​

A stored connection: provider, endpoint, and a secret reference. Never an inline credential.

The credential boundary is enforced by a pure function that walks the scan descriptor at any depth and raises if it finds anything credential-shaped. The descriptor that leaves the planner carries only a handle.

Resolution happens in the service that owns the data path, audited, and the resolved secret opens the client and never leaves that process.

What it reads, exactly​

For relational sources, system catalogs only: no data manipulation, no schema changes, no temporary tables, no locks, with result limits to bound what is read.

For object stores, listings with format detection (Parquet, CSV, JSON, Avro, ORC, and Iceberg and Delta metadata), size, content type, modification time and entity tag.

For an access-graph scan, metadata only: principals, privileges and policy bindings. The returned inventory is credential-free.

Refusal beats a clean report​

A scan is refused before a row is created when no backend exists for the kind being scanned.

That ordering matters. The alternative is a scan that sits pending behind a missing backend, or worse, completes and reports an empty estate. A clean report on an estate that was never looked at is the most expensive wrong answer this system could produce, so it is structurally prevented rather than warned about.

Bounded by construction​

Every scan is a parameter object with three hard bounds:

  • Scope: a finite set of catalogs, schemas and object patterns.
  • Budget: maximum objects, bytes, wall-clock and compute. Hitting a cap ends the scan as over-budget. It never retries or escalates.
  • Depth: the ladder above.

The model the scan produces is a validated intermediate representation with a deterministic encoding, so the same estate re-scanned produces byte-identical output. That is what makes drift detection meaningful rather than noisy.

Structured and unstructured, in one model​

Structured sources yield objects, columns and relationships, with sample-based profiling: null rates, distinct cardinality (exact when small, estimated above a threshold), numeric ranges, confidence-graded type inference, row counts, candidate primary keys, and suggested inclusion dependencies across tables. From there the platform lifts entity-relationship graphs and star schemas.

Unstructured sources run the discover, parse, understand, govern chain: file shares, object stores, Office documents, content platforms with delta cursoring, source code, mailboxes, medical imaging and imagery.

The bridge between them is a schema-fingerprint match: a document-derived object is matched to a catalog table with a compatible shape, so a quarterly report and the table behind it are recognisably the same thing.

Governance findings​

Scanning produces findings, not just an inventory.

Sensitive-data classification is deterministic: column names and sample values against patterns, producing findings for unlabelled and unmasked sensitive data. Labelled and masked columns produce no finding. An optional model-based pass layers onto the same finding contract, so the scorecard never changes shape depending on whether it ran.

Coverage introspection is pure metadata: missing owner, missing documentation, missing lineage, unset retention.

The scorecard rolls those into an overall health score across four dimensions, a control-gap grid against common regulatory and AI-governance frameworks, and a prioritised remediation roadmap. At most one live assessment per engagement: a new run supersedes the last rather than accumulating.

Storage findings are conservative by design. Where age, ownership or hash data is absent, the rules that depend on them do not fire, rather than firing on a guess.

Assessment​

Eight lenses ship: data, security, cloud, virtualization, data security, storage, storage exposure, and AI governance. Each declares its status, and a test cross-checks every shipped lens against the live runner so the catalog cannot drift from reality.

The engines behind them are pure functions. The runner is dispatch, map and write: one run identifier, findings persisted, nothing clever in the middle.

Fan-out ships findings, not bytes. Targets are partitioned across cells, with data-gravity pinning where it matters, and each cell returns findings. Findings can then be materialized into a governed table or the knowledge graph so they are queryable with lineage captured automatically.

Rules of engagement, signed​

For a regulated or government engagement the procurement question is not whether the scan found things. It is whether the scanner could have touched anything outside the authorized scope.

So the scope is signed. The authorized scope is canonicalized and then signed, which means re-serializing it still verifies while any semantic change breaks the signature. Out-of-scope targets are denied at the boundary, and an engagement can only assess targets it names.

Where no signing key is provisioned, the system refuses to sign rather than producing a signature that proves nothing.

A second, separate signature covers the deliverable: the per-engagement hash chain proves integrity but is anonymous, so approving a deliverable signs the approved content into a manifest the client re-verifies independently. Integrity and attribution are different claims and get different mechanisms.

Sovereign engagements carry an impact level and export-control marking, require a recorded approving authority before the record is stored, and refuse a lower-partition enumerator, fail-closed.

Deliverables​

Reports render from one structured source into both markdown and self-contained HTML: estate inventory by kind, the read-only guardrail audit, recommendations by severity, migration plan and model evaluation.

Diagrams are generated as pure functions, one per kind, topology only. They never invent data: an empty source yields an empty but valid document rather than a placeholder. Export is Mermaid, editable draw.io, or deterministic SVG laid out to the same coordinates as the on-screen canvas. A diagram kind with no wired source raises rather than rendering a plausible picture.

Automation​

Recurring scans decide what is due from schedules and the clock, and a missed window catches up to a single next run rather than a backlog burst. Only the live-scannable lenses auto-run; the lenses that need supplied inventory are deliberately not scheduled. The recurring path reuses the interactive run pipeline exactly, so there is one path rather than two that drift.

Continuous workers run three ticks: drift detection diffing the current inventory against the prior snapshot, a recommender that emits deltas, and scan health. Each is best-effort and logs rather than failing a cell.

The consulting assistant proposes plans against a declared tool catalog, each tool carrying its own required permission. Step approvals are a first-class table, so human-in-the-loop is in the schema rather than in a convention.

Edge leaves, and reconciling into a trunk​

A device or a remote cell scans locally and ships signed inventory, not raw data. Four policies decide how much work happens where, chosen from the device's capability and the central budget:

PolicyWhat leaves the edge
Full at edgeInventory and relationships only
SummarySchema fingerprints and counts. Raw bytes never leave
SampleSchema plus a small sample
Central onlyThe edge triages type and structure; extraction happens centrally

Convergence is a conflict-free replicated type: a per-object register keyed on scan provenance, newest wins, with deterministic tie-breaking, so merge order does not matter. Every entry is verified under the authoring leaf's key, and an unsigned entry, an unknown key or a wrong key is dropped rather than converged.

Anti-entropy compares roots first and drills in only where they differ, so reconciliation bandwidth is proportional to actual divergence.

Approved decisions flow back to devices: object-type assignments, sensitivity labels, schema decisions, quality grades and deletion flags.

Air-gapped transfer​

Where there is no link, inventory moves as a signed bundle on physical media, and an operator command ingests it. Per-record dispatch is isolated so one bad record does not poison a bundle, while a bundle-level failure aborts entirely, because a tampered batch cannot be meaningfully partly applied.

What is available today, and what is not​

AreaStatus
Object-store estate scans, end to endAvailable
File-import scans and the semantic-model parsersAvailable
All eight lenses, findings, supersede semanticsAvailable
Governance scorecard, sensitive-data classification, coverageAvailable
Diagrams, reports, signed scope, signed deliverablesAvailable
Leaf convergence, anti-entropy, trunk reconciliation, air-gapped bundlesAvailable
Continuous drift, recommender and health workersAvailable
Relational, business-intelligence and lineage-tool scanningAvailable. A relational scan walks a real database through a live backend; the business-intelligence connectors inventory published content (workbooks, dashboards, reports, notebooks) and emit lineage edges back to the sources behind them; the lineage-tool connectors read an existing catalog's objects, tags, descriptions and lineage for migration onto this one
SaaS scanningConnector interfaces defined, with backends pending. A scan refuses by name rather than reporting an empty estate
Cross-cell scan handoffReceiver mounted, fails closed pending the cross-cell scan engine
Cloud enumerationAWS today; other partitions register as explicitly pending

Across the deployment matrix​

A leaf scans and normalises entirely locally, and the signed inventory is a local projection, so discovery holds sovereign and air-gapped with no coordinator, and the offline bundle is the deliberate answer to the no-link case. It works single-cell today. The honest multi-cell boundary is the one stated above: the leaf↔trunk anti-entropy logic exists and is signature-verified, but continuous catalog convergence across cells is blocked by the identity gap, so a fleet's catalogues do not yet converge live, they exchange by signed bundle instead.

Run it at the edge​

An Edge Agent acts as a discovery leaf through the s9 discovery-mapping command family. Register it, scan a source in place into the canonical model, and export a signed bundle for the trip home:

# Register this device as a discovery leaf, then scan a source in place
s9 discovery-mapping register --workspace-id ws-acme --node-endpoint https://node-a.system9.local
s9 discovery-mapping scan --workspace-id ws-acme --source /mnt/share \
    --policy edge_summary --leaf-id leaf-gateway-07

# Inspect, then export a signed sneakernet bundle and browse it offline
s9 discovery-mapping status --detailed
s9 discovery-mapping export-bundle --workspace-id ws-acme --out /media/usb/leaf.tar.zst
s9 discovery-mapping frontend --bind 127.0.0.1:9090

Scan policy is one of edge_full, edge_summary (default), edge_sample, or central_only; --leaf-id authors the result as a signed inventory delta. The full walkthrough (including carrying the bundle to a trunk) is the tutorial Discover a data estate and carry it across a gap.