Skip to main content

Data lifecycle

Data does not live forever, and how it ends matters as much as how it is stored. The Data lifecycle capability governs the aging of data, expiry by TTL, the purge of tombstoned rows, compaction of storage, and retention policy, so that a row that should be gone is gone, a deletion leaves a recorded tombstone rather than a silent gap, and storage is reclaimed without a human running a cron job. It is a distinct feature group that ships on by default with the data engines.

In one paragraph​

Data expires by TTL down to the field, path, and column; deletions are recorded as tombstones and then purged; storage is compacted; and retention is a policy, not a manual sweep, with the deletion itself written to the audit record.

The four mechanisms​

MechanismWhat it does
TTL (time-to-live)Time-based expiry driven by a hierarchical timing wheel that reaches key, field, member, path, row, and column level, so a single field can expire without dropping the record around it.
Tombstone purgeA deletion is first recorded as a tombstone, a marker that the row existed and was removed, and is purged later, so a reader distinguishes "deleted" from "never present."
CompactionReclaiming space by merging and rewriting storage after expiry and deletion, so the footprint tracks the live data rather than the high-water mark.
RetentionA policy that decides how long data is kept before it becomes eligible for expiry or purge.

Deletion is audited: the catalog records deletion with tombstones, so a removal is a first-class, recorded event rather than an absence a verifier has to infer.

How a row ages out​

A row's end of life runs through the same policy plane as its reads: retention decides eligibility, TTL expires it, a tombstone marks the deletion, and compaction reclaims the space, with the deletion recorded on the way.

*Figure: a governed end of life. Retention decides whether a row is still within its window; once eligible, TTL expires it (down to the field or column via the timing wheel); the deletion is recorded as a tombstone on the audit record; and compaction reclaims the space before the row is finally purged.

Table equivalent

StageMechanismRecorded?
KeepRetention policyPolicy governs eligibility
ExpireTTL (timing wheel, field/column granularity)
DeleteTombstoneYes: deletion audited with a tombstone
ReclaimCompactionSpace returned
RemovePurgeRow no longer present

Data lifecycle is about the normal aging of data under policy. It is not the same capability as a regulatory right-to-erasure request, which is a separate, partially-built control on the security plane. Keep the two claims apart: TTL and retention age data out routinely; GDPR-style erasure is bounded and covered only for registered providers. See the ceiling on Compliance & Audit.

Why it is different​

  • A deletion is recorded, not silent. A tombstone means a verifier can tell a removed row from one that never existed, the same "counted gaps" discipline the platform applies to its audit chains.
  • TTL reaches inside the record. Field-, path-, and column-level expiry lets a sensitive attribute age out on its own schedule without dropping the row.
  • Aging is a policy, not a chore. Retention and compaction are governed behaviors of the engine, not an operator's scheduled sweep.

Across the deployment matrix​

  • Single-cell. TTL, tombstoning, and compaction run against the cell's own stores.
  • Multi-cell, a deletion propagates as the table's replication policy dictates; tombstones let peers converge on a removal rather than resurrecting it.
  • Sovereign / air-gapped, every lifecycle mechanism runs locally with no external scheduler in the path.
  • Cloud-enabled, compaction composes with object-store offload and tiering as storage options.