Skip to main content

Node log rolling

Every node writes its own logs to its own disk, and those logs roll so they never grow without bound and fill the volume. This page covers the rolling and retention mechanics on a node: how a log file is rotated, how many are kept, and the one distinction that matters in a distributed estate, per-cell logs are local diagnostics that do not federate. For the wider telemetry and structured- logging pipeline this sits under, see Monitoring & logging.

Rotation is in-process on both runtimes. There is no logrotate unit and no platform-managed journald retention. What differs is the trigger: the edge rolls by time, the core rolls by size.

How a node rolls its logs​

Figure: node log rolling, both runtimes. The outlined edge runtime rolls its single log stream daily (UTC) and keeps the 14 most recent daily files. The filled core runtime writes one stream per service and rolls each by size, keeping a fixed backup count with a directory byte quota as a second, independent bound. Both write to the cell's own disk, and the amber node states the outcome: logs stay on the cell and never federate, an estate-wide view is reconstructed from correlation IDs stamped into the lines, not by shipping log files. Ceiling: rotation is in-process on both (no logrotate, and platform packaging sets no journald retention, so any journald copy follows the host's own defaults); the edge thresholds are compiled-in constants, not operator-tunable, while the core's are config-driven; only metrics roll up into an estate view, logs do not.

Table equivalent

RuntimeRotation triggerRetention bound(s)PathOperator-tunable?
Edge AgentTime: daily (UTC)Keep the 14 most recent daily files; oldest pruned<state_dir>/logs/s9.log.<date>No (compiled-in constants)
Core (cell)Size: per-file maxBytesbackup count of rotated files plus a directory byte quota (prunes oldest)logs/<env>/<service>/<service>.logYes (logging: block, per-service overrides)

The edge: daily, with a 14-file cap​

The Edge Agent writes one log stream per device (or per named profile) to a logs/ directory under its state directory, /var/lib/s9/logs/s9.log when it runs as a daemon, ~/.s9/logs/s9.log for an interactive user, or a profile root when a profile is named. A rolling file appender rotates the file daily on the UTC boundary and keeps the 14 most recent daily files, pruning the oldest so the state directory cannot fill. The stem and the 14-day retention are compiled-in constants, the only operator-facing log controls are verbosity and the log filter, not the rotation thresholds. The writer is non-blocking and best-effort: if the directory is not writable, the file layer is simply disabled and the console stream continues, so logging never blocks the agent.

These files are explicitly local diagnostics, nothing here is transmitted off-device. (A separate mechanism buffers error/warning events for the heartbeat drain, but that is telemetry, not log-file shipping, and it is distinct from Proactive Support.)

The core: by size, with a backup count and a directory quota​

A cell routes each record to a per-service log file, admin, compute, data, messaging, metrics, node, security, storage, system, under logs/<env>/<service>/<service>.log. Each file rotates by size: when it exceeds the per-file byte limit it rolls to a numbered backup and the oldest backup beyond the backup count is deleted. A second, independent bound then applies, a directory byte quota enforced after each rollover that sums every file in the directory and deletes the oldest rotated files until the directory is back under quota. The rollover is hardened for multi-process operation: it uses a delayed open and an atomic replace, suppresses shutdown races, and never raises into the caller, a lost rotation must not break the operation being logged.

Config key (logging:)Default in codeMeaning
max_file_size / max_log_file_size10MBPer-file rotation size
backup_count10Rotated files kept per log
max_log_dir_size1GBTotal directory byte quota (prunes oldest)
file_handler_scopemain_onlyWhether child processes also write files

These are overridable per service and per environment; shipped configs commonly tighten the per-file size (for example to 1–2 MB) and the backup count. The directory quota is the backstop that holds even when a single service logs heavily.

Per-cell logs versus the estate view​

The property that matters across a fleet is that log rolling is per-cell, and logs do not roll up. Each cell rotates and retains its own files on its own disk; there is no cross-cell log federation. What crosses cells is the correlation ID, stamped into every structured line and propagated across the edge-to-core boundary, so a single request can be followed through many cells' logs even though the files themselves never leave the machines they were written on. This is the deliberate counterpart to metrics, which do federate into an estate view; logs stay local and are stitched by identity, not by shipment. See Monitoring & logging for the correlation and redaction plane this rides on.

Log rolling is not the Activity Ledger

Rolled, retention-capped log files are high-volume operational diagnostics meant to be pruned. The Activity Ledger is a different plane entirely, an append-only, hash-chained, per-workspace record of what was done, retained as an account of activity, never rotated away as debug output.

Across the deployment matrix​

  • Single-cell: one cell rolls its own files locally; nothing depends on a peer or a collector.
  • Multi-cell: every cell rolls independently; there is no coordinator and no shared log volume. Correlation IDs let a request be traced across them.
  • Sovereign / air-gapped: rolling and retention run entirely on-device with no external dependency; because logs never egress, no rotation choice can leak data off a disconnected cell.
  • Cloud-enabled: identical local behavior; an operator may additionally pipe the structured stream to an external aggregator, but the platform ships nothing automatically.