Skip to main content

Coordination and shared state

Fan-out is the easy half. The hard half is many workers on one problem without doing the same task twice, overwriting each other, or funnelling every decision through a parent that becomes the bottleneck.

The control plane and the data plane are separate​

The governing split:

  • The control plane runs through the parent. Spawning, scoping, budgeting and revoking are the parent's business, and every one is audited.
  • The data plane is peer to peer. Workers post and claim work directly against a shared board, so a fifty-worker team does not serialise on its coordinator.

A parent that had to broker every message would be the slowest component in its own team.

The blackboard​

The shared board is not a bespoke store. It is built on the platform's own structure engine, which means it inherits classification, workspace scope, authorization, audit and cross-cell replication rather than reimplementing them.

Each coordination need maps to the structure that actually fits it:

NeedStructure
Task board: post and claimA stream with consumer groups
Claim ledger, for exactly-onceA hash with compare-and-set keys
File tree or manifestA JSON document
Files against build and test statusA columnar frame
Shared artifactsA hash
Team signalsPublish and subscribe
Per-path edit lockA compare-and-set key with a lease
Repository-level run lockA compare-and-set key with a lease

Access is scoped by lineage against the team's membership, so only members of that team can read or write its board, and it is garbage-collected when the team dissolves.

Claiming a task exactly once, across cells​

This is the mechanism that makes a distributed team correct rather than merely parallel.

Figure AC1: claiming a task. Two compare-and-set gates, not one. The reason this holds fabric-wide rather than only within a cell is that every command for a given key serialises on the cell that owns that key, so two workers on two different cells racing for the same task cannot both win. The losing worker is told it lost and moves on, rather than duplicating the work.

Crash before completion is handled by lease, not by hope. A claim carries a reclaim window; past it, the task is redelivered and re-acquired under the same gate. Completion verifies the claimant still holds the lock, so a worker that was preempted and then finishes late is refused rather than overwriting the result of the worker that took over.

Redelivery is bounded. Past a delivery limit the task is dead-lettered with its failure metadata and a supervisor notification, and the dead-letter depth is readable. A task that poisons every worker that touches it stops being handed out.

Tiers, and the rule about them​

A board exists at three tiers: local to a team, cell, and fabric.

The load-bearing rule is:

Spill up. Never start up.

There is deliberately no interface to post a task above local. A task can only reach the fabric tier by having been posted locally, gone unclaimed, and spilled. A task is visible only at its current tier, and only an unclaimed task spills.

That single constraint is what stops the fabric board becoming a dumping ground that every team in the estate has to scan.

Leases and elections​

Coordination beyond a single team runs on leases:

  • Service-group leadership is a lease. Winning it publishes the leader's endpoints; losing it withdraws them, so routing follows the lease rather than a stale assumption.
  • Cross-cell singletons use a node-converged lease with exactly one row per scope across the node's cells. This matters more than it sounds: a per-cell lease table means every cell wins its own copy of the "single" lease, which is the classic way a scheduled job silently runs N times in an N-cell node.
  • Scheduled work is deduplicated the same way, so a due schedule fires once.

Leases can be configured to fail open for idempotent work and closed for anything that mutates shared state, because those two want opposite behaviour under uncertainty.

For genuinely collective decisions there is a voting plane with quorum, super-majority and several tally algorithms over append-only ballots. Eligibility is fail-closed and deliberately narrow: workers compute, coordinators deliberate.

Everyone can see the population​

Every running session, run, sub-agent, shell and job registers in a live work registry, time-leased, carrying its parent, its lineage, its depth, its resource class and its current supervision signal.

That registry is what makes admission contextual rather than blind: a new spawn can see its peers, a cell can see it is already saturated, and the fabric can see its own population against a ceiling.

Messaging between agents​

Agents message each other with a typed envelope whose message type is a closed enum, so an unrecognised type is a decode failure rather than something passed through and interpreted loosely.

Two properties are the ones that matter:

The sender cannot be forged. The acting agent is resolved server-side from the authenticated call. It is never a field in the message body, so an agent cannot send mail as another agent.

Peers inform, never command. Messages default to informing. Only a verified, authorised sender may issue a directive that re-tasks the recipient. A peer cannot simply tell another peer what to do because it asked confidently.

Inbound mail is drained before a run's reasoning loop starts, so an agent begins its turn already holding what its peers sent. Two safeguards on that path:

  • Peer content enters on an untrusted channel and is screened by the adversarial-input validator, rather than being spliced into the trusted system context. Another agent's text is input, not instruction.
  • A drain failure never fails the run. Mail is not on the critical path.

A workflow graph can also block on a message, with an inbox node that waits for a correlated reply, which is how one team's run waits on another's result without polling.

Handing work to another cell​

Work crossing a cell boundary travels as a signed, hash-chained envelope, size-, hop- and skew-bounded, so a receiving cell verifies provenance without trusting the transport that carried it.

Where the cross-cell paths stand

Agent work dispatched cell to cell runs on the compute dispatch plane, which answers unconfigured rather than pretending when it has not been set up.

The workflow runtime's own cross-cell handoff is constructed but not carrying work: it is deployed without a signing key, so a peer placement fails and is recorded as failed rather than being silently skipped. That is the right failure mode, and it is worth knowing which of the two planes a deployment is relying on.

Device-to-device coordination inside an enclave, two edge devices coordinating with no cell between them, is on the roadmap.