Corey James- 6leakage rules between the delivery, coordination and durability planes
- 100 msbudget on the live-control stream, kept apart from the command stream
One Redis would do. One queue would do. One tunnel would do. I run several of each, because the roles that share them have different needs, and the first time a bulk job starves a control message or a cache eviction silently drops a write you learn why. Delivery, coordination and durability are separate planes with explicit rules about what may cross between them, and infrastructure is sized and configured per role.
The shape
Name the roles: a job queue, a write-behind queue, a control stream, a coordination bus, a durability store. Give each the configuration its role needs, eviction or no eviction, append-only persistence or not, a tight latency budget or a throughput budget. Put a rule at every boundary about what may cross, and make the rule something that fails loudly.
A pause command waits at the back of a million rows.
The pause arrives at once and bulk carries on.
Where it appears
In Agent Swarm, the fabric keeps its planes apart, and the mesh checks every message locally so peer authorisation never waits on a central round-trip.
In FAFO Core and Function Forge, the write path has a main lane and a bulk lane so bulk work cannot starve ordinary writes, admission is lane-separated by cost, and the control plane keeps a command stream apart from a high-priority live-control stream. The write-path diagnosis in 196 deadlocks to zero was possible because the planes were separable.
In the Auth and access plane, Redis runs per role. Ingress is one outbound tunnel per purpose, so each surface can be cut independently.
- Job queueevictionpersistence
- Write-behindeviction
- GPU job control its own instance
- Coordination buspersistence
The fill under each instance is illustrative.
In the Inference Fabric, the boundary between vector storage and the CUDA engine is a type-only crate, the sole contract both sides compile against.
What it buys and what it costs
A failure in one plane is a failure in one plane.
It buys blast-radius control. It buys the ability to tune each plane for its job instead of for the average. It buys diagnosability, because a slow write path can be measured apart from the queue in front of it.
It costs more infrastructure to run and watch. I accept that, and I document the topology so the cost is visible.
The judgment in it
The question at every boundary is what happens under pressure. If the answer is "the other role slows down", the planes are shared and need separating. If the answer is "the other role is unaffected and this one refuses", the design is right.