Technology · Datastore

Redis

Redis Streams carry my execution substrate's command bus with Lua-atomic job promotion, the agent runtime's messaging, and the write-behind queue. Topology is explicit and documented.

Streams, consumer groups, LuaCommand bus, agent messaging, write-behindInstances split by role
Command bus · two streams, one consumer group
Consumer group reads control first
Lua, server-side job promotion

A control backbone for FAFO Core

FAFO Core and Function Forge needed a backbone for job state and real-time control that could take bursts, and I put it on Redis. Each function has pending, processing and dead-letter queues and per-job state keys, and multi-step state changes run as server-side Lua, so each is atomic in one round trip. Control commands travel on a Redis stream read by a consumer group, with a separate high-priority stream so a pause or drain never waits behind bulk traffic.

Pending Processing Dead letter

Container port ranges are leased with set-if-absent keys, and cloud GPU spend is counted per second in Redis and persisted to PostgreSQL. The API integration platform runs on a stock Redis 8 with append-only and snapshot persistence. It holds the entity store, search indexes, write stream and rate buckets, and a Redis lease elects its sync workers.

Messages that survive a crashed reader

Agent Swarm needed agent-to-agent messages to survive a crashed reader. Each inbox reads through a consumer group, unacknowledged messages are reclaimed after a fixed fence, and a Lua script moves poison messages aside so one bad message cannot loop. Registration is an atomic set-if-absent on a hash, and a compare-and-set Lua script guards control of an agent.

AgentOS lets a person watch any agent seat, and taking the keyboard is a Redis lease, so a dead holder simply expires. Agent events also enter AgentOS through Redis.

One inbox
acknowledged
reader crashed
poison
outlined: held by a reader that died; solid: reclaimed after the fence. Poison goes to its own tray.
Leases for singletons
seat keyboardsync worker

The memory capability of FAFOHive captures observations in bursts that could starve search, so its writes go through a Redis stream on their own connection pool. The cache stores MessagePack compressed with zstd, and Garnet works as a drop-in backend.

writes arrivingstream drainssearch
Illustrative shapes. Cached entries are MessagePack compressed with zstd, on Redis or Garnet.

A topology with chosen failure modes

Auth and access plane is where I designed the Redis topology for the estate. I split the instances by role and gave each its own failure mode, including a separate instance for Inference Fabric's GPU container job control.

What each instance does when memory is full
Job queue · OAuth sessionsEvicts the least recent
bounded memory, LRU, append-only on disk
Write-behind cacheRefuses new work
eviction off, append-only on; pressure becomes an error, no write is dropped
Swarm message busEvicts, keeps nothing
large, nothing on disk
GPU job controlIts own instance
drain and idle signals never queue behind other traffic

The OAuth server keeps authorization codes, refresh tokens and registered clients in Redis, and every instance has its own alert rules.

I treat Redis as a component with chosen semantics.

Redis is a third-party server that I configure and program against.