Corey James- 0.3 sa batch update that had taken more than 25 seconds
- 400 GBof indexes that no query had ever read, removed
- 16,000lines of Rust in the writer and its library, the queue logic in Lua
Every system in my estate reads PostgreSQL directly and writes it through one path. Hundreds of GPU and CPU workers produce results at the speed of the hardware, and a database that received each of those writes as it happened would spend its life holding connections and locks across GPU waits, with write load tracking GPU throughput rather than what the database can absorb. PostgreSQL Fabric is the layer that turns that firehose into batched, ordered, admitted writes, and that carries the work which belongs inside the database to the database.
The idea
Producers never hold a database connection for a write. Each write is serialised into a stream message with its destination, its operation, its payload and the width of that payload, and the producer moves on. One writer service consumes the streams, groups messages by destination table and operation, and writes them in batched transactions. An admission controller, separate from the writer, measures the database's own pressure and spends it. Lattice-wide operations, the redistribution passes and health computations over millions of rows, run as SQL functions inside PostgreSQL rather than being dragged out through a client.
Statements shaped for the database
Bulk updates bind each column as one array and unnest the arrays into a row set, so the number of parameters equals the number of columns whatever the row count, and the planner joins on the key instead of materialising a derived table. That change and the index removal are the story in Twenty thousand rows a second. Batches are sized per destination from the parameter limit and the payload's width, which the producer stamps on every message, and writer workers scale with queue depth, up fast and down slowly, within bounds that came from a benchmark: A write path that sizes itself.
Rows times columns: the count grows with every row until the protocol's bind-parameter limit stops the batch.
Six columns, six parameters, however many rows the arrays carry; unnested into a row set the planner joins on the key.
The queue as a design, and the time it was ignored
Each worker takes one full batch for one destination through an atomic claim, partial batches release by priority and age, and consumers deregister themselves and are swept when they do not. That design was once silently bypassed: workers read the stream directly, the release rules sat under a mistyped key, and almost every registered consumer was a ghost. The diagnosis and the rebuild are in Rules that never ran.
Admission from the database's own pressure
A sampler reads the write-ahead-log rate, active writers, lock waits and per-function statement statistics into a pressure index. Token buckets in several dimensions refill fully when the database is healthy, partially under warning, and not at all at critical pressure, with gradual recovery. Row writes travel one lane and heavy in-database functions another, run one at a time under a lock that cannot be held by a crashed or stale holder. The controller ran in shadow mode, computing every decision without enforcing one, before it was allowed to refuse a write: Pricing writes by what they cost the database.
Measured at the source
Throughput is read from the database's own tuple and commit counters, never from queue length, because one message can carry thousands of rows or none. The deadlock investigation in 196 deadlocks to zero was done that way, and the in-database redistribution engine of Redistribution inside the database runs on this fabric.
The judgment in it
I removed the batch-size settings from the operator's tool surface.
A self-tuning system that can be hand-tuned will be mistuned.
I kept the admission controller separate from the writer, so the writer stays simple and admission can be run in shadow mode against the real queues.
I chose to acknowledge a batch that keeps failing rather than let it block the queue, and I keep that trade-off visible rather than hidden behind a dead-letter mechanism that does not exist.
I count index maintenance as part of the cost of every write, and I drop indexes that no query reads.
What it demonstrates
Database engineering at the scale of a GPU estate: PostgreSQL's limits handled as engineering constraints, throughput measured where it is true, and a write path whose sizing and admission are derived from the database rather than configured by hand. The admission controller is about 5,000 lines of Rust.
Stories
Twenty thousand rows a second · A write path that sizes itself · Rules that never ran · Pricing writes by what they cost the database · 196 deadlocks to zero · Redistribution inside the database · The database decides when an index goes stale · Rebuild only what moved · Three shards that move together
Technologies
PostgreSQL · PostgreSQL bulk-update engineering · Write-behind persistence · Admission control and rate limiting · Redis streams and Lua · sqlx · Database performance diagnosis · pgvector · Rust