Engineering story · PostgreSQL Fabric

Pricing writes by what they cost the database

An admission controller that reads the database's own pressure, write-ahead-log rate, active writers and lock waits, and admits work only as fast as the database can take it

Rust, Lua and SQLPostgreSQL FabricFleet-wide admission
Read from the database itself, combined into one pressure index WAL growthwriterslock waitsper-function timehealthywarningcritical
For the database as a whole and for each function. Readings illustrative.
0–100the scale admission spends against
  • 4pressure signals, all read from the database's own statistics
  • 3modes: off, shadow and enforce
  • 1lattice-wide function running inside the database at a time

GPU-scale producers can push a database into trouble long before any query fails. The write path described in A write path that sizes itself keeps the database from being hammered by individual workers; the admission controller keeps the whole fleet from exceeding what the database can absorb.

It measures the database and spends the measurement.

Pressure, read from the database

A sampler reads the database's own statistics, the write-ahead log, its writers, its locks and its statement statistics. Each signal is normalised between a healthy level and a critical one, and the signals are combined into one pressure index. The weights are tuned to this estate and stay private; the shape of the index does not.

Write-ahead logActive writersLock waitsLatency, for everyoneQuery errors
Shapes illustrative; the line across the write-ahead log is its flush rate. Each rising trace is a pressure the database reports about itself.

Tokens that refill with headroom

Admission is a token bucket in several dimensions at once: log bytes, rows and write slots, per lane and per schema. When pressure is healthy, the buckets refill in full. In a warning band they refill partially. At critical pressure they stop refilling until the database recovers, and recovery is gradual so admission does not oscillate. Each batch presents a cost estimate for its function, and an atomic check in the queue store admits it or holds it. Schemas take turns so one tenant cannot monopolise the buckets, a batch that has waited gains priority with age, and an emergency priority exists for the writes that must land regardless.

One bucket after the same interval, by pressure band
Healthyrefills in full
Warningrefills partly
Criticalno refill
Illustrative levels. Log bytes, rows and write slots each have a bucket like this, per lane and per schema.

One at a time for the heavy work

Lattice-wide functions that run inside the database, the redistribution passes in Redistribution inside the database, travel the bulk lane and run one at a time, under a lock with a heartbeat and a time-to-live, so a crashed holder cannot block the lane forever and a stale holder cannot release a lock it no longer owns. Row writes on the main lane are gated per function and schema.

Main lane · four functions writing at once
admitted at full pace
a slow function, held back
A slow function throttles itself and nobody else.

Shadow before enforce

The controller has three modes: off, shadow and enforce. In shadow mode it computes every decision and logs what it would have done without holding anything, which is how its thresholds were set against real load before it was allowed to refuse a write.

Shadow
Enforce
would hold, logged only
held

What this demonstrates

Rate limiting derived from the resource being protected rather than from a fixed quota: the database says how much pressure it is under, admission spends against that, and the heavy work is serialised where the contention would be worst.

The pressure model and the lane design are mine; AI coding agents, a mix of frontier and open-source models, implemented the controller under my governance system.