Engineering story · PostgreSQL Fabric

Rules that never ran

The writer's batch-release rules had been silently unused, workers were grabbing whatever arrived, and 145 of 153 stream consumers were ghosts; how the write path was found out and rebuilt

Rust and LuaPostgreSQL FabricStream-backed write path
Batch sizes as they reached the database As the release rules were written
Seven rows, 150, 200: whatever had arrived since a worker's last read.
0errors raised while the rules went unused
  • 8 of 153registered stream consumers that were alive when measured
  • 47+stale consumers left behind by restarts
  • 3defects found at once

The write path had rules for when a batch should be released: enough messages for a destination, or enough age, with priorities so urgent writes went first. The rules were written in scripts that ran inside the queue store, and they were correct. They were also never consulted. Batches were arriving at the database in random sizes, with several workers contending for the same messages, and the throughput looked nothing like the design.

Three defects

The investigation found three things at once. Workers were reading new messages from the stream directly, which bypassed the release scripts entirely, so each worker took whatever had arrived since its last read. The scripts themselves read their rules from one configuration key while the configuration had been written under a slightly different key, so even when they ran they ran with no rules. And the consumer group had filled with stale consumers from restarts, so pending messages assigned to the dead ones sat unclaimed.

Three defects, each enough on its own
a bypassrelease rulesworkers read the stream directly
a typorules written herescripts read hereone key, two spellings
a hygiene gappending work stuck on dead consumers
Consumer counts illustrative.

The rebuild

The quick fix routed every read through the gating scripts, corrected the key and added consumer cleanup. The lasting fix replaced the path. A coordinator now scans each destination's queue for its count, oldest age, highest priority and payload width, decides which destinations have a releasable batch, and hands each worker one full batch for one destination through an atomic claim that takes the worker's own pending messages first, then stale pending messages from any dead consumer, then new ones, and leaves non-matching messages pending for other workers. Each worker deregisters itself from the group on exit, and a periodic sweep removes any that did not. Messages that fail are retried with growing delays and a per-worker circuit breaker, and a batch that keeps failing is acknowledged and logged rather than left to block the queue, a trade-off I chose knowingly and keep visible.

One claimed batch, filled in order
own pendingfrom dead consumersnew
Illustrative proportions. One destination per batch; other destinations' messages stay pending for other workers.

Why it was invisible

Messages were consumed, rows were written, the queue drained eventually.

The design was being ignored rather than violated.

Messages consumedfineRows writtenfineQueue depthdrained, eventuallyBatch size against the rulesdid not matchConsumers against processesdid not match
The last two are now metrics.

What this demonstrates

Reading a system against its own design rather than against its error log: the symptoms were statistical, the causes were a bypass, a typo and a hygiene gap, and the fix made the design the only path.

The diagnosis and the rebuild were my decisions; AI coding agents, a mix of frontier and open-source models, implemented them under my governance system.