Corey James- 85.9requests per second in a burst of 2,048 simultaneous clients
- 0requests in that burst failed, rejected or corrupted
- 2,740MiB, the decoder's memory budget beside a resident serving model
I engineer for the overloaded case first, make failure modes typed and distinct, and report capacity misses with their cause.
Bounded endpoints under swarm load
FAFOHive serves search to whole teams of agents at once, and under that concurrency its embedding and reranking endpoints saturate. I decided callers would never see a raw HTTP error from those endpoints, only a result or one of two typed outcomes, and I had both endpoints wrapped in a bounded-endpoint primitive to deliver that.
Embedding calls are gated only on cache misses, and observation writes have their own connection pool.
When the benchmark was the bottleneck
FAFOHive also had an observation write that looked slow under benchmark. Before I blame the system, I check that the measurement is not the bottleneck, so I ran the fix in phases that separated the drain, the producer and the benchmark.
phases: drain, producer, benchmark, each judged by an independent gatekeeper against bands set in advance
- a hypothesised cause not specific to the write pathretracted
- the harness forked a process per callthe cause
- direct stream append, bypassing HTTPthe fix
A ladder, and a target I missed
Capabilities model had to serve thousands of concurrent agents from a decoder sharing an NVIDIA Blackwell GPU with a resident serving model. I made two thousand concurrent clients a requirement and tested it with a ladder from one client to 2,048, with defined abort conditions at every rung. A failing rung stops the ladder, so a failure cannot be averaged into a headline.
The sustained target for mixed traffic was written down before the engine existed, and the engine missed it.
The full account is A router beside a production model.
Writing the target down before the engine existed meant the miss could not be reframed afterwards.