Corey James- 0.955recall at 5 on 469 frozen queries; the best classical baseline scored 0.949
- 2,048simultaneous clients in one burst, none failed or rejected
- 2.7 GiBfor the whole discovery engine on the shared card
An agent connected to a serious tool estate pays for that estate on every turn: the schema of every tool it might call is loaded into context whether it uses one or not. On my estate that was 44 tools across three servers, measured on the wire. The quieter failure is worse: an agent that cannot see the right tool concludes it has none and improvises. I replaced the catalog with a single discovery tool backed by a 0.6-billion-parameter model that has the catalog memorised and cannot name a tool outside it. Around that model is a complete small-model system: a registry-driven data factory, a CUDA decoder whose output bits do not depend on the batch, a serving engine that lives beside a larger model on the same card, and a frozen evaluation whose strongest baseline tied the model.
The idea
Treat the catalog as something a small model can memorise, and treat the decoder as the place to enforce validity. The agent sees one tool and describes what it needs. The model returns pointers to the tools that fit. The registry supplies full schemas for only those. A grammar compiled from the registry is applied during decoding so that names outside the catalog cannot be produced; the guarantee is validity, and the model can still pick the wrong real tool. The description of that one tool is written as a trigger, telling the agent to look before concluding no tool exists, and its size is capped by a unit test. See A hundred tokens instead of forty-two thousand.
What I built
A full fine-tune of a 0.6-billion-parameter open-weight base model on a corpus generated from the tool registry. Every parameter trains, a run takes minutes, and the model is rebuilt whenever the catalog changes.
A purpose-built CUDA decoding engine: batched and continuous, with paged key-value memory, admission against measured memory, canary-gated binding and typed refusals. It shares one GPU with a 30B model. See A router beside a production model.
Reductions ordered so that a request's output bits are identical across batch compositions: one block per output row, fixed-order shuffle trees, no K-split tiling, and the same row kernels for the serial and batched paths. See Bit-identical across batch compositions.
A frozen evaluation instrument: 469 held-out queries over 42 tools, fifteen long buried-request probes, known-answer controls that can catch the team, and a baseline built with the same care as the model. See The baseline that tied us.
A vendor-path study that tried the industrial serving stack first and closed it on measurements. See Trying the industrial path first.
In the same lane, my published 4-bit quantization of a 27-billion-parameter hybrid recurrent-attention open-weight model for Intel hardware. See A 27B model on one 32 GB Intel card.
Results
No run produced a tool name outside the catalog. The sustained-throughput target I pre-registered for mixed traffic was missed by a wide margin, for the memory reason drawn below. That result and its cause are in the serving story.
The judgment in it
Full fine-tune, not adapters, because at 0.6B parameters the cost that justifies adapters was not there.
Discovery only. The model finds tools; the agent calls them. A wrong answer stays small.
I measured the break-even for fetching schemas on demand, about three lookups per turn, and documented it instead of hiding it.
My first headline had the model beating classical retrieval by seventeen points. The tie is what I publish.
The baseline was weak. I struck it, built the strongest retriever I could assemble, and the model tied it.
What it demonstrates
End-to-end ML systems architecture from data to weights to kernels to protocol, framing a problem so a small model is the right tool, and measurement integrity under pressure.
Stories
A hundred tokens instead of forty-two thousand · The baseline that tied us · A router beside a production model · Bit-identical across batch compositions · Trying the industrial path first · A 27B model on one 32 GB Intel card · Constraint made cheaper than no constraint · Every request accounted for · The recovery that drained the card · Prove the mechanism before the model · Reading a file format by correlation
Technologies
PyTorch · Hugging Face Transformers · Full-parameter fine-tuning · Grammar-constrained decoding · CUDA · Rust · Model Context Protocol · Pre-registered evaluation design · Weight-only quantization · NVIDIA Blackwell GPUs · Intel Arc GPUs