Corey James- 17points of recall the first headline claimed, struck before publication
- 469held-out queries, frozen before any model result was reported
- 5embedding models in the bake-off for the baseline
My tool-discovery model had a headline: a large lead in recall over classical retrieval. The results were about to be written up and every incentive pointed at believing the number. I struck it. The comparison was against a weak baseline, and a weak baseline makes any model look good. I replaced it with the strongest classical retriever I could assemble. That retriever tied the model at the cutoffs that matter and beat it on long, buried requests. The tie is the number I publish.
The challenge
The first comparison for the capabilities model was against an external small tool-calling model whose results carried no information: anything would have won. The second was against a retrieval baseline that, on inspection, had been ranking a catalog of two thousand entries while the model chose among forty-two.
Only a fraction of the field is drawn.
Same queries, a far easier choice. Any model wins this comparison.
Constraints
The evaluation set was frozen before any model result was reported. The write-up was scheduled, and the headline was already in a draft. A baseline has to be fair in both directions: it must see the same catalog the model sees, and it must be tuned as carefully as the model was.
My approach
The bar became the strongest classical retriever we could assemble, built with the same care as the model, and the instrument kept known-answer controls so it could catch the team as readily as it could catch the model. Every constant in the baseline was found by sweep. If the model could not beat that, the honest claim was a tie, and a tie on a hundred-token footprint is still a result.
What I built
A frozen evaluation pair: the 469-query set plus fifteen long, buried requests where the needed tool is mentioned deep inside a verbose description.
A full-catalog fusion baseline combining dense embeddings with BM25, with every weight and cutoff chosen by sweep.
An embedding bake-off, which replaced an incumbent that had been chosen by habit.
Disqualifiers for latency and context length, so a baseline that was accurate and too slow or too large to deploy did not count.
A known-answer control inside the harness. When it failed to reproduce during one run, the whole experiment matrix was stopped and re-run after the cause, a numerical precision setting that changed rankings, was pinned.
An experiment matrix that ran every ranker on both catalog universes, so no comparison could again mix a 42-tool model against a 2,000-entry baseline.
Difficult decisions
Choosing the stronger baseline knowing the outcome. I could see before the run that the fusion retriever would be close. I ran it anyway.
A result I would not accept from someone else is a result I cannot publish.
A never-quote list. The headline figure and the comparisons it came from went onto a written list of results that are never quoted, with the reason beside each.
Results
Like for like, on the same 42 tools and the same 469 queries:
The model's surviving advantages, a standing cost of about a hundred tokens instead of forty-two thousand and a decoder that cannot produce a tool name outside the catalog, are what I quote.
Technologies used
Rust evaluation harness · ONNX Runtime · BM25 · Dense embedding models · Pre-registered evaluation design · Known-answer controls
What this demonstrates
Measurement integrity under pressure. I build the baseline that could beat me, I stop the run when the control fails, and I publish the result that hurts.