Technology · Discipline

ML evaluation methodology

Frozen evaluation sets, pre-registered targets, known-answer controls, a list of results never to quote, and a baseline built to beat me. It tied, and the tie is what I publish.

Capabilities modelFrozen held-out evaluation setDense and BM25 fusion baseline
Recall on 469 frozen queries over 42 tools
  • 41,990tokens of tool catalog the model replaces
  • 15buried-request probes
  • 5embedding models in the bake-off
  • 4 armscrossed over across two cards to attribute a throughput gain

An instrument fixed before the first result

Capabilities model replaces a whole tool catalog with one discovery tool backed by a fine-tuned 0.6-billion-parameter open-weight base model. A model like that is only worth publishing if its evaluation can catch it being wrong, so I built the evaluation as an instrument with fixed rules before any result came in. Its core is a frozen pair: a held-out query set and a set of long probes in which the needed tool is buried deep inside a verbose request. Nothing about the instrument could move to help the model.

Held-out queries

One request, one needed tool.

Buried-request probes

The needed tool sits deep inside a verbose request.

Shapes illustrative.

Comparisons like for like

Every comparison is like for like, and I keep a written list of results that must not be quoted, with the reason beside each.

The 42 toolsAbout two thousand entriesModelBaseline
Compared within a column
The early cross-universe pair; its seventeen-point headline was struck

A baseline built to win

The bar was the strongest classical retriever I could assemble, and I ran it knowing it would be close. Every baseline constant came from a sweep, which gave the baseline the same effort as the model.

Built from
Dense embeddings
BM25
Fused ranking

The bake-off replaced an incumbent chosen by habit.

Disqualified if
Within limits
keptToo slow
latencyToo large
context length

Controls that can catch the team

When one control failed to reproduce, I stopped the whole matrix.

Known-answer controls run inside the harness so it can catch the team as readily as the model.

A control failsMatrix stoppedPrecision tracedPinnedAll rerun
The setting had changed rankings.

Results published when they went against me

The account of the tie is The baseline that tied us. Serving got the same treatment: I wrote the sustained mixed-traffic target down before the engine existed.

Buried-request probes · recall
Baseline0.90Model0.67
Sustained mixed traffic · requests a second
Target200Reached6.9

Both sit beside the advantages that held. The serving gap has a structural memory cause, told in A router beside a production model. The throughput gain of my published 4-bit build of a 27-billion-parameter hybrid recurrent-attention open-weight model was attributed the same way; see A 27B model on one 32 GB Intel card. The methods are standard ones; my contribution is applying them under deadline pressure without bending them.