Corey James- 336templated entries the statistical detector quarantined
- 1,958synthetic distractor tools generated for the evaluation
- ≈30Bparameters in the local open model on the panel
Authored from the tool registry
Capabilities model learns its tool catalog from data, and I decided every example would be generated from the tool registry it serves, so retraining after a catalog change is a pipeline run. Seeds come from the tool servers' own listings into a YAML registry.
For each tool a panel writes intents of several kinds: paraphrases, hard negatives from pairs of tools that are easy to confuse, requests that need several tools, and long paragraphs with the real request buried inside. The panel mixes frontier models from several vendors with a local open model, and the work is split per tool and kind so no one model's habits dominate. The authoring models are third-party and open models; the factory, the gates and the evaluation design are mine.
Gates between the panel and the training set
Most of the engineering is in the gates. A statistical detector catches a panel model that scripts templated output and quarantines what it writes. Every item is deduplicated and checked for leakage against the frozen evaluation set, so the evaluation measures generalisation, and items are tagged cold or warm by how far their phrasing sits from the training data.
Templated output looks plentiful and teaches little.
Distractors that pass for real
For the evaluation, a synthetic catalog of distractor tools was generated with its style measured against real entries. Before it was used, it was attacked with classifiers trained to tell real from synthetic, so no classifier can pick out the fakes by style. See A hundred tokens instead of forty-two thousand and The baseline that tied us.
Tens of millions of traceable items
FAFO Core and Function Forge runs my other large generation pipeline: a Rust function that produced tens of millions of model-generated dataset items, each traceable to the knowledge it came from. It submits work through the token-aware broker to GPU inference, stores results with full provenance and recovers buffered items after an infrastructure failure. The design of the items themselves belongs to research. See The workload that shaped the platform.