Corey James- ~44%of the base's parameters in feed-forward layers
- 28layers, all with full attention
A model rebuilt on every catalog change
Capabilities model holds an agent's tool catalog in the weights of a small model, and the catalog changes every week. If retraining took a day, the model would be stale on arrival. I chose a full fine-tune of every parameter of a small open-weight base model with Hugging Face Transformers, on a corpus generated from the tool registry. At this size the run takes minutes, so the model is rebuilt whenever the registry changes.
I ran a full fine-tune over adapters, because at this size the cost that usually justifies adapters was absent and the full fine-tune removes a layer of indirection.
a frozen base and a small trained layer beside it
every parameter trained
Choosing the base by its architecture
I chose the base model by checking each candidate. The product is a catalog recalled from a short query, which is a parametric-knowledge task, so a standard feed-forward transformer was a requirement.
- Configuration read: where the parameters sitcounted
- Full attention on every layer, no sliding windowrequired
- Loads under the installed release with remote code offrequired
- Tokenizer probed: the chat format applies without vocabulary surgeryrequired
- The newest small base in the same family, on its configurationrejected
I choose models by reading their architecture against the task, and I make training cheap enough to repeat on every catalog change.
A capacity ladder
The capacity ladder changes total size while keeping the architecture comparable, so the comparison measures capacity alone.
Handing weights to my own engine
Trained weights leave Transformers as safetensors and load directly into my own CUDA decoding engine, and parity checks guard that hand-off. This is fine-tuning of an existing base model with Transformers used as provided. The full problem is A hundred tokens instead of forty-two thousand, and the evaluation that judged the model is The baseline that tied us.