Corey James- 32,768tokens of model context the voice requests outgrew before the fix
One wire shape for every agent model
In Agent Swarm agents use models from my own GPUs and from hosted providers, and I did not want provider-specific code anywhere in the agent loop. I standardised every model on the OpenAI-compatible chat and tool-calling interface, so routing, budgets and tool checks are written once. Local open-weight models served by vLLM already speak it. For hosted providers that do not, I built small bridge services that translate their APIs into the same shape, rate-limit in Redis, and register and deregister themselves, so the agent loop never learns a vendor's API.
With one interface, the router picks an endpoint by policy and load.
Checking every tool call before it runs
Tools go out as function definitions in the standard request. Every tool call that comes back is checked against the tool's JSON Schema in strict mode before it runs, because local models produce malformed calls often enough to matter. The coding agent in A coding agent that never holds the keys ran over this path.
A seam between the voice engine and its models
In Helix the same interface is the seam between the voice engine and its language models, so I can change models without touching the engine. The engine streams server-sent events from an OpenAI-compatible endpoint, one of them a 27-billion-parameter hybrid recurrent-attention open-weight model with prefix caching on an Intel Arc Pro GPU, and speaks clause by clause as text arrives. The seam exposed a real defect in how tools reached voice requests, and I removed a step to fix it.
Tool catalogs folded into the request until one overflowed the model's context; a simple question waited many seconds for its first word.
The answer streams at once, and a tool is resolved only when the model asks for one.
Adopting ActiveKV at no cost to clients
ActiveKV changes where vLLM keeps a model's attention history, and I wanted adopting it to cost clients nothing. It comes as compiled binaries plus a patch to the user's own vLLM, and clients keep talking to vLLM's standard OpenAI-compatible server. The benchmark matrix behind every ActiveKV figure drives that same server with an ordinary client. See Context beyond VRAM.
Clients and the benchmark harness talk to vLLM's standard OpenAI-compatible server.
A common interface lets one agent runtime span local and hosted models, lets a voice engine swap models freely, and lets a deep change to an inference engine land without its clients noticing. The convention belongs to vLLM and the providers; the bridges, router and validation are mine.
I use a common interface as an architectural boundary.