Corey James- 3Bactive parameters per token, of 30B, in the mixture-of-experts build
- 1,024dimensions in the memory search embedder
Depending on the job, I fine-tune a model, quantize it, convert it, build an inference engine for it, or measure it against alternatives. The work spans most of the model families in use today.
Small dense models as routers
My Capabilities model is a dense model of about half a billion parameters, fully fine-tuned so that a tool catalog lives in its weights. I trained a ladder of small dense models at sub-billion, half-billion and low-billion sizes, and chose the size from measured recall and training time. A half-billion-parameter model became a tool index because a full fine-tune takes minutes.
I size the model to the problem.
I try the standard runtime first. When TensorRT-LLM could not hold my decoding guarantees beside a resident model, I built an engine; see Trying the industrial path first and A hundred tokens instead of forty-two thousand.
A mid-size dense model as a workload
A 7B-class dense instruction model sits at the centre of the Inference Fabric. My first research pipeline interrogated it through vLLM. It was then the source of large-scale probe generation and of hidden-state extraction.
Sequences of every length, decoded together.
One forward pass; the hidden state gathered at the last token; no key-value cache.
See From 44K to 163K tokens per second and The workload that shaped the platform.
Hybrid recurrent-attention models with vision
ActiveKV serves a 27B-class hybrid model that interleaves linear-attention layers with full-attention layers and carries a vision encoder. I quantized it projection family by projection family into 4-bit weight-only and mixed 4-bit/FP8 builds for one 32 GB Intel Arc Pro GPU, measuring each choice on that card. One build is published.
The vision tower, embeddings and output head stay bit-identical BF16 in both builds.
A quantized checkpoint is a build with provenance.
I also built FP8 checkpoints of a 9B-class model with the same hybrid design for TensorRT-LLM. See A 27B model on one 32 GB Intel card and Context beyond VRAM.
Large dense and mixture-of-experts models
I ran a 70B-class dense model through the same quantization method, and quantized a mixture-of-experts coding model for TensorRT-LLM. When a quantization needs calibration data, it comes from the model's own training distribution, never a generic default set.
A weight row in groups of 64, each with its own scale; 4-bit and mixed 4-bit/FP8 candidates, with per-family quality pools.
Experts, a few active per token: 4-bit weights, BF16 activations, built into a TensorRT-LLM engine.
ActiveKV's second-architecture checks used a small dense model from a different family, so the engine is proven beyond one model.
A 30B-class local model as a colleague
A third-party 30B-class model in NVFP4 runs under vLLM on a Blackwell GPU next to my own models. Sharing the card with it is the memory constraint my custom engine was designed around; see A router beside a production model.
The colleague writes training data on my authoring panel and is one of Helix's voice brains; my engine is sized to the memory it leaves.
Encoders, embedders and rerankers
FAFOHive's memory capability searches with a multilingual embedding model behind Text Embeddings Inference and reranks with a cross-encoder. I converted the same embedder to OpenVINO at several sequence lengths for Intel hardware, and it embedded the research lattice through Rust workers.
The bake-off kept the strongest one measured; see The baseline that tied us.
Speech, audio and diffusion
Helix listens and speaks in meetings, and its media service runs image and video diffusion workflows through ComfyUI on NVIDIA and Intel GPUs, with durable job tracking around an unmodified engine.
Runtimes that follow the hardware
I choose, reshape and serve models across these families on NVIDIA and Intel hardware, and the runtime follows the card and the job. My training runs on a single GPU.