Corey JamesEngineering stories
More from consumer GPUsThroughput, memory and context on consumer cards23
Context beyond VRAM
Making a 27-billion-parameter model hold more conversation than its GPU can, exactly, on one card
From 44K to 163K tokens per second
Raising one 7-billion-parameter model's throughput on consumer GPUs, one measured layer at a time
A message broker that understands tokens
Scheduling GPU inference by the shape of the work, so two different cards ran at a 99.9 percent cache hit rate with almost no padding
The workload that shaped the platform
Generating tens of millions of traceable dataset items on two consumer GPUs, and letting that job decide what the substrate needed
Judge a representation by the bytes it stops moving
Rewriting an acceptance gate that kept rejecting good compression, and what it bought at 128K tokens
Bringing a serving stack up on new silicon
Patching Intel's Xe2 kernel stack where it had gaps, with outputs that stayed identical
A 27B model on one 32 GB Intel card
A measured, attributed and published 4-bit quantization of a 27-billion-parameter hybrid recurrent-attention open-weight model for an Intel Arc Pro GPU
Spend structure before spending waste
The lane-packing algorithms that let a GPU broker pad 0.4 percent and keep two different cards at a 99.9 percent cache hit rate
Half the bytes, exactly merged
The INT8 history format and the SYCL attention kernels that halve ActiveKV's bus traffic, and the log-sum-exp merge that keeps the result exact
Staging without a stall
The fence-safe lifecycle and the admission arithmetic that let ActiveKV stream history across the bus without ever reading a stale buffer
Built for Blackwell
CUDA 13 on a Blackwell GPU at the bench, with a provisioning path that reaches datacentre Blackwell capacity in the cloud, and an honest line about what carries over
One call to a GPU anywhere
Provisioning a GPU consumer node on the workbench, on a LAN host or in the cloud with one tool call, under a daily budget that only a human can set
Engines for the card you have
One calibrated FP8 checkpoint, a dozen inference engines sized to two consumer Blackwell GPUs, and the benchmarks that decided what the tiers would be
Shapes have a lifecycle
Every switch of prompt shape costs a GPU its prefix cache, so the broker stages a shape until it is worth running, warms the next one before the current one drains, and alarms when a shape ran too briefly
The 128-lane wall
Padding waste ran under one percent until every shape used all of its 128 lanes, then climbed past 95 percent on a 22-million-item load test; the fix was structural
Starting full and ending clean
A pressure gate holds dispatch until the first batches can be good, every batch closes on its token budget, and a cleanup mode drains the tail when the GPUs go quiet
Throughput that finished nothing
The runtime reported more tokens per second than it ever had while finished sequences fell below one per second; the fix gated new work on how many sequences were already alive
A second engine behind the same door
The vendor's executor could not run engines with no key-value cache or an extra output, so a raw-TensorRT backend went in behind the same C interface, and the Rust side never knew
The queue that should not exist
I built an inference queue manager with two-stage backpressure and permit-carrying requests, then found the runtime already had a token-based queue, and reduced mine to a wrapper
A doorbell the GPU waits on
Batches cross from the CPU to the GPU through a pinned ring and a 64-byte doorbell the GPU's stream waits on directly, with no CPU synchronisation; and why that path is used for the extraction engines and left off for the vendor's executor
The recovery that drained the card
Four failed engine loads took a shared GPU from 1,801 MiB free to 2, because partial allocations were never released; the fix made creation crash-safe and made the admission arithmetic match the allocation it predicts
Prove the mechanism before the model
Three load-bearing ideas behind the custom decoder, a device-side graph loop, a persisting L2 window and a constant-memory trie mask, were each proved on the real GPU in a small lab before the model existed, and one of them contradicted the documentation
Reading a file format by correlation
The four-bit weight format's layout was established by correlating candidate decodings against the known weights before a kernel was written, 0.9955 against 0.0018, and four-bit was positioned as a footprint win rather than a speed win
Proving a system is what it claimsParity, instruments that lied, and tests that passed for the wrong reason13
Rewriting an engine extension without changing a token
Porting a GPU serving layer from Python to native C++ with bit-identical output, and proving every remaining line of Python was required
The baseline that tied us
Retiring a flattering benchmark, building the strongest retriever I could, and publishing the result that hurt
Proving a rewrite is the same system
Parity discipline for a GPU runtime, and the day I voided over a thousand passing trials
Bit-identical across batch compositions
Designing nondeterminism out of a GPU decoder so an agent's choice of tool never depends on its neighbours
The observer that changed the answer
Discovering that my own profiler caused the serialisation it reported, and retracting a verdict I had already acted on
Trying the industrial path first
Evaluating the standard vendor serving stack for a custom decoder, and closing that path on measurements
A detector for a silent corruption
How the hidden-state extraction backend proves which token it is reading before it writes tens of millions of rows
Every metric gets a CPU twin
Thirty-eight GPU cluster metrics defined by a nine-thousand-line CPU reference library, and a three-way consistency check between kernels, references and documentation
Asking the engine what it meant
One input's convention was ambiguous in the vendor's documentation, and guessing wrong would silently extract the wrong token; so at startup the runtime runs a tiny batch both ways and refuses to start unless exactly one answer is consistent
The thirteen percent that was never missing
A profile said thirteen percent of the GPU was idle; it was the share of time in non-matrix kernels. Reading what each metric measured before optimising found utilisation at 98.7 percent
Tests that passed without a GPU
Stub builds returned fake tokens and every test went green; the build now refuses to produce a binary without the real engine library, and the generation contract was fixed at the engine boundary before a corpus was reused
Machinery I built and then did not need
Fixed-size epochs and dense slots were built so decode steps could be captured as CUDA graphs; profiling showed the vendor's executor already does it. What I kept, what I stopped relying on, and the rule I took from it
A proof that proved nothing
An echo-cancellation score looked excellent until an unrelated reference scored higher; the rebuilt proof uses a negative control, and the same discipline runs through the voice engine's tests
Data paths that cannot lose a rowWrite paths, indexes, recovery and exactly-once delivery15
Millions of queued items that were not there
Fixing what 'done' meant in a GPU work broker until the token ledger balanced to zero
196 deadlocks to zero
Measuring a write path from the database itself, and finding that one index was the whole problem
Redistribution inside the database
The lattice functions that grow, protect and score concept clusters over millions of vectors without leaving PostgreSQL
Done means delivered
A result counts as delivered only after it has been written to the receiver's socket, and polling a job costs what is ready rather than what is pending
Recovery that trusts only what happened
Tombstones as the single source of truth, a parallel scan on restart, a stale entry rewound before it is forgotten, a reconnecting GPU that reports what it still holds, and advisory checks that may never decide
Twenty thousand rows a second
A batch update that took twenty-five seconds became a third of a second: one SQL rewrite that makes parameter count independent of row count, and nearly 400 GB of indexes that nobody had ever read
A write path that sizes itself
Batches sized from the payload's own width under PostgreSQL's parameter limit, writer workers that follow queue demand, and the batch-size knobs deliberately taken away from operators
Rules that never ran
The writer's batch-release rules had been silently unused, workers were grabbing whatever arrived, and 145 of 153 stream consumers were ghosts; how the write path was found out and rebuilt
Pricing writes by what they cost the database
An admission controller that reads the database's own pressure, write-ahead-log rate, active writers and lock waits, and admits work only as fast as the database can take it
Ghost bytes in a zero-copy ring
A pinned-memory ring deadlocked with the GPU idle and hundreds of thousands of items in flight, because wrap-around padding was never freed; and a GPU worker that registered twice was starved as dead
An index you can rebuild from its receipt
Approximate-nearest-neighbour indexes are usually irreproducible, so a different result cannot be told from a bug; three hashes, a hash-seeded sample, fixed iterations and a byte-identical rebuild test make mine reproducible
The database decides when an index goes stale
Index invalidation moved into SQL triggers and load-time checks, so a stale index cannot be served, and partial builds are failed before the daemon accepts its first connection
Rebuild only what moved
Column-scoped change capture, a rule that chooses a full or a delta rebuild from the size of the change and never from its age, and a segment merge whose output is byte-identical to a full rebuild
Three shards that move together
Cluster, member and phantom vector files are interdependent, so they activate together in one transaction or not at all, only after the files are confirmed on disk, with a single-active invariant the database enforces
Storage that refuses to become a coordinator
A tensor store that appends, returns a stable address and knows nothing about its consumers; the coordinator tables it once had were removed and the boundary is enforced by checks in the build
Measuring embeddings on the GPUCluster health, diversity and physics-style metrics as kernels7
Thirty-eight metrics in a dozen passes
A hand-written CUDA engine for Blackwell that scores thousands of embedding clusters at once and searches them with the same kernels
Removing the sampling caps
From kernels that looked at fifty members and twelve hundred pairs to exact statistics over a hundred thousand members, by redesigning every kernel into one of three scaling classes
Where a cluster sits
Ten neighbourhood metrics that place each cluster among the clusters around it, computed from precomputed neighbour lists in a handful of GPU launches
Diversity in three kernels
Why entropy over pairwise distances needs a second pass, and how redundancy, information density, coverage and concept breadth come out of the same tiles
Physics as a lens on embeddings
A characteristic length scale from a histogram median, then potential depth, curvature, escape probability, superposition, interference and entanglement as cluster metrics
Two-means as a measuring instrument
Exactly three Lloyd iterations, seeded at the first and last member, as the fixed definition behind clustering tendency, fragmentation, sub-cluster count and silhouette
From health scores to decisions
Five composite scores built from the per-cluster metrics with weights that live in a database table, feeding the pressure model that decides what the corpus does next
Keeping AI agents honestEvidence, separation of duties and control planes that hold the state12
Why long agent runs fail
Diagnosing agent drift as working-memory overflow, and building the control plane that holds the state instead
Directing AI coding agents at program scale
How I run engineering programs with teams of frontier and open-source models, and why evidence, not the agents, decides
Closed by evidence
Building an API integration platform through gated AI work orders, and refusing to close it until every item had independent proof
Agents with real tools and no keys
A governed gateway that lets AI agent teams use payments, email and social-publishing services with every action attributed and no credential in any agent
Separation of duties for AI work
Independent gatekeepers, reviewers from a different vendor, and failures that stay failures
Tools are the platform
One frozen, tested pipeline for every action an agent takes, and catalogs that cannot drift between hosts
A coding agent that never holds the keys
A sandboxed agent on a local open-source model that edits, tests and proposes changes without a repository credential
A code explainer that is not allowed to invent
Compute the call graph first, narrate second: putting deterministic evidence in front of the model
The hardening fix that locked everyone out
Reversing my own security change within a day, because the threat model said a different control was the one that mattered
Tools that build tools, under governance
A generator that lets agents add new capabilities to the estate, inside a sandbox, callable only after a human restart
What the work cost, to the penny
One real engineering program, attributed to the penny, and a cache discipline that returned four-fifths of the bill
Two thousand concurrent workers
Load-testing the memory and code-intelligence service with a fleet of AI agents on one workstation, with zero failures
Small models that constrain big onesTool catalogs, routers and decoders that cannot invent5
A hundred tokens instead of forty-two thousand
Compiling an agent's tool catalog into a 0.6-billion-parameter model that cannot invent a tool
A router beside a production model
Serving two thousand concurrent agents from a 0.6B model in the memory left over on a shared GPU, and reporting the target I missed
Constraint made cheaper than no constraint
A grammar-bounded CUDA decoder in which enforcing the tool catalog costs less than not enforcing it
Every request accounted for
The tool-discovery model's fleet daemon has a nine-point service contract: 512 requests offered to 64 clients with a queue of sixteen gave 244 answered, 268 refused with a typed reason and none dropped, and a shutdown that answered every request in flight
A tool catalogue bigger than the model
Why a voice assistant played its opener and then fell silent, the two-stage fix I rolled back, and the boundary that replaced it
Voice that behaves like a personLatency, interruption, floor control and speech on local GPUs9
A self-hosted colleague in a Teams call
Bridging Microsoft's real-time media stack to a voice AI that never leaves my own hardware
Fast transcripts that were wrong
GPU speech recognition that was far faster than the CPU path and quietly dropped sentences, and the correctness gate every GPU integration in the voice engine now passes
From two seconds to under one
Question to first sound, taken apart stage by stage: the silence window, recognition, reasoning, the first spoken sentence and synthesis, each cut named and each benchmark row carrying its own definition
Building the streaming my speech model did not ship
The open-weight voice model I chose had no streaming hook, so I ported its token loop, decoded its audio in chunks and pinned its diffusion noise so the joins are seamless
Three hundred milliseconds of hello
The lines an assistant says most, the greeting, the thinking fillers, the short acknowledgements, are pre-rendered per voice at session start and play in under a millisecond, through the same cancellation path as synthesised speech
Saying numbers out loud
A chemical formula came out as a sneeze and a thousand as ten thousand; a speech contract and a deterministic scrubber give the voice speakable text while the screen keeps the written answer
Interruptions that only people can make
Echo sounded like a person to the detector; the rules were set so a person always wins the floor, the agent's own voice never does, the first word of an interruption is heard, and a false stop re-speaks instead of going quiet
Listen only to me
An opt-in filter for a desk microphone that hears its operator and not the television, enrolled without keeping a recording, switched by asking, and deliberately not used on Teams, with the reasons written down
One percent network, seventy-eight percent GPU
A slow Teams turn was blamed on the cloud hop; measurement showed the network was about one percent of it and a speech GPU shared with an embedding service was most of the rest, so the compute moved and the cloud stayed
Humans and models in one roomSeats, councils, identity and durable turns6
Solving the digital-employee identity
Working out, from real tenant evidence, how an AI employee can legitimately answer, call and act inside Microsoft 365
A meeting attendee that will not fake it
An AI participant that chats, presents slides and shares a screen in Teams, and says plainly what the platform will not let it do
Several minds in one room
A team chat where humans and several model seats work together, each seat answering only when addressed, with no loop an agent can start
Three councils I threw away
Four designs for several models deliberating toward one plan, and why only the one that behaves like a real group conversation survived real use
The same brain, typed or spoken
Why a spoken request and a typed request produce exactly one durable turn in the same conversation, and what it took to keep the two from drifting apart
From a conversation to a work order
Helix wraps my governance platform without modifying it: talk happens in the room, durable work goes through the platform's own commands, and results come back