Technology · Language

C++

ActiveKV's runtime is C++ throughout, ported from Python under bitwise-parity gates, and the Inference Fabric's TensorRT boundary is C++.

ActiveKV · Inference FabricCMake, pybind11 and ATen
ActiveKV's C++ engine · one decode step, layer by layer
Layer nstaged in, attended, written back
Layer n+1staged the same way, behind its own fences
prepare, bind, completehost-to-device transferfence; an unknown outcome is quarantined
proportions illustrative
  • 77,600lines of non-test C++ in ActiveKV's engine, across 361 source files
  • 568test files beside them
  • 17,900lines in the per-step layer that prepares, binds, transfers and completes each step
  • 9,500lines of C++ at the Inference Fabric's TensorRT boundary

Porting ActiveKV's runtime out of Python

ActiveKV began as Python inside vLLM. That was the right way to find out whether keeping attention history in host RAM worked, and the wrong way to ship it: the per-token path was slow, and the product logic sat in readable source. I decided to move the whole runtime to C++ with bit-identical output as the gate.

Every optimisation waited until parity held, because a port that changes behaviour and speed at once cannot be trusted for either.

The result is a C++ execution engine that loads as the vLLM plugin entry point and binds to Python through pybind11 and PyTorch's ATen. The hard problems are lifecycle problems under asynchronous DMA: staging regions are fence-protected leases with generation counters, and a fence whose outcome is unknown is quarantined and never reused.

The release payload holds no Python of mine. It is told in Rewriting an engine extension without changing a token, and its parity method in Proving a rewrite is the same system.

Python dispatch sites
Before the port867After1
Parity cells, bit-identical

13 of 13 exact

12 of 12 compressed

Reasoning from the interpreter's contract

Every line of Python that stays has to cite the framework source line that requires it.

I also check that native code is on the served path, because a port can pass every test and still fall back to Python where it matters.

"Impossible outside Python's async syntax"
The CPython C API exposes the asynchronous protocola spike proved it; the module was ported
123 translation units, scanned

one missing pybind11 type caster, only on the device route

Rules the vendor engine does not impose

Inference Fabric needed a GPU runtime that stays saturated. The runtime is Rust with a C++ layer at the NVIDIA TensorRT boundary, built with CMake: the engine loader, the decode step, a pinned-memory allocator and a stream manager. The C++ enforces three rules the vendor engine does not: no device allocation after warm-up, which an Nsight profile shows; compute-stream synchronisation refused at run time; and no CUDA-graph capture on the runtime's own streams.

Device allocations, from start to steady state
steady state: none
~4,200 in the first second
Warm-up owns every allocation, so steady state never waits on the allocator.

The hidden-state backend on raw TensorRT measures an under-documented plugin tensor convention at warm-up by running one micro-batch two ways, and an ambiguous answer stops the worker. See From 44K to 163K tokens per second.

I use C++ where control of memory, lifecycle and the language boundary decides whether a system can be trusted. The fabric runtime uses NVIDIA's kernels with no custom model kernels, and ActiveKV's kernel work is separate, as SYCL patches to Intel's open stacks.