Technology · Framework

pybind11

pybind11 binds ActiveKV's C++ engine into vLLM and PyTorch; a line-by-line audit proves every remaining Python line is a framework contract.

ActiveKVThe C++ to Python boundaryInside vLLM and PyTorch
A thin Python surface over a native engine
Python · vLLM and PyTorch contracts
pybind11 bindings
C++ engine on ATen
context geometry · key-value state · admission policy · host-device transfers · tiled attention · telemetry
proportions illustrative
  • 67pybind11 modules expose the C++ engine to the parts of vLLM that call it
  • 866of 867 places where Python dispatched work became direct native calls
  • 0lines of my own Python in the release payload

From a Python prototype to C++ inside vLLM

In ActiveKV the Python version proved the design and was too slow on the per-token path, and it left the product logic as readable source. I moved the whole runtime to C++ inside vLLM, with pybind11 as the boundary, built against PyTorch's ATen.

Exact-mode parity cells
Compressed-mode parity cells
Every cell matched bit for bit. The Python reference stays in the tree, unwired, for exactly this comparison; cell counts illustrative.

The port is told in Rewriting an engine extension without changing a token, and how I proved the native code was the code that ran is in Proving a rewrite is the same system.

What may stay in Python

The hard part of a binding layer is deciding what may stay in Python. My rule: only an explicitly justified vLLM or PyTorch interface requirement remains, and a framework that requires a Python callable gets a thin binding with the logic behind it in C++.

The language boundary is a contract I minimise and justify line by line.

  • Framework contract, with the framework source line citedstays
  • Convenienceported
  • Delegationported
  • An async protocol method, defended as Python-onlyported
The CPython C API exposes the async protocol directly; a spike proved it, and that module moved to C++ too.
After the port
Direct native callsStill Python

The engine owns the process

I also inverted the usual arrangement: a native launcher embeds the Python interpreter that vLLM needs. A type-conversion defect at the boundary that broke startup was fixed in the binding code before the release cut.

Usual
Python process
native extension
ActiveKV
Native launcher · the engine
embedded Python interpreter for vLLM

A release with the boundary enforced

The release payload comes as compiled binaries plus a patch to the user's vLLM, and every change is screened at intake.

A native change
throughA new Python module
rejectedA Python import from C++
rejectedA per-step Python callback
rejected
intake screen

I can carry a large Python-to-C++ port inside someone else's engine to bit-exact parity, and I reason about a language boundary from what the interpreter and framework actually require. pybind11, vLLM and PyTorch are third-party; the engine, bindings, launcher and boundary rules are mine.