Technology · Runtime

CUDA Driver API

My CUDA analytics engine talks to the driver API directly from Rust, built for a GPU architecture the Rust ecosystem had not caught up with.

Inference FabricContexts, modules, host memoryCUDA 13 on Blackwell
Before any lease or kernel touches the device
Blackwell GPUs · SM 120 cubins, loaded eagerly
no PTX embedded; nothing compiles behind my back
compute capability 12.0
driver at least 580, the CUDA 13 minimum
anything else refused
bindgen over the CUDA 13 header at build time; raw FFI from Rust
  • 2Blackwell GPUs the engine loads its own cubins onto
  • 3memory classes a lease can hold
  • 13.1the CUDA version the build defaults to

Raw FFI for an architecture nobody had wrapped

In the Inference Fabric I was targeting CUDA 13 and NVIDIA Blackwell when no Rust CUDA crate supported either, and I needed to ship compiled kernels to remote workers and control streams, events and devices directly.

I chose raw CUDA Driver API FFI from Rust, because no wrapper supported the toolkit and the architecture, and because cubin loading had to stay under my control. The Driver API is the same across the Blackwell generation.

Compile targetchanges
Hardware checkchanges
Driver API layercontexts, modules, streams, host registration: unchanged

I work at the level of contexts, modules, streams and host memory registration when the higher-level tools do not yet reach the hardware I am using.

One primary context for a moving thread pool

The analytics service runs on Tokio, and CUDA calls happen on blocking-pool threads that change from one call to the next.

CUDA calls on Tokio's blocking pool · illustrative
Thread 1
Thread 2
Thread 3
Every call, on whichever thread is free, uses the one primary context retained per process. Non-primary contexts crashed with segmentation faults in this setting.

Pinning host memory for leases

The Vector Shard Manager's lease layer calls libcuda directly to pin host memory with cuMemHostRegister. A type-only lease crate is the boundary between storage and CUDA, so storage code reaches the GPU only through it; the storage daemon never links CUDA, and a build-time check enforces that boundary.

mapped shard
page cache: the kernel will not pin it
anonymous buffer
staged copy
pinned host
registered with the driver
device
resident
Mapped data reaches pinned memory only through the staging copy.

A doorbell the GPU waits on

Where latency matters, the GPU waits and the host does no polling.

Host and GPU stream · illustrative
Host advances
GPU stream
cuStreamWaitValue64 holds the stream on a sequence value in mapped memory; each tick releases it, with no CPU involvement once the wait is queued.