Technology · Platform

NVIDIA Nsight Systems

Nsight traces are where I found what the engine was actually doing, and where I learned to check that the trace was not itself the cause.

Inference FabricTimeline traces on NVIDIA Blackwell
One compute stream, traced · illustrative
Compute stream
warm-upsteady state · no allocation · no stream sync
kernelidle gap, found by gap analysis
  • 44Ktokens a second on one NVIDIA Blackwell GPU at the start, 7B dense model in FP8
  • 163Kfive-minute mean across two NVIDIA Blackwell GPUs at the end
  • <1%gained by custom CUDA graphs over eager execution
  • 6.8%slower with the key-value cache offloaded to the host

Proving the runtime's hot-path rules

Inference Fabric depended on a GPU runtime that had to stay saturated, and some of its properties could only be settled with a timeline. The runtime is a C++ layer at the NVIDIA TensorRT boundary, driven from Rust, feeding NVIDIA TensorRT-LLM for generation and raw TensorRT for hidden-state extraction.

I gave its code two rules: after warm-up the decode loop allocates no memory, and it never synchronises the compute stream. The rules live in code and stop regressions; a timeline trace shows the property holds.

Allocations per step, as a trace shows them
warm-upsteady state · no allocation, no stream sync
Heights are illustrative; the rule in code keeps the steady state flat.

"Zero allocations after warm-up" is easy to assert and easy to break in a later change, and a trace settles it.

Finding the idle time between kernels

A stray allocation or a hidden synchronisation on the hot path costs throughput on every step and disappears inside an average. I profiled the runtime on an NVIDIA Blackwell GPU with Nsight Systems and ran a gap analysis of idle time between kernels. I profile on the card the system targets, because launch overheads and idle gaps differ by architecture.

What an average hides
Timeline
Average

Dropping CUDA graphs and the other dead ends

Nsight also decided a design question. I captured the runtime twice on the same card, once in eager mode and once with custom CUDA graphs batching kernel launches. The graphs cost complexity and constrained shapes for too little, so I dropped them with the programme's other measured dead ends, and wrote the dead ends down with the wins so nobody tries them again. See From 44K to 163K tokens per second.

Throughput against eager execution
Eager executionkeptCustom CUDA graphsdroppedHost KV offloaddropped
INT4 weights, also measured, were slower than FP8 and were dropped too.

I use a system-wide profiler to make performance claims provable and to kill attractive ideas that do not pay. Nsight Systems is NVIDIA's tool; the runtime rules it checked, the experiments and the decisions taken from the traces are mine.