Engineering story · Inference Fabric

The thirteen percent that was never missing

A profile said thirteen percent of the GPU was idle; it was the share of time in non-matrix kernels. Reading what each metric measured before optimising found utilisation at 98.7 percent

Nsight Systems traceInference FabricOne consumer Blackwell GPU
Kernel time, as the first pass read it
86.7% "efficiency": matrix multiplicationsthe "gap"
13.3%that was never missing
  • 98.7%streaming-multiprocessor utilisation in the trace
  • 332,965kernel launches in sixty seconds
  • 4,236memory allocations, every one in the first second

An early profiling pass on the inference runtime reported a GPU compute efficiency figure and a gap, and the obvious next step was a programme of work to close the gap. Before starting it, I went back to the profile to find out what the number measured. It was the share of kernel time spent in matrix multiplications. The rest was attention, activation and normalisation kernels, all of them necessary, none of them idle.

What the profile said

A sixty-second trace on one consumer Blackwell GPU gave the whole picture.

Share of kernel time
Matrix multiply, two kernel families86.8%Attention3.8%Activation2.8%Normalisation2.5%Everything else4.1%
Kernel activity in the window

59.73 seconds of kernel time in a 61.06-second window: 97.8 percent.

Light blue: the necessary non-matrix work the first pass had counted as a gap.
Memory allocations over the trace
first secondthe sixty seconds after: none

Zero allocation after warm-up, as it looks in a profile.

Copy time · 9,788 operations

Host-to-device transfers: 96.3 percent of copy time.

The experiment it saved and the one it ran

The gap-closing programme would have chased a number that could not move, because the only way to reduce the non-matrix share is to remove the model's own layers. Instead the profile was used to test a different hypothesis. Fixed-size epochs and a dense slot map had been built so the runtime could capture its decode steps as CUDA graphs, on the theory that launch overhead was a lever. Profiling settled it, because the vendor's executor already manages graphs internally.

Kernel time · billions of nanoseconds
Eager execution30.81Graph capture30.77
A tenth of a percent: launch overhead was not a lever.

The epoch machinery stays for slot integrity, and graph capture was dropped as a lever.

The written rule

The runtime's performance notes now open with the rule this produced.

Define the metric
Compare like with like
Write the cache configuration beside it
Use the figure

The same rule separated the cached and uncached tier sweeps into two tables that are never shown as one curve, and overturned a bandwidth-bound conclusion that had rested on an estimate for one card and a measurement for the other.

What this demonstrates

The habit that matters most in performance work: reading the definition of a metric before acting on its value.

A tenth of a percent and a thirteen percent that was not there are both results, and both are written down.

The decision to re-read the profile and the measurement rules were mine; AI coding agents, a mix of frontier and open-source models, ran the profiling under my governance system.