Technology · Discipline

GPU profiling and tracing

I instrument ActiveKV down to the runtime call level and I measure the instrument: a profiler that serialised the copy engine it was observing got its verdict retracted in the plan.

ActiveKV on Intel ArcStep profiler with token-life viewsLD_PRELOAD runtime shim
Copy and compute on one card
Timing events on
timestamp commands make the copy engine wait
Timing events off
copies hide under compute
host-to-device copycompute
  • 4capture levels, from runner phases to every runtime call
  • 0.21copy overlap at best with timing events on, down to 0.02
  • 1.09the same overlap at best with them off, down to 0.79

Capture levels chosen per run

ActiveKV is only faster if I can see where each decode step spends its time, on the host and on the device, and the act of looking changes the answer. I decided to build the instrument and then measure the instrument. I directed the build of a step-profiling suite with capture levels chosen per run.

1 · Lowest
runner phases
2 · Host
per-step region budget, layer windows
3 · Full
device kernels and copies
4 · Edge
every Unified Runtime call
One step, illustrative. Each level adds to the one above and costs more to capture; the edge level ties each runtime call to the device intervals and host waits around it.

Readers that follow every token

Readers sit on top of the captures. The token-life view follows every generated token from the scheduler through each device milestone to the client.

Token-life view · one generated token
schedulerdevice milestonesclient

Also read from the same captures: token tracer, per step and concurrency · runtime-call association · step-profile report · HTML browsing · two-run comparison

I keep association and causation apart, and I treat an idle-looking queue as a question about the hardware. Each chart prints its own uncertainty, including the unproven device-to-host clock conversion.

I removed automatic verdicts from the comparison output, because a profiler should collect and the engineer should judge.

The observer that changed the answer

Two problems shaped the suite. A trace showed host-to-device copies failing to overlap with compute, and I acted on it. A controlled experiment then showed that the timing-enabled events used to see the overlap were being lowered into timestamp commands that made the copy engine wait. I retracted my verdict in the plan. That is The observer that changed the answer.

Every claim about overlap or serialisation now gets an uninstrumented control before anyone acts on it. I know the GPU runtime below the framework well enough to catch my own instrument distorting a result, and I correct my conclusions in writing when that happens.

Development host · deliberately busy

compare a candidate with the last run under the same load; instrumented runs never count

Second host · quiet, capture off

where speed is proved

Illustrative background load.

A shim when the vendor tracer leaked

Later, edge-level captures grew the server process for its whole lifetime. Micro-loop experiments put the leak in the vendor's tracing path, so I had the runtime calls intercepted directly with an LD_PRELOAD shim and left the tracer alone.

Vendor tracing path

the server process grows for its whole lifetime; one capture at a time

LD_PRELOAD shim

one entry per call in flat memory, read by a Rust log scanner with a Python fallback; several captures at once

Shapes, not measurements.

The suite is built for ActiveKV on Intel hardware; the vendor profilers and runtime are Intel's and PyTorch's, used and intercepted, and unmodified.