Engineering story · ActiveKV

The observer that changed the answer

Discovering that my own profiler caused the serialisation it reported, and retracting a verdict I had already acted on

3 min readActiveKVPyTorch XPULevel Zero
The plan, as it read
Copies serialise behind compute; defer and batch them differently Retractedthe timing events were the serialisation
Seen through timing events Seen without them
computehost-to-device copies
  • ≤ 0.21copy and compute overlap with timing-enabled events on, from 0.02
  • ≥ 0.79overlap with no events, or with release-build events only, up to 1.09
  • 3variants interleaved in one process to test the instrument

A trace from my instrumented runtime showed that host-to-device copies were not overlapping with compute on the Intel card. I believed it, and I made a design decision on it. Then a controlled experiment showed that the timing events I had enabled to see the overlap were themselves the reason there was none. I retracted the verdict in the plan, and every overlap claim since gets an uninstrumented variant before anyone acts on it.

The challenge

Whether copies and compute overlap on a GPU decides the whole economics of a memory layer that stages data across the bus every step. You cannot see overlap without instrumentation, and instrumentation touches the thing it measures. The trace said the copy engine was serialising behind compute, and a design response, deferring and batching copies differently, was already in motion.

Constraints

Instrumentation is unavoidable: there is no way to see the timeline without events on it. And a decision had already been made, so the cost of being wrong was work in flight on a false premise.

My approach

Run the same workload in one process, interleaving three variants. If the overlap differs across variants, the profiler is part of the result.

one process, runs interleavedsame card, same driver, same host load for all three
no events
release-build events
timing-enabled events: the only one where the overlap disappeared
Run order illustrative.

Measure the measurement.

What I built

A confound experiment showing that timing-enabled events were lowered into timestamp commands on the compute engine, and that those commands forced the driver-offloaded copy engine to wait. With them off, the copies overlapped.

Compute engine queue, timing events on
Copy engine, events off

Copies run beside the kernels, offloaded by the driver to their own engine.

A retraction in the plan, naming the earlier verdict, the evidence that overturned it, and the work that had been started on it.

A standing rule that every claim about overlap or serialisation gets an uninstrumented control before it is acted on, and that instrumented runs are never quoted as product speed.

Difficult decisions

Retracting publicly. The verdict was mine and acted on. The cheap path was to quietly correct course. I wrote the retraction into the plan so anyone reading the plan later would see the wrong turn and why it happened.

Results

On one Intel Arc Pro GPU, the design response built on the false reading was withdrawn once the variants were in.

Technologies used

PyTorch XPU events · Level Zero runtime · Intel GPU copy engines · Controlled instrumentation experiments · Intel Arc GPUs

What this demonstrates

Knowledge of the GPU runtime below the framework, and the habit of checking that the instrument is not the result. When I am wrong in writing I correct it in writing.