Technology · Discipline

LD_PRELOAD symbol interposition

Symbol interposition let me intercept runtime calls directly when the vendor's tracing path was itself distorting measurements.

Generated wrappers · Rust parserActiveKV step profilingIntel Unified Runtime
Cross-section of one call
ActiveKV servercalls the runtime as usual
Observer shim · LD_PRELOAD
interposed by the dynamic linker at load
Unified Runtime and driveruntouched
  • 192Unified-Runtime entry points wrapped, generated from the runtime's own interface
  • 1instrumented capture at a time through the vendor path; concurrent through the shim
  • 0per-call state kept by the shim: forward, write one flat entry, return

Finding the leak before replacing anything

ActiveKV: the step-profiling suite follows each generated token from the scheduler through every device kernel and copy to the client. Its most detailed level associates Intel Unified-Runtime calls with device intervals and host waits. That capture first went through the vendor's tracing layer, and every capture grew the server process for its whole lifetime.

I wanted to know where the memory went before replacing anything. I ran controlled micro-loop experiments that set my collector against a subscriber with no callback attached. The subscriber with no callback still kept memory on every driver call.

Server memory across one capture · shape, not scale
Vendor tracing, my collectorgrows for the process lifetime
Vendor tracing, no callbackgrows the same
Observer shimbounded

That put the leak in the vendor's tracing path and cleared my collector, so the shim that followed fixed a measured fault rather than a suspicion.

An observer shim generated from the runtime

So I replaced that path with an observer shim loaded through LD_PRELOAD. The shim interposes on the Unified-Runtime entry points and forwards each call to the real implementation. It is generated: a script emits a wrapper for every entry point from the runtime's own interface.

The runtime's interface · one cell per entry point
Generated wrappers · the same cells

No allocation outlives a call, and a runtime upgrade is one regeneration.

The shim observes and forwards; it leaves the driver and Intel's runtime as they are. When a vendor tool distorts what I am measuring, I can go under it at the dynamic-linker level and replace it with something that does only the job I need.

From tens of gigabytes to host-device associations

A Rust log parser, with a Python fallback, turns the flat logs into associations between host calls and device work. It parses in parallel chunks with trace caches, because captures run to tens of gigabytes.

One capture, associated
Host calls
Uncertainty
Device
Host and device share no identifiers, so every association is by time and every figure prints its clock-conversion uncertainty.

Instrumented runs are for diagnosis only; speed figures come from uninstrumented runs on a quiet host.

The instrument must not become the result.

That rule is the subject of The observer that changed the answer, and the tracing stack beneath it is on the Intel oneAPI page.