Corey James- 2plausible gather conventions, and the engine accepts either
- 0.1%checksum change that proves the engine reads the input
- 25,800tokens per second over 10.8 million tokens of extraction
The extraction mode of my inference runtime reads a model's internal representation, the input to its language-model head, for every probe, in one forward pass with no generation, at very large batch. The engine's packed-sequence interface takes an input that tells the attention plugin which token's state to gather for each sequence. Two conventions are plausible: the cumulative length, or the cumulative length minus one. The documentation did not settle which this engine expected. A wrong guess still produces well-formed vectors, just the wrong token's, or the next sequence's first token, for every probe. The result would be a silently corrupt dataset of tens of millions of rows. So the backend proves the convention before it extracts anything.
The extraction path
The engine's build configuration is parsed at load, and a disabled key-value cache selects the raw TensorRT backend: no executor, no cache, a direct enqueue on a deserialised engine. Every tensor binding is classified by role, packed per-token, per-batch, static with dynamic dimensions or output, and by location. Device and pinned host buffers are sized for the maximum shape and allocated once; a failure frees everything. A warm-up pass with one token populates the buffers, because zero-initialised context lengths and token ids crash the attention plugin. Per-batch metadata reuses capacity reserved at warm-up, so there are no per-batch allocations.
The detector
At warm-up the backend builds a two-sequence micro-batch with lengths one and two, three packed tokens, and runs it twice: once with exclusive-end ids and once with inclusive ids. For each run it decodes each sequence's output vector from half-precision bits on the host, handling denormals, infinities and not-a-number explicitly, and forms a cheap checksum, the sum of absolute values.
The chosen mode is frozen into the engine, a single function becomes the only producer of these ids, and an unknown mode at batch time is a hard error.
The checksum is a fingerprint rather than a full comparison, and the both-consistent tie-break is a heuristic rather than a proof.
The test runs once per engine load, and it is the difference between a dataset and a disaster.
Downstream
What this demonstrates
The habit of asking what a wrong answer would look like before producing a hundred million of them, and building the check into the load path where it cannot be skipped. It is the same instinct as The observer that changed the answer, applied to a data pipeline instead of a profiler.