Corey James- ~137metrics exported by the broker
- 31panels on the broker's dashboard
- ~203,000hidden-state extractions in a single test run
Dashboards built around the measures
Inference Fabric made claims about GPU efficiency that I wanted to prove from a screen, so I built its dashboards around the measures that justify the design. The panels show tokens per second split into prefill and decode, padding waste, shape switches, key-value cache hit rate, the broker's lanes and cold tier, probes and extractions per second, and a token-accounting difference that should always read zero. I chart domain measures first, because padding waste and token-accounting drift say more about a GPU pipeline than host load does, and I built each fabric dashboard alongside the feature it measured, so a change was visible the day it landed.
Every fabric figure on this site comes from these panels. See From 44K to 163K tokens per second and Millions of queued items that were not there.
A 7-billion-parameter dense model in FP8 on two NVIDIA Blackwell GPUs; 12.3 billion tokens each way.
I decide what to measure as part of the design, and I build the instruments that let me defend each number.
Further dashboards cover the GPU runtime, the vector store's storage health, Redis queue health with a capacity projection, and a system overview, and DCGM and node exporters cover the GPUs and hosts.
Grafana and Prometheus are third-party; my work is the choice of measures, the discovery wiring and the alert routing.
Alerts that reach one operator
Auth and access plane is where one operator has to learn about a failure before anyone else does. The same Prometheus and Grafana stack carries the estate's alerting. A separate Prometheus and Grafana pair serves the agent runtime.
Spend and cache reuse per seat
AgentOS stores the cost of every agent session in PostgreSQL, rolled up by work order, task, role, model and phase. I wanted to choose models per role from measured spend, so Grafana dashboards read those rollups and show cost and cache reuse per seat for a work order.