Corey James- 12scrape jobs across hosts, GPUs and inference nodes
- 0tokens between received and routed after the accounting fix
Discovery wired to provisioning
Auth and access plane holds the observability layer for an estate one person operates, and that shaped every choice: nothing may depend on someone watching a dashboard. My Rust services expose native metrics endpoints, and a containerised Prometheus scrapes them, covering hosts, GPUs and inference nodes.
I wired discovery to provisioning. When the execution substrate stands up a GPU node, locally, on a LAN workstation or on rented cloud capacity, it writes the node into Prometheus's file-based service discovery and installs the host and GPU exporters on it, so a new node is monitored from its first minute with no one editing a scrape configuration.
Alerts on work that stops
Alertmanager routes alerts by severity through escalation tiers, and a small webhook turns them into text messages on my phone, because a single operator cannot watch screens. The rules watch for what stops work, across Redis and the queues on top of it.
Alert delivery is tested too: a helper script fires a synthetic alert end to end. Others compute capacity headroom and capture a performance baseline, so a slow change shows up against a known reference. Grafana dashboards cover the Redis queues, the system overview and the GPU pipeline.
Dashboards for GPU economics
Inference Fabric is where the instrumentation mattered most, because every number I publish about the fabric comes from a dashboard capture. I built its dashboards feature by feature as the fabric grew, each feature arriving with its panel, around the quantities that decide GPU economics. The received-against-routed panel is how the broker's accounting fix showed it held. See Millions of queued items that were not there and From 44K to 163K tokens per second.
The instrumentation found the fabric's lost-work bug, it backs every number I publish, and it lets one person run a multi-service estate.
I instrument systems as I build them and treat the instrumentation as part of the product.
Cost on the seat that spent it
AgentOS has its own Prometheus and Grafana pair with a PostgreSQL exporter. Its dashboards show cost and usage per role and per model from read models over the governance database, so the cost of an AI engineering programme lands on the seat and model that spent it.
Prometheus, Alertmanager, Grafana and the exporters are standard open-source tools; the discovery wiring, dashboards, alert rules and paging path are mine.