Technology · Platform

Prometheus

Prometheus carries the numbers that decide whether GPUs earn their keep, follows a fleet that changes under it, and pages one operator's phone.

Inference FabricAuth and access planeToken accounting and alerting
The token-accounting panel
Tokens received12.3B
Tokens routed12.3B
Difference0
Read across the broker fix in the Inference Fabric.
  • ~137metrics the broker exports of its own
  • 31panels on the fabric's Grafana dashboard
  • 96.9%SM utilisation on the larger card in the 163,000-tokens-a-second run
  • 12scrape jobs across hosts, GPUs and inference nodes

Panels for the numbers that decide GPU economics

In Inference Fabric generic service health told me nothing about whether the GPUs were earning their keep, so I instrumented the quantities that decide GPU economics: padding, cache reuse and token conservation. The broker exports its own metrics into a Grafana dashboard, and GPU utilisation comes from NVIDIA's DCGM exporter beside the runtime's counters.

Padding waste
Shape switches
Key-value cache hits, per GPU
Tokens received minus routed
Items in flight
GPU utilisation · DCGM
Shapes illustrative. Orange: the panel that must read zero.

The token-accounting panel is how I proved the broker fix in Millions of queued items that were not there.

A panel that should read zero is a test; any non-zero difference between tokens received and routed is a defect.

Each step of From 44K to 163K tokens per second is a capture from these dashboards.

Monitoring that follows the fleet

In FAFO Core and Function Forge GPU nodes appear and disappear as agents provision them locally, on a LAN workstation or on rented cloud capacity, and I did not want anyone editing a scrape configuration.

The provisioning code writes the scrape list
Local GPU nodes, scraped
Added when an agent provisions a LAN workstation or rented cloud capacity
Dropped when drained
Through Prometheus's file-based service discovery.

The governed write path and the model-facing gateway export their own metrics, including latency percentiles, so queue depth and write latency sit beside the GPUs they feed.

Alerts that reach a phone

The Auth and access plane includes operations for the whole estate, and one operator cannot watch a screen, so alerts reach my phone. I built the stack end to end: a containerised Prometheus scraping hosts, GPUs and inference nodes; Grafana dashboards for the Redis queues, the hosts and the fabric; and alert rules for Redis and the hosts.

Alert rule firesRouted by severityEscalation tiersText message
Scripts test alert delivery, compute capacity and performance baselines.

A separate stack for AgentOS

AgentOS has its own Prometheus and Grafana pair, with a PostgreSQL exporter over the cost and runtime tables.

Fabric and estate
Broker, DCGM, hosts, RedisPrometheusGrafanaTo my phone
AgentOS
Cost and runtime tablesPostgreSQL exporterPrometheus and Grafana

Usage by work order, phase, role and model; cache reuse per agent seat.

The two systems fail for different reasons, so one incident never hides in the other's dashboards.

I build observability alongside the system and design it around the numbers that prove the system works, which is what lets me publish figures with their context. Prometheus, Grafana, Alertmanager and the exporters are third-party; the metrics, dashboards, discovery wiring and alerting are mine.