Corey James- 8written runbooks, from startup to performance degradation
- 1 dayto roll back the registration lock that broke every client
- 0open inbound ports
One declared fleet
Auth and access plane holds the operations layer for my whole estate. As its only operator, I need the estate to start, fail and recover the same way every time, so the fleet is declared and versioned and startup order is a property of configuration.
A few dozen services run under one systemd stack target. PartOf cascades carry a dependency's restart to its dependants, drop-ins make data-store clients wait for the network, and timers run routine work such as container cleanup. Unit files and a readable inventory are versioned with the code, so the fleet is rebuilt from source.
AgentOS gets the same practice for its own releases. Deploys go through gated, logged windows, and its background workers run as systemd services and timers in the same fleet. Public surfaces reach all of it through outbound tunnels and a private mesh.
Recovery written down
I wrote the runbooks, so incident response does not depend on memory. Each has thresholds, a severity model and an escalation path, with scripts for checks, rollback and draining.
The hardening fix that locked everyone out is one incident handled this way. I keep adversarial audits of my own systems even when they overturn an earlier sign-off.
I reverse my own change when the evidence calls for it.
Failure chosen per component
A cache and a queue of pending database writes need to fail differently, so Redis instances are split by role and each has its own failure mode. Services expose metrics to Prometheus, which discovers new GPU nodes from files. Grafana holds the dashboards, and Alertmanager routes alerts by severity, with SMS paging for anything that needs a person.