The story

An alert fires at 3 a.m. The agent reads it, follows the runbook, restarts what needs restarting and moves work off a machine that has failed. Before any step that could lose data it texts a person and waits. The morning report says what happened and what was done.

The crew

The specialists behind this role, grouped by the kind of work they do.

Runs infrastructure
  • Night-Shift Fleet WatchWatches machines, GPUs and services overnight, texts your phone only for what matters, runs the standard health checks before waking you, and has a summary ready in the morning.Also serves The Watch Office
  • Incident ResponderReads the alert and the service logs, matches them to the written procedure, restarts the failing worker or service, rolls back a bad deployment if needed, and files the incident on the issue tracker.Also serves The Incident Commander
  • Night-Shift On-Call ResponderWhen an alert fires, this agent works through the matching runbook, takes the safe fixes itself and texts the on-call person, and it waits for a human before any step that could lose data.
  • Write Queue FirefighterTakes the page when the database write queue backs up, follows the recovery procedure, drains the queue if it has to, and confirms that writes are flowing again.Also serves The Database Guard
  • Alert TunerThis agent keeps alert rules and text-message delivery working by testing each rule and its route from start to finish.Also serves The Watch Office
  • Service CaretakerThis agent watches the status, memory and logs of every core service and restarts only the services it is allowed to restart.Also serves The Housekeeper
  • Failed Machine Recovery SpecialistNotices when a GPU worker gets sick or goes silent, then moves its work: to a free local card first, then to the same provider at today's price, then to another provider, and otherwise into a waiting line.Also serves The Bulk Run Manager
Speaks to people
Supervises other agents

Where this crew also works

Each bar counts the specialists this role shares with another.

Routes

Routes this role works in. Its place in each one is filled in.

The engineering behind this role