Runs infrastructure · Machines, data and operations

Failed Machine Recovery Specialist

Notices when a GPU worker gets sick or goes silent, then moves its work: to a free local card first, then to the same provider at today's price, then to another provider, and otherwise into a waiting line.

Asked byThe Bulk Run Manager
The Night Shift
Serves
  • The Bulk Run Manager
  • The Night Shift

What it does

Notices when a GPU worker gets sick or goes silent, then moves its work: to a free local card first, then to the same provider at today's price, then to another provider, and otherwise into a waiting line. For jobs you flag, it pauses and asks you first.

For an on-call operations engineer.

Roles it serves

The Bulk Run ManagerFor a team with millions of prompts to process.The Night ShiftFor a one-person operations team.

Works alongside

The specialists it shares the most crews with.

Agent Fleet OperatorSupervises other agentsAgent Replacement SpecialistSupervises other agentsAlert TunerRuns infrastructureBulk Inference Run ManagerRuns infrastructureCapacity PlannerRuns infrastructureCloud GPU BuyerRuns infrastructure

Routes it travels

An alert becomes a customer notice
  1. 1The Night Shift
  2. 2The Reputation Guard
  3. 3The Record Keeper
  4. 4The Stand-up Host
An outage becomes a lesson
  1. 1The Night Shift
  2. 2The Incident Commander
  3. 3The Record Keeper
  4. 4The Handbook Keeper