Corey JamesThe story
The agent serves the candidate next to the current model and puts both through the same hidden test set. It load-tests the candidate, checks that answers have stayed the same where they should and reports where each version fails. A candidate that clears the bar rolls out in stages.
The crew
The specialists behind this role, grouped by the kind of work they do.
Runs infrastructure
- Model Comparison EvaluatorThis agent puts two or three model versions through the same hidden test set, runs any generated code safely, scores the answers strictly and explains where each version fails.Also serves The Model Broker
- Model Serving EngineerBuilds a version of the model sized for each GPU card you own, measures how fast each card runs it, sets each card's limits for the scheduler, and copies the right build to every machine.Also serves The Engine Builder
- Capacity PlannerReads how fast each card runs, how full the queues and the database are, what is running now and what spot GPUs cost, then proposes where the next workload should go and whether renting is worth it.Also serves The Bulk Run Manager
- AI Speed InvestigatorThis agent finds where the time goes in each AI response, from the full request down to waiting, copying and the slowest steps, and compares serving modes side by side.Also serves The Speed Detective
- Serving Capacity TunerSets how much GPU memory holds conversations in progress and when new requests wait, then measures how many run at once.Also serves The Speed Detective
- Answer Consistency SetterSets how predictable or varied a model's answers are, from fully repeatable to more varied, for each deployment.
- Model Rollout ManagerMoves a new model version into service: builds it, sends a harmless test job through, benchmarks it against the current version, brings up new workers, drains the old ones, and puts the old version back if the new one is worse.Also serves The Model Shrinker
Challenges decisions
- Release Readiness ReviewerThis agent pulls together the regression results, accuracy checks, load tests and package review for a new version and writes one go or no-go summary.Also serves The Model Shrinker
Keeps knowledge
- AI Model LibrarianBrings a requested AI model in from its exact source, checks access and the licence question with the owner first, installs it, checks the files are intact and keeps the catalogue of every model and where it came from.Also serves The Studio Engineer
Audits and guards
- AI Service Load TesterThis agent runs the model under realistic traffic against a standard setup, checks each memory-saving path and compares runs.Also serves The Speed Detective
- AI Answer Accuracy CheckerThis agent checks whether a change alters the model's answers, finds where any difference starts and gives a clear yes or no.Also serves The Model Shrinker, The Speed Detective
Where this crew also works
Routes
Routes this role works in. Its place in each one is filled in.