Infrastructure and operations

The Pipeline Foreman

For a data team lead who needs a new processing job by Thursday.

One signal · 10 specialists · one result
Data Job BuilderService Function BuilderJob DispatcherWorker Fleet ScalerSelf-Starting Job SchedulerBatch Job ControllerInternal Automation BuilderDatabase Traffic ControllerLaunch Pre-flight CheckerSmoke Test Runner
  • Does the work1
  • Runs infrastructure8
  • Audits and guards1
How the work moves

Four steps, each done by one specialist

Starts with

A new processing job needed by Thursday

First
1Generates the skeleton, checks it builds and waits for approvalData Job Builder2Runs a harmless test batch through the new jobSmoke Test Runner
Then
3Starts, follows and recovers the job once in serviceJob Dispatcher4Adds or removes workers as the backlog movesWorker Fleet Scaler
Ends with

A working processing job that is started, watched and scaled

the signalthe resultEach step is done by one specialist on the crew

The story

The agent generates the skeleton of the job so that only the business logic is left to write. It checks that the job builds, waits for a person to approve the restart and runs a harmless test through it. Once the job is in service the agent starts it, follows it and recovers it, and adds or removes workers as the backlog moves.

The crew

The specialists behind this role, grouped by the kind of work they do.

Runs infrastructure
  • Data Job BuilderGenerates the full skeleton of a new processing job so the only thing left to write is the business logic, checks that it builds, waits for a person to approve the restart, runs a harmless test job through it, and opens a merge request.Also serves The Automation Builder
  • Database Traffic ControllerKeeps heavy bulk writes from flattening the database: it prices each write by its expected load, lets cheap work through, holds heavy work until there is room, and tunes the write workers.Also serves The Database Guard
  • Self-Starting Job SchedulerThis agent turns a job into a standing service that starts itself when work piles up or on a schedule, and lets you list or stop those services at any time.Also serves The Automation Builder
  • Service Function BuilderPlans a new data-processing function, prepares its database work queue, generates its files, registers it as a tool and runs a harmless smoke test through it.Also serves The Automation Builder
  • Job DispatcherThis agent starts, follows and recovers processing jobs across every automated job the team owns, and reports their status on request.Also serves The Automation Builder
  • Worker Fleet ScalerThis agent adds or removes background workers as the backlog grows and shrinks, restarts stuck ones and clears out dead connections.Also serves The Database Guard
  • Batch Job ControllerRuns a large job on a local card or a rented one, pauses, scales or stops it mid-flight, turns a one-off job into an always-on service, and isolates a job that misbehaves.Also serves The Database Guard
  • Smoke Test RunnerSends a small harmless batch through the service, once or on a trigger, and confirms it came out the other end.Also serves The Safe Change Desk
Does the work
Audits and guards

Where this crew also works

Each bar counts the specialists this role shares with another.

The engineering behind this role

A human at the privileged stepIdeaTools that build tools, under governanceEngineering storyA write path that sizes itselfEngineering storyAll capabilitiesThe engineering, by what it does