Recovering runners — the rebuild plan; the registration tokens

Runners are the virtual machines or containers that execute workflows. A CI/CD disaster that takes down the runners leaves the control plane unable to dispatch jobs. The rebuild is the second step in the recovery, after the control plane exists and before the secret store and artifact registry are brought back online.

flowchart LR
    A["runner rebuild"] --> B["image built"]
    A --> C["token obtained"]
    A --> D["scope configured"]
    A --> E["labels applied"]
    B --> F["runner registered"]
    C --> F
    D --> F
    E --> F
    F --> G["runner joins pool"]
    G --> H["control plane dispatches jobs"]

The four artefacts a runner rebuild must produce are the image, the registration token, the runner scope, and the runner labels. A runner that lacks any one cannot join the pool.

The image

The runner image is the container or virtual machine image the runner executes inside. For self-hosted runners, the image is built from a Dockerfile in the team’s repository; the Dockerfile is the source of truth for the tools the runner has available.

The rebuild rebuilds the image from the Dockerfile. The image is pushed to a registry that survives the disaster (or to the rebuilt artifact registry from lesson 05):

docker build -t $REGISTRY/runner:latest .
docker push $REGISTRY/runner:latest

A runner image rebuilt from a Dockerfile is reproducible; a runner restored from a snapshot may contain drift the team has forgotten about.

The registration token

The registration token is the secret the runner uses to identify itself to the control plane during the ./config.sh step. The token is issued by the control plane, is short-lived (typically one hour), and is single-use.

The registration command:

./config.sh --url $REPO --token $TOKEN

$REPO is the repository, organisation, or enterprise URL the runner registers against. $TOKEN is the registration token obtained from the control plane’s runner settings immediately before the ./config.sh invocation. The token is not stored on disk after the registration; the runner receives a long-lived credential it stores locally.

The runner scope

The runner scope determines which workflows the runner can execute. The three scopes are:

• Repository. The runner is registered against a single repository. Only workflows in that repository schedule on the runner.
• Organisation. The runner is registered against the organisation. Workflows in any repository in the organisation can schedule on the runner.
• Enterprise. The runner is registered against the enterprise. Workflows in any organisation in the enterprise can schedule on the runner.

The wider the scope, the more workflows can run on the runner; the wider the scope, the more trust the runner must hold. A runner that holds production deploy credentials should be repo-scoped, not org-scoped.

The runner labels

The runner labels match the runs-on label in the workflow file. The labels are set during the ./config.sh invocation and are stored in the runner’s local configuration.

A workflow that requests a label the runners do not have queues forever; the workflow does not fail, it waits. The DR runbook includes a verification step that the runner labels match the workflow labels.

Production discipline

1. The image is rebuilt from a Dockerfile, not restored from a snapshot. The Dockerfile is the source of truth; the rebuild is reproducible.
2. The registration token is fetched at rebuild time, not stored in the repository. A token in the repository is a token an attacker can read.
3. The runner scope is the narrowest scope that meets the workflow’s needs. A production deploy runner is repo-scoped.

Cross-course references

• Git, CI/CD & GitOps — Part XCVII-02 (Recovering the Control Plane) covers the control-plane recovery the runners register against.
• Container Security for Production Sysadmins — Part IV (Image Hardening) covers the image-build discipline.
• Terraform for Production Sysadmins — Part XXIV (Runner Provisioning) covers the Terraform that provisions the runner VMs.

Quiz