The ephemeral runner fleet — the execution surface

The runner fleet is the execution surface the Actions control plane and every other CI tool depend on. It is the layer this course has called “ephemeral” from Part C onwards, and this lesson names the four primitives a production runner fleet depends on. Treating runners as cattle - one image, one OCI artefact, one autoscaler - is what makes the build plane reproducible.

The four primitives

A production runner fleet depends on four primitives. Forgetting any one of them turns the build plane into a place where scripts can run but no one can reproduce the machine they ran on.

flowchart LR
    Q["Actions queue depth"] -->|"metric"| KEDA["Autoscaler"]
    KEDA -->|"scale"| ARC["Runner controller"]
    ARC -->|"create pod"| POD["Runner pod"]
    POD -->|"pull by digest"| IMG["Runner image\n(signed OCI)"]
    POD -->|"execute"| WF["Workflow step"]
    POD -->|"discard"| END["Terminated"]
    SEC["Scoped secrets\n(OIDC)"] --> POD

• Ephemeral lifetime. A runner exists for one job and is discarded when the job ends. A long-lived runner accumulates state between jobs - cached dependencies, modified config files, leftover credentials - and that state becomes part of the audit trail whether the team intended it or not.
• Immutable image. The runner VM or pod boots from a pinned OCI image that is signed by the platform team’s CI. A change to the runner’s contents is a change to the image, which is a change in a Git repo, which is an auditable change.
• Autoscaler. A controller (ARC, KEDA, or equivalent) that scales the runner population up with queue depth and down to zero when the queue empties. The mechanism that makes “we have N runners” mean “we have at most N runners running concurrently, but the price is paid only while the queue is non-empty”.
• Scoped secrets. The runner receives only the secrets the current job needs, ideally via OIDC, and the secrets are revoked when the job ends. A runner with broad secrets is a runner whose breach exposes every secret at once.

The control plane is separate from the fleet

GitHub operates the Actions control plane: workflow routing, the OIDC issuer, the queue. The customer operates the runner fleet: the machines, the images, the autoscaler. The split, again, is deliberate.

• GitHub-hosted runners are GitHub-operated fleet: the customer’s trust is in GitHub’s image and isolation. Convenient for small teams, but limited in customisation and locked to GitHub’s pricing.
• Self-hosted ephemeral runners are customer-operated fleet: the customer owns the image, the autoscaler, and the network egress. Standard for any team whose builds must reach private cloud resources.

The split has three consequences:

• Egress is the customer fleet’s problem. A self-hosted runner in a private subnet has different network reach than a GitHub-hosted runner in a public cloud. The build’s ability to reach a private registry or a private kube API server is a fleet decision, not a control-plane decision.
• Image pinning is the customer’s job. GitHub-hosted runners are pinned by GitHub; self-hosted runners are pinned by the customer’s own image, which means the customer is also the signer of that image.
• Autoscaling is the customer’s job. GitHub-hosted runners scale automatically; self-hosted runners scale by the customer’s autoscaler, which is itself software in a repository and therefore subject to audit.

Building the image

The runner image is an OCI artefact built by the platform team’s CI and signed. A production-grade image contains:

• The runner agent.
• The toolchains the workflows call (Python, Go, Node, Terraform, kubectl).
• The root certificates and CA bundles the runner needs to verify the cloud and the registry it reaches.
• A non-root user the workflow steps run as; the agent itself runs as root only to orchestrate.

echo "Inspect the tags a runner image has been published with:"
crane manifest "${RUNNER_IMAGE}:${TAG}" 2>/dev/null \
  | jq -r '.tags[]' | sort

The output above lists the human tags the image carries. A production runner fleet pulls by digest, never by tag; the human tag is for engineers to reason about, the digest is for the autoscaler to actually fetch.

Operating the fleet

A production runner fleet is operated as a product with three recurring concerns:

• Image freshness. Toolchain versions drift; a runner image that is six months old has a six-month-old Go and a six-month-old kubectl. The fleet operates a rolling rebuild cadence and a “runner canary” workflow that exercises the latest image against the canary cluster.
• Capacity planning. The auto-scaler is configured against a target queue depth, and the team knows what the steady-state spend looks like at that target. A fleet that scales without a target is a fleet whose bill is whatever the queue happens to be.
• Isolation between jobs. A pod-per-job model is the default; a VM-per-job model is the alternative for workloads that cannot share a kernel. The choice is a fleet decision; the GitHub UI offers both, but only the pod model is the path the lesson series assumes.

Production discipline

1. Runner image is an OCI artefact, signed and pinned by digest. Treat it like any other artefact in the registry.
2. OIDC for cloud credentials in the runner; long-lived secrets are a fleet violation.
3. Autoscaler scales to zero when the queue empties. Standing runners are standing cost.
4. Pod-per-job (or VM-per-job) is the default; long-lived runners are an explicit exception with a named justification.

Cross-course references

• This course, Part C (Runner Pool Sizing) - the capacity-planning primitive this lesson scales.
• This course, Part IV (ARC) - the autoscaler this lesson assumes.
• Containers for Production Sysadmins - Parts VII-X cover the image-building primitive the lesson depends on.

Quiz