Ephemeral runners — clean state every job, and the cost of throwing away state

An ephemeral runner is a runner that exists for the duration of one job. When the job ends, the runner is destroyed. The next job starts on a freshly-provisioned runner with no memory of the previous job’s state.

The pattern is the self-hosted analogue of GitHub-hosted runners’ isolation: every job gets a clean host. The difference is who builds and destroys the host.

Persistent versus ephemeral

flowchart TB
    subgraph P["Persistent runner"]
        P1["Job 1"] --> H1["Host\n(state accumulates)"]
        H1 --> P2["Job 2\n(inherits state)"]
        P2 --> P3["Job 3\n(inherits state)"]
    end
    subgraph E["Ephemeral runner"]
        E1["Job 1"] --> H2["Fresh host A\n(destroyed at end)"]
        E2["Job 2"] --> H3["Fresh host B\n(destroyed at end)"]
        E3["Job 3"] --> H4["Fresh host C\n(destroyed at end)"]
    end

On a persistent runner, three jobs share one host. The host’s filesystem, environment, and installed tools are shared. On an ephemeral runner, each job has its own host; nothing survives across jobs.

What the safety buys

Ephemeral runners close the persistent-state risks that Part XL-02 identified:

• Secret leak across jobs. A GITHUB_TOKEN written to /tmp by job 1 is destroyed when job 1 ends; job 2 has a clean filesystem.
• Backdoor survival. A malicious step that writes a crontab entry in job 3 is destroyed when job 3 ends; job 4 starts clean.
• Environment drift. Each job runs the same image; the image is built once and reused, so jobs are reproducible across days and weeks.

What the cost is

Ephemerality is not free. Three costs show up:

• Provisioning latency. Each job pays the cost of booting a VM, pulling the image, and starting the runner daemon. On AWS EC2 this is 60-90 seconds; on Kubernetes it is 10-30 seconds for the runner pod to start.
• No warm caches. A persistent runner that has built the project’s Docker image once has it cached. An ephemeral runner pulls it on every job. The fix is a remote cache (S3, GCS, an internal registry), not a local cache on the runner host.
• Image-build investment. Ephemeral runners require an image that contains everything the job needs. The image must be built, versioned, and stored in a registry the runners can pull from. This is real work the team owns.

The pattern: just-in-time provisioning

flowchart LR
    J["Forge enqueues job"] --> W["Listener / webhook"]
    W --> P["Provision host\n(VM or pod)"]
    P --> C["Configure runner\n(config.sh)"]
    C --> R["Runner takes job"]
    R -->|"job done"| X["Tear down host"]
    X --> W

The canonical pattern:

1. A listener watches the forge’s job queue. When a job for the runner pool appears, the listener triggers provisioning.
2. A host is provisioned from a prebuilt image. The image contains the runner binary, the tooling, and the configuration. Provisioning is fast because the image is ready.
3. config.sh registers the runner against the repo or org. The registration token is short-lived and pulled by the listener from the forge API.
4. The runner takes the job. The runner process connects, downloads the job, and runs it.
5. On job end, the host is destroyed. The runner deregisters; the VM or pod is torn down.

Implementing the pattern

Two practical implementations:

On a VM (e.g. AWS EC2). A controller (Lambda, a small service) listens to GitHub’s workflow_job webhook, launches an EC2 instance from a prebuilt AMI, SSHes in to register the runner with a fresh registration token, and terminates the instance when the job ends. The AMI is built by Packer with the runner binary, common tools, and the project-specific toolchain.

On Kubernetes (Actions Runner Controller). ARC is a Kubernetes operator that scales runner pods based on job queue depth. Part XL-04 covers ARC in detail; the relevant property here is that each runner pod is destroyed when its job ends, giving Kubernetes-native ephemerality.

# A workflow that requests an ephemeral self-hosted runner
jobs:
  build:
    runs-on: [self-hosted, ephemeral, linux]
    steps:
      - uses: actions/checkout@v4
      - run: ./build.sh

The ephemeral label is a convention: the listener (or ARC) matches it, provisions a host, and tears it down at job end. The label does not, by itself, make the runner ephemeral; the provisioning pattern does.

Caches that survive ephemerality

flowchart LR
    R["Ephemeral runner"] -->|"pull"| C["Remote cache\n(S3 / GCS / registry)"]
    C -->|"cached artifact"| R

Local caches on the runner are wasted (destroyed with the host). Push the cache out:

• Build outputs. Push to an internal S3 bucket; pull on cache-hit.
• Docker layers. Pull from an internal registry, not Docker Hub; the registry caches layers across runners.
• Language dependencies. Use actions/cache with a key derived from the lockfile; the cache lives in the forge’s cache backend, not on the runner.

Production discipline

1. Build the runner image once and reuse it. Packer (or equivalent) + a registry is the standard toolchain.
2. Listen for jobs, do not poll blindly. A controller triggered by workflow_job webhooks scales faster and costs less than a fleet of idle runners.
3. Move caches out of the runner. Local caches are destroyed; remote caches survive.
4. Measure job-startup latency. A slow image or a slow pull is the most common ephemeral-runner bug.

Cross-course references

• Linux for Production Sysadmins - Part XXIV (ImmutInf) covers the immutable-infrastructure principles that ephemeral runners embody.
• Ansible for Production Sysadmins - Part XXXVII (RepoArch) covers the runner-pool patterns that interact with ephemeral provisioning.
• Kubernetes for Production Sysadmins - Parts IX-XII (Workloads) cover the pod-lifecycle primitives that ARC uses for ephemeral runners in Part XL-04.

Quiz