Runner autoscaling and Actions Runner Controller — scaling on Kubernetes

A fixed-size runner pool wastes money at night and queues jobs at peak. The pool that is “always warm” spends all day sitting idle; the pool that is “right-sized” starves when demand spikes. The answer is autoscaling, and on Kubernetes the canonical answer is Actions Runner Controller.

ARC is an upstream project (originally community-built, now part of the GitHub Actions ecosystem) that runs self-hosted runner pods inside a Kubernetes cluster, scaling the pod count based on the forge’s job queue depth.

The problem: fixed pools are wrong-sized

flowchart LR
    subgraph FIXED["Fixed pool of 20 runners"]
        R1["Runner 1"] --- R20["Runner 20"]
    end
    Q["Forge queue"] -->|"jobs"| FIXED
    FIXED -. "20 idle at 3am,\n20 busy at 10am" .-> COST["Cost / SLA\ntrade-off"]

A fixed pool of 20 runners is over-provisioned at night and under-provisioned at peak. Either the team pays for capacity they do not need, or the team waits for capacity they do not have, or both.

ARC architecture

flowchart TB
    F["Forge\n(GitHub Actions)"] -->|"workflow_job\nwebhook"| L["ARC listener\n(long-lived pod)"]
    L -->|"scale to N"| K["Kubernetes API"]
    K --> P["Runner pods\n(ephemeral, one per job)"]
    P -->|"status, logs"| F
    L -->|"listens for"| F

ARC has three moving parts:

1. Controller. A Kubernetes controller that reconciles RunnerDeployment and RunnerSet resources. It watches the listener’s desired replica count and creates or deletes runner pods.
2. Listener. A long-lived pod that subscribes to the forge’s workflow_job webhook (or polls, in older versions). It translates job-queue depth into a desired replica count.
3. Runner pods. Short-lived pods, one per job, that register against the forge, take the job, run it, and terminate.

The listener is the bridge between the forge’s queue and Kubernetes’ scaling primitives. Without it, ARC would not know how many runner pods to create.

A minimal RunnerDeployment

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: prod-runners
  namespace: actions-runner-system
spec:
  replicas: 0
  template:
    spec:
      repository: my-org/my-repo
      labels:
        - self-hosted
        - linux
        - x64
        - prod

flowchart LR
    A["RunnerDeployment\n(desired state)"] --> B["Controller reconciles"]
    B --> C["RunnerSet\n(manages replicas)"]
    C --> D["Runner pod 1"]
    C --> E["Runner pod 2"]
    C --> F["Runner pod N"]

The controller creates the RunnerSet, the RunnerSet creates runner pods, and the listener watches the forge’s queue and asks the controller to scale.

Scale sets and min/max

ARC’s modern API uses EphemeralRunnerSet with RunnerScaleSet. The configuration expresses minimum and maximum replicas:

apiVersion: actions.github.com/v1alpha1
kind: EphemeralRunnerSet
metadata:
  name: prod-runners
spec:
  githubConfigUrl: https://github.com/my-org/my-repo
  maxRunners: 30
  minRunners: 0
  runnerGroup: prod
  template:
    spec:
      containers:
        - name: runner
          image: my-registry.example.com/runner:prod-v12
          env:
            - name: ACTIONS_RUNNER_INPUT_EPHEMERAL
              value: "true"

• minRunners: 0 scales the pool to zero when idle. No cost at night.
• maxRunners: 30 caps the burst. The listener will not request more than 30 runner pods.
• ephemeral: true (set via the env var above or the template) ensures each runner pod is destroyed at the end of its job.

The listener-to-pod flow

sequenceDiagram
    participant F as Forge
    participant L as ARC listener
    participant K as Kubernetes
    participant P as Runner pod
    F->>L: workflow_job queued
    L->>K: scale to 5
    K->>P: create runner pod
    P->>F: register, take job
    P->>F: run steps
    F->>P: job complete
    P->>K: terminate pod
    L->>K: scale down to 4

The flow that matters:

• The listener is the only long-lived component. A single ARC listener per cluster, watching all runner pools. It is the bridge.
• Runner pods are short-lived. Created when a job arrives and destroyed when the job ends. The Kubernetes pod lifecycle replaces the VM-rebuild loop of the cloud-controller pattern.
• Scale-down is graceful. When the queue empties, the listener asks for fewer pods; the controller removes idle ones. minRunners sets the floor.

When ARC is the right answer

ARC is the right runner platform when:

• The team already operates a Kubernetes cluster.
• The runner pool needs to scale to zero to control cost.
• The jobs are container-friendly (most CI is; some are not, e.g. macOS or GPU jobs that need host devices).
• The team can manage custom runner images in a registry.

ARC is overengineered when:

• The team has fewer than 50 jobs per day (the operational cost of running Kubernetes outweighs the savings).
• The jobs need non-container hosts (bare metal, specialised hardware that cannot be pod-attached).
• The team is already running a cloud-vendor autoscaler (e.g. EC2 fleet with runners.tf) and does not need Kubernetes.

Production discipline

1. Run the listener as a singleton per cluster. Multiple listeners do not scale better; they conflict.
2. Pin the runner image. A runner:latest image that breaks at 2am breaks every job in the pool.
3. Set minRunners deliberately. Zero is cheapest; one is fastest; more is rarely needed.
4. Monitor pod-creation latency. The listener-to-pod round-trip is the cold-start budget.
5. Separate pools by trust level. A prod pool and an untrusted pool with different IAM, different egress, different secrets.

Cross-course references

• Kubernetes for Production Sysadmins - Parts IX-XII (Workloads) cover the pod-lifecycle primitives ARC uses.
• Linux for Production Sysadmins - Part XXIV (ImmutInf) covers the immutable-image pattern that ARC’s runner template embodies.
• Terraform for Production Sysadmins - Part XXIV (IaCRepos) covers the IaC patterns for managing ARC manifests.

Quiz