Runner autoscaling with ARC — Kubernetes-based scale to zero

ARC’s value is the ability to scale runner pods to zero at night and back up to dozens at peak. The control plane that makes this work is a Kubernetes controller reconciling an EphemeralRunnerSet against the forge’s queue depth. The operational question is: when something is wrong with scaling, what does the team look at?

Inspecting an ARC deployment

The three kubectl commands that matter when investigating scaling behaviour:

ARC_NS=actions-system

kubectl get actionsrunnersets -n "$ARC_NS"

This lists every EphemeralRunnerSet (or RunnerSet, on the older ARC API) in the cluster, with name and age. The output tells the operator which runner pools exist. It does not yet tell them how the pool is configured.

kubectl get hpa -n "$ARC_NS"

HorizontalPodAutoscalers do not directly scale ARC runners

• the listener does that. But ARC’s autoscaling behaviour is constrained by maxRunners on the EphemeralRunnerSet and by the cluster’s capacity. The HPA view tells the operator whether the cluster can scale the workload class that ARC creates, which matters when the bottleneck is cluster capacity rather than ARC’s configuration.

kubectl describe actionsrunnersets -n "$ARC_NS" \
  prod-runners

This shows the current and desired replica counts, the listener connection status, and any conditions the controller has reported. “Progressing: False” with a reason of “RunnerScaleSetReconcilerError” is the typical signal that the listener cannot reach the forge.

The two scalers

ARC’s autoscaling is the composition of two scalers:

flowchart TB
  Q["Forge queue\nworkflow_job events"] --> L["ARC listener"]
  L -->|"desired replicas"| ERS["EphemeralRunnerSet"]
  ERS -->|"creates pods"| K["Kubernetes API"]
  K -.->|"capacity check"| HPA["HPA / cluster autoscaler"]
  HPA -.->|"nodes"| K

• Listener-driven scaling. The ARC listener watches the forge’s job queue and writes the desired replica count into the EphemeralRunnerSet status. This is the primary scaler - it reacts to actual job arrivals.
• Capacity-driven scaling. The Kubernetes cluster must have nodes to schedule the pods the listener asks for. If the cluster is full, the pods stay Pending and the queue grows. The HPA and cluster autoscaler respond to resource pressure; they do not respond to job arrivals.

A common confusion is treating ARC scaling as if it were HPA scaling. It is not. ARC’s listener is the queue-aware scaler; HPA is the resource-aware scaler that backstops it.

The operational knobs

The EphemeralRunnerSet exposes three knobs the operator sets deliberately:

• minRunners. The floor. 0 means scale to zero when idle; 1 keeps one warm; N keeps N warm. Cost versus cold-start latency is the trade.
• maxRunners. The ceiling. The listener will not request more than this. Set it to the cluster’s surge capacity for the runner class, not to a number that pleases a budget.
• Runner template resource requests/limits. The pod template’s resources.requests determine how the scheduler places pods; resources.limits determine the pod’s runtime ceiling. A pod that exceeds its limits is throttled or killed, which is rarely what the team wants during a build.

What to alert on

Four signals cover most ARC operational issues:

1. Listener connected. The EphemeralRunnerSet conditions report this. A False condition with reason ListenerNotConnected is the alert.
2. Pending runner pods. kubectl get pods -n "$ARC_NS" with Pending for more than 60 seconds means the cluster cannot place the pods.
3. Queue depth. From the forge’s metrics - jobs waiting for a runner. A depth that exceeds maxRunners for more than 5 minutes means the pool is permanently saturated.
4. Cold-start latency. The time from workflow_job queued to runner-pod-ready. A regression here is the first sign that the runner image is growing.

When autoscaling is not enough

ARC’s listener-driven scaling handles burst within the configured maxRunners. It does not handle:

• Cluster capacity. If the cluster is full of other workloads, ARC cannot schedule new pods regardless of maxRunners.
• Image-pull time. A 5 GB runner image delays every cold start. The fix is at the image layer.
• Job duration. A job that takes an hour is not made faster by autoscaling. The fix is at the workload layer.

Production discipline

1. Set minRunners from cold-start budget, not from intuition. Measure cold-start first; then decide.
2. Set maxRunners from cluster surge capacity. A maxRunners higher than the cluster can deliver is a number, not a plan.
3. Alert on listener health first. The listener is the bridge; if it is broken, nothing else scales.
4. Separate pools by trust level. A prod and an untrusted pool with different IAM, network, and templates.
5. Trim the runner image. A 1 GB image cold-starts in 10 seconds; a 5 GB image cold-starts in 50.

Cross-course references

• Git, CI/CD & GitOps - Part XL (Runners) covers the ARC architecture this lesson extends.
• Kubernetes for Production Sysadmins - Part XXII (HPAMath) covers HPA scaling in depth.
• Observability for Production Sysadmins - Part VIII (K8sObs) covers the alerting patterns used here.

Quiz