Skip to main content
RunBook Academy

← All runbooks in Git, CI/CD & GitOps

high riskservice affecting~30 min

Runbook: Restore Runner Capacity

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

  • · Capture the queued job count and the longest-waiting job age: gh api /repos/<org>/<repo>/actions/runs?status=queued --paginate --jq "length, [.workflow_runs[] | .created_at, .run_started_at, (.created_at | fromdateiso8601) | @tsv]" | head -50 (or glab ci list --scope queued)
  • · Identify the runner set(s) affected: gh api /repos/<org>/<repo>/actions/runners --jq ".runners[] | {name,status,busy,labels:[.labels[].name]}" and group by label
  • · Confirm the queue is growing, not just long: re-check the queued count 60 seconds after the first check (gh api .../actions/runs?status=queued --jq "length")
  • · Capture the autoscaler state: for ARC, kubectl -n actions-runner-system get hpa -o yaml | head -100; for self-hosted with autoscaler, cat /var/log/runner-autoscaler.log | tail -200 (or the equivalent)
  • · Identify the bottleneck: runner pool size, runner concurrency per host, image pull latency, PVC binding, or platform API rate-limit. The choice between scaling-out and optimizing depends on the bottleneck

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

  1. 1STEP 1 - Drain queued jobs of lowest priority first: jobs with the longest wait time and the lowest impact (lint, docs build, nightly) are moved to a manual-approval workflow or skipped. gh api -X POST /repos/<org>/<repo>/actions/runs/<run-id>/cancel (with reviewer approval for any non-nightly job)
  2. 2STEP 2 - For ARC: scale the runner scale set's autoscaler max replicas higher than the current value. Read the current: kubectl -n actions-runner-system get hra -l actions.github.com/scale-set-name=<set> -o yaml. Edit: kubectl -n actions-runner-system edit hra actions-runner-scale-set-<set> and increase maxReplicas. This takes effect immediately without a deploy
  3. 3STEP 3 - For self-hosted runners with a cluster autoscaler: ensure the underlying node pool can absorb the new runner pods. aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names <asg> (or kubectl get nodes --show-labels | grep <label>). If the node pool is at capacity, increase the ASG desired capacity or the Karpenter NodePool max
  4. 4STEP 4 - Provision additional runner hosts out-of-band for a fleet that does not autoscale. Use the IaC: cd infra/runner && terraform plan -out=tfplan && terraform apply tfplan (with the change ticket referenced). The new hosts auto-register with the platform
  5. 5STEP 5 - Wait for the new runners to register and become healthy: gh api /repos/<org>/<repo>/actions/runners --jq "[.runners[] | select(.status==\\"online\\")] | length" (expect the count to grow as new hosts come online)
  6. 6STEP 6 - Re-check the queue and confirm it is draining: gh api /repos/<org>/<repo>/actions/runs?status=queued --jq "length". If the queue is flat or growing, the new runners are not claiming jobs (label mismatch, image pull stuck, or platform rate-limit)
  7. 7STEP 7 - If the new runners are online but not claiming, run git-cicd-gitops-rb-10-troubleshoot-runner per runner. Common reasons: label mismatch (the new runner has different --labels from the workflow's runs-on), runner registration to a different repo or org, or the autoscaler target metric is wrong (scaling on a metric that does not correlate with queue depth)
  8. 8STEP 8 - For GitHub-hosted: capacity is platform-managed. Escalate to GitHub support via https://support.github.com with the queue depth, the longest-wait timestamp, and the runner labels. GitHub does not sell "more hosted runners"; the answer is either self-hosted migration or running jobs on a different time
  9. 9STEP 9 - For AWS/GCP/Azure-hosted runners on Spot/Preemptible: capacity loss may be due to Spot interruption. Check kubectl get nodes -l karpenter.sh/capacity-type=spot -o wide or the cloud's Spot interruption dashboard. If Spot is unreliable for the runner image, switch to on-demand for the affected scale set (with cost approval)
  10. 10STEP 10 - Document the incident: the original capacity, the trigger (sustained queue, autoscaler misconfiguration, Spot interruption), the time-to-recovery, and the change applied. Open a follow-up ticket to either autoscale the runner pool further or migrate the workload to a different runner type

4 · Verification

Confirm the procedure actually fixed the problem.

  • ✓gh api .../actions/runs?status=queued --jq "length" returns to baseline within 30 minutes of the scale action
  • ✓The longest-waiting queued job age drops below 10 minutes (gh api .../actions/runs?status=queued --jq "[.workflow_runs[] | (.created_at | fromdateiso8601) | . - now | -.] | max" in jq terms: the largest of now - created_at)
  • ✓The runner count for the affected labels is greater than or equal to the pre-incident count
  • ✓A test job targeting the runner label completes within the SLA (gh run watch <run-id>)
  • ✓The autoscaler reports the expected metric: for ARC, kubectl get hra shows the desired replicas are between min and max, and the autoscaler is not in a CrashLoopBackOff
  • ✓CI success rate (gh run list --workflow <workflow> --limit 50 --json conclusion | jq "[.[] | select(.conclusion==\"success\")] | length / 50") returns to baseline within 1 hour

5 · Rollback

If verification fails, undo the procedure in reverse order.

  • ↶If the scaled-up runners consume too many resources (CPU, memory, cost) and the queue drains faster than expected: scale back down to the original max replicas (kubectl -n actions-runner-system edit hra actions-runner-scale-set-<set> and lower maxReplicas)
  • ↶If the new runners fail health checks: drain them (kubectl drain <node> --ignore-daemonsets --delete-emptydir-data for k8s; sudo systemctl stop actions.runner.* for self-hosted) and remove from the platform (gh api -X DELETE /repos/<org>/<repo>/actions/runners/<id>)
  • ↶If scaling the runner pool did not relieve the queue (the bottleneck is upstream, e.g. image pull or PVC binding): revert the scale action and re-diagnose. The autoscaler may be hitting a node-pool limit or a quota that needs an admin request
  • ↶If Spot interruption is the cause and on-demand is too costly: revert to Spot after the interruption window, but document the reliability cost in the ticket. Spot for CI runners is cheap but unreliable
  • ↶If the autoscaler was the cause (misconfigured metric, target too low): revert the autoscaler manifest to the last known-good SHA (git revert <bad-sha>) and apply the scale-up via a different path (direct replicas, not the HPA)

6 · Escalation

When the runbook isn't enough, contact:

  • · Runner capacity loss is regional (us-east-1 outage, GCP zone down): engage the cloud provider, and consider region failover for the runner pool (provision in a second region with cross-region label routing)
  • · Runner capacity loss is due to a security event (compromise, CVE in the runner image): take affected runners offline per git-cicd-gitops-rb-13-respond-to-compromised-runner before scaling up
  • · Runner capacity loss is sustained (queue does not drain within 30 minutes of scaling): the bottleneck is not capacity. Engage platform team for diagnosis
  • · GitHub-hosted runners are saturated: file a support ticket with the workflow SHA, queue depth, and the time window. The SLA for hosted-runner capacity incidents is 4 hours; plan accordingly
  • · Cost cap reached for the runner pool: get finance approval for the spend, then scale up. Do not bypass the cost cap

Runner capacity restoration is a queue management problem. The runner is not broken; it is full. The decision is “scale up” or “scale down the queue” — both are valid. Draining the queue (cancelling low-priority jobs) is faster and cheaper than provisioning new runners, but it requires reviewer approval for the cancellations.

The runbook is: capture the queue state, identify the bottleneck, apply the cheapest fix that works (cancel, then scale), validate that the queue drains, and document. The scale-up path differs by runner type: ARC uses HorizontalRunnerAutoscaler, self-hosted with autoscaler uses the cloud”s ASG/Karpenter, GitHub-hosted is platform-managed and out of the team”s hands.

1. Capture the queue state

Read-only / Safe
$ echo '--- queued job count ---'
gh api /repos/REPLACE_WITH_ORG/REPLACE_WITH_REPO/actions/runs?status=queued --paginate --jq 'length'
echo '--- longest-waiting job age (seconds) ---'
gh api /repos/REPLACE_WITH_ORG/REPLACE_WITH_REPO/actions/runs?status=queued --paginate --jq '[.workflow_runs[] | (.now - (.created_at | fromdateiso8601))] | max // 0'
echo '--- runner state ---'
gh api /repos/REPLACE_WITH_ORG/REPLACE_WITH_REPO/actions/runners --jq '[.runners[] | {name,status,busy,labels:[.labels[].name]}] | sort_by(.name)' | head -40
echo '--- runner count by label ---'
gh api /repos/REPLACE_WITH_ORG/REPLACE_WITH_REPO/actions/runners --jq '[.runners[] | .labels[].name] | group_by(.) | map({(.[0]): length}) | add'

A queue depth of 0 with runners all busy is normal. A queue depth that grows while runners are idle is a label mismatch (git-cicd-gitops-rb-10-troubleshoot-runner). A queue depth that grows with runners all busy is the case this runbook covers.

2. Identify the bottleneck

Read-only / Safe
$ echo '--- runner concurrency vs matrix size ---'
gh api /repos/REPLACE_WITH_ORG/REPLACE_WITH_REPO/actions/runners --jq '.runners[] | {name,busy}'
echo '--- max parallel per runner ---'
gh api /repos/REPLACE_WITH_ORG/REPLACE_WITH_REPO/actions/runners --jq '.runners[] | {name,busy_count:.busy}'
echo '--- matrix jobs queued ---'
gh api /repos/REPLACE_WITH_ORG/REPLACE_WITH_REPO/actions/runs?status=queued --jq '[.workflow_runs[] | .name] | group_by(.) | map({(.[0]): length}) | add'
echo '--- autoscaler state ---'
kubectl -n actions-runner-system get hra -l actions.github.com/scale-set-name=REPLACE_WITH_SET -o yaml 2>/dev/null | head -100 || echo no-arc

Bottlenecks: runner count is at autoscaler max (maxReplicas), the node pool is full (no new pods schedulable), image pull is slow (initial job latency), PVC binding fails (PVCs stuck in Pending), or the platform API is rate-limited (rare; check https://www.githubstatus.com).

3. Drain queued low-priority jobs (with approval)

Read-only / Safe
$ echo '--- jobs older than 30 minutes, by workflow ---'
gh api /repos/REPLACE_WITH_ORG/REPLACE_WITH_REPO/actions/runs?status=queued --paginate --jq '.workflow_runs[] | select((now - (.created_at | fromdateiso8601)) > 1800) | {id: .id, name: .name, age_min: ((now - (.created_at | fromdateiso8601)) / 60 | floor)}'
echo '--- cancel one nightly job ---'
gh api -X POST /repos/REPLACE_WITH_ORG/REPLACE_WITH_REPO/actions/runs/REPLACE_WITH_NIGHTLY_RUN_ID/cancel
echo '--- a real PR build cannot be cancelled without reviewer approval ---'
# reviewer: "approve cancel of PR build #REPLACE_WITH_ID for incident INC-XXXX"

4. Scale up ARC

Read-only / Safe
$ SCALE_SET="prod-runners"
NAMESPACE="actions-runner-system"
CURRENT=$(kubectl -n "$NAMESPACE" get hra "actions-runner-scale-set-$SCALE_SET" -o jsonpath='{.spec.maxReplicas}')
NEW=$((CURRENT * 2))
[ "$NEW" -gt 50 ] && NEW=50
echo "current maxReplicas: $CURRENT"
echo "new maxReplicas: $NEW"
kubectl -n "$NAMESPACE" patch hra "actions-runner-scale-set-$SCALE_SET" --type=merge -p "{"spec":{"maxReplicas":$NEW}}"
kubectl -n "$NAMESPACE" get hra "actions-runner-scale-set-$SCALE_SET" -o yaml | head -40

Doubling max replicas is a safe emergency action. The HPA scales within the new max when the queue-depth metric triggers.

5. Scale up node pool for self-hosted

Read-only / Safe
$ echo '--- for AWS ASG ---'
aws autoscaling update-auto-scaling-group \
--auto-scaling-group-name "$ASG_NAME" \
--desired-capacity "$NEW_DESIRED" \
--max-size "$NEW_MAX"
aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names "$ASG_NAME"

echo '--- for Karpenter ---'
kubectl get nodepool -o yaml | grep -E '(name:|min:|max:|weight:)' | head -20
kubectl patch nodepool REPLACE_WITH_NAME --type=merge -p '{"spec":{"limits":{"cpu":"100","memory":"400Gi"}}}'
kubectl apply -f - <<EOF
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
name: runners
spec:
limits:
  cpu: "200"
  memory: 800Gi
EOF

For ASG, the change is the desired and max size. For Karpenter, the change is the NodePool limits.cpu and limits.memory. The NodePool must allow new nodes to be created before the HPA can schedule pods on them.

6. Wait for new runners to register

Read-only / Safe
$ echo '--- runner count growth ---'
INITIAL=$(gh api /repos/REPLACE_WITH_ORG/REPLACE_WITH_REPO/actions/runners --jq 'length')
echo "initial: $INITIAL"
for i in 1 2 3 4 5 6 7 8 9 10; do
sleep 30
CURRENT=$(gh api /repos/REPLACE_WITH_ORG/REPLACE_WITH_REPO/actions/runners --jq 'length')
ONLINE=$(gh api /repos/REPLACE_WITH_ORG/REPLACE_WITH_REPO/actions/runners --jq '[.[] | select(.status=="online")] | length')
echo "t=$((i*30))s: total=$CURRENT online=$ONLINE"
done

The new runners should come online within 1-3 minutes of the scale action. If they do not, the runner image is failing to start or register — kubectl logs on the runner pods, or journalctl -u actions.runner.* on the self-hosted host.

7. Verify the queue drains

Read-only / Safe
$ echo '--- queue depth over time ---'
for i in 1 2 3 4 5 6; do
QUEUE=$(gh api /repos/REPLACE_WITH_ORG/REPLACE_WITH_REPO/actions/runs?status=queued --jq 'length')
LONGEST=$(gh api /repos/REPLACE_WITH_ORG/REPLACE_WITH_REPO/actions/runs?status=queued --jq '[.workflow_runs[] | (now - (.created_at | fromdateiso8601))] | max // 0')
echo "t=$((i*60))s: queue=$QUEUE longest_wait=${LONGEST}s"
sleep 60
done

A draining queue drops by approximately the runner concurrency per interval. If the queue is flat or growing, the runners are online but not claiming — diagnose per git-cicd-gitops-rb-10-troubleshoot-runner.

8. Re-balance if a Spot interruption caused the loss

Read-only / Safe
$ echo '--- check Spot interruption notices ---'
kubectl get nodes -l karpenter.sh/capacity-type=spot -o wide | head -20
kubectl get events --sort-by='.lastTimestamp' -A | grep -i spot-interrupt | tail -10
echo '--- for the affected nodes, cordon and replace ---'
for node in $(kubectl get nodes -l karpenter.sh/capacity-type=spot -o name | head -10); do
kubectl cordon "$node"
done
echo '--- temporarily switch the runner scale set to on-demand ---'
kubectl -n actions-runner-system patch hra "actions-runner-scale-set-$SCALE_SET" --type=merge -p '{"spec":{"template":{"spec":{"nodeSelector":{"karpenter.sh/capacity-type":"on-demand"}}}}}'

Switching to on-demand costs roughly 3x more than Spot. Document the cost in the change ticket and revert to Spot after the interruption window passes.

9. Document the incident

Read-only / Safe
$ gh issue create --repo REPLACE_WITH_ORG/REPLACE_WITH_REPO \
--title "runner capacity restoration $(date -u +%Y-%m-%d)" \
--body "Original capacity: <X runners>. Trigger: <queue, interruption, autoscaler>. Time-to-recovery: REPLACE_WITH_MINUTES. Action: <scale, cancel>. Cost delta: REPLACE_WITH_USD. Follow-up: <autoscale, migrate>." \
--label runner --label capacity --label incident

Verification

gh api .../actions/runs?status=queued --jq "length" returns to baseline within 30 minutes of the scale action. The longest-waiting queued job age drops below 10 minutes. The runner count for the affected labels is greater than or equal to the pre-incident count. A test job targeting the runner label completes within the SLA. The autoscaler reports the expected metric (desired replicas between min and max, no CrashLoopBackOff). CI success rate returns to baseline within 1 hour.

Rollback

If the scaled-up runners consume too many resources, scale back to the original max. If the new runners fail health checks, drain them and remove from the platform. If scaling did not relieve the queue (the bottleneck is upstream, e.g. image pull or PVC binding), revert the scale action and re-diagnose. If Spot interruption is the cause and on-demand is too costly, revert to Spot after the interruption window but document the reliability cost. If the autoscaler was the cause, revert the manifest to the last known-good SHA and apply the scale-up via a different path.

References

  1. Actions Runner Controller — Horizontal Runner Autoscaler
  2. GitHub Docs — About self-hosted runners
  3. Karpenter — NodePool autoscaling
  4. Kubernetes — Horizontal Pod Autoscaler
  5. AWS — Spot Instance interruption notices