Skip to main content
RunBook Academy

← All runbooks in Git, CI/CD & GitOps

high riskservice affecting~30 min

Runbook: Troubleshoot a Runner

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

  • · Identify the runner by name and label: gh api /repos/<org>/<repo>/actions/runners --paginate --jq ".runners[] | {id,name,status,os,labels:[.labels[].name]}" (or the GitLab/Jenkins equivalent)
  • · Capture the runner's platform-side state: gh api .../runners --jq ".runners[] | {id,name,os,status,busy}". The runner object has no last-contact timestamp — status reports only "online"/"offline". For last-activity evidence, read the host's _diag/Runner_*.log (and journalctl -u "actions.runner.*") and the job timestamps from the jobs API
  • · Check the platform's job queue for jobs waiting on this label: gh api /repos/<org>/<repo>/actions/runs?status=queued --paginate --jq "[.workflow_runs[] | select(.runner_labels[] | contains(\\"<label>\\"))] | length" (or glab ci list --scope queued)
  • · For self-hosted: confirm SSH or console access to the runner host (ssh runner@<host> "systemctl status actions.runner.*"). Without access, the runbook cannot inspect the runner process
  • · For Kubernetes runners: confirm kubectl to the runner controller namespace (kubectl -n actions-runner-system get pods for ARC, or kubectl -n gitlab-runner get pods for GitLab)

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

  1. 1STEP 1 - Connectivity check (network and platform agent): on the runner host, curl -fsS https://api.github.com/zen (or https://gitlab.com/api/v4/version) to confirm outbound network reachability to the platform. Then systemctl status actions.runner.<org>-<repo>.<runner-name>.service (or the GitLab equivalent) to confirm the agent process is running
  2. 2STEP 2 - Inspect the runner service log: journalctl -u actions.runner.<org>-<repo>.<runner-name>.service --since "1 hour ago" --no-pager | tail -200 and look for the last successful "Listening for jobs" line. If the last log line is older than 5 minutes, the agent is stuck; restart it
  3. 3STEP 3 - Labels check (self-registration, label drift): gh api /repos/<org>/<repo>/actions/runners --jq ".runners[] | select(.name==\\"<runner-name>\\") | .labels[].name". Compare against the workflow's runs-on value: grep -r "runs-on" .github/workflows/. Drift (runner has self-hosted,linux,x64 but workflow needs self-hosted,gpu) means jobs will never be claimed
  4. 4STEP 4 - Capacity check (CPU/memory pressure, concurrent job limit): on the runner host, top -bn1 | head -10, free -h, df -h /home/runner/_work. If memory is >90% used, jobs are being OOM-killed. If disk is >90% used, the work directory is full and jobs will fail to checkout. Also check cat ./_diag/Runner_*.json 2>/dev/null for the GitHub Actions runner's self-diagnostic
  5. 5STEP 5 - Concurrent job limit: the runner config /home/runner/.runner has "maxParallel" which limits simultaneous jobs. If the workflow has a matrix with >N entries and the runner has maxParallel=N, jobs are queued indefinitely. Compare grep -A2 "strategy:" .github/workflows/<workflow> against jq .AgentListener .MaxParallel /home/runner/.runner
  6. 6STEP 6 - Self-hosted runner registration: confirm the runner is registered with the correct repo or org. cat /home/runner/.runner | jq .AgentConfig .GitHubRepository should match the repo, or .GitHubEnterpriseUrl for enterprise. A runner registered to repo A but used by repo B will not claim jobs
  7. 7STEP 7 - For Kubernetes runners: check the runner controller logs: kubectl -n actions-runner-system logs -l app.kubernetes.io/name=actions-runner-controller --tail=200. Then check the runner pods: kubectl -n actions-runner-system get pods -l actions.github.com/scale-set-name=<set-name>. Stuck pods in Pending mean the runner set cannot scale (resource quota, PVC binding, node selector mismatch)
  8. 8STEP 8 - Restart the runner service if the agent is stuck: sudo systemctl restart actions.runner.<org>-<repo>.<runner-name>.service && journalctl -u actions.runner.<org>-<repo>.<runner-name>.service --since "1 minute ago" --no-pager -f (watch the reconnect). The runner should re-register within 30 seconds and show "Listening for jobs"
  9. 9STEP 9 - If the runner is healthy but jobs are queued: scale the runner set (for ARC: kubectl scale autoscaling/actions-runner-scale-set-<name> --replicas=<N>; for self-hosted: provision additional runners with the same labels). One runner saturated is not a runner bug; it is a capacity gap. See git-cicd-gitops-rb-11-restore-runner-capacity
  10. 10STEP 10 - Document: capture the runner name, the label, the diagnosis category (connectivity / labels / capacity / registration), the action taken, and the time-to-recovery. This goes into the change ticket so the next runner incident has a starting point

4 · Verification

Confirm the procedure actually fixed the problem.

  • ✓gh api /repos/<org>/<repo>/actions/runners --jq ".runners[] | select(.name==\\"<runner-name>\\") | .status" returns online
  • ✓journalctl -u actions.runner.<org>-<repo>.<runner-name>.service --since "5 minutes ago" --no-pager shows a fresh "Listening for jobs" line
  • ✓The queued job count for the runner's labels is decreasing: gh api /repos/<org>/<repo>/actions/runs?status=queued --jq "[.workflow_runs[] | select(.runner_labels[] | contains(\\"<label>\\"))] | length" shows progress
  • ✓A test job targeting the runner (runs-on: <label>) completes successfully on the next attempt
  • ✓CPU/memory/disk on the runner host are below 80% used
  • ✓For Kubernetes runners: kubectl get hpa --namespace actions-runner-system shows the autoscaler is healthy and scaling on the expected metric

5 · Rollback

If verification fails, undo the procedure in reverse order.

  • ↶If restarting the runner service does not bring it online: stop the service (sudo systemctl stop actions.runner.*) and re-register the runner. Use the ./config.sh script and the registration token from the repo Settings → Actions → Runners → New self-hosted runner. The existing registration is replaced, not duplicated
  • ↶If the runner came online after restart but jobs still queue: the labels are wrong. Update the runner's labels in the repo Settings → Runners UI, or re-register with --labels <label1>,<label2>
  • ↶If scaling the runner set did not relieve the queue: check the autoscaler metrics (kubectl get hpa -n actions-runner-system -o yaml) and confirm the desired metric (queue depth, CPU, etc.) is increasing the replica count. If the HPA is at max but the queue is still growing, the bottleneck is upstream of the runner (e.g. PVC, network)
  • ↶If the runner host is irrecoverable (disk failure, kernel panic, network outage): drain the host (sudo systemctl stop actions.runner.* && sudo systemctl disable actions.runner.*), remove the runner from the repo (Settings → Runners → Remove), and provision a replacement. Document the host's removal in the change ticket
  • ↶If the runner appears healthy but jobs fail with "no suitable runner" intermittently: check for runner-label race conditions (grep -r "runs-on: \\[self-hosted" .github/workflows/). The label selector may be specifying a list when the runner only has one label

6 · Escalation

When the runbook isn't enough, contact:

  • · Runner host is unreachable from the network (ping fails, SSH fails): host-level outage, escalate to infrastructure. The runner cannot be diagnosed remotely
  • · Runner host is reachable but all runners on it are offline simultaneously: a host-level event (reboot, OOM, disk full) took them all down. Diagnose the host, not the runners
  • · Runner has been compromised (unexpected processes, outbound connections, modified binaries): take the host offline per git-cicd-gitops-rb-13-respond-to-compromised-runner
  • · Runner cannot reach the platform API (DNS or egress blocked by corporate firewall): engage the network team. The runner config may need to use a proxy (https_proxy=<proxy> in /home/runner/.env)
  • · Multiple runners across the fleet are offline: platform-wide issue, escalate to the platform team

A runner that does not run jobs has three plausible causes: it cannot reach the platform (connectivity), it can reach the platform but does not match the label selector (labels), or it can match but is saturated (capacity). The runbook triages in that order because each diagnosis requires the previous one to be ruled out.

The runner is a stateless worker. Its diagnosis is “is it online, is it labelled correctly, and can it actually run?” — not “what is the job’s failure?”. Job failures are diagnosed with git-cicd-gitops-rb-09-troubleshoot-failed-ci-pipeline. Runner failures are diagnosed here.

1. Connectivity check

Read-only / Safe
$ RUNNER_HOST="runner-01.example.com"
RUNNER_NAME="runner-01"
ssh "$RUNNER_HOST" '
echo "--- network reachability ---"
curl -fsS https://api.github.com/zen 2>&1 | head -3
curl -fsS https://github.com 2>&1 | head -3
echo "--- runner service status ---"
systemctl status "actions.runner.*.service" --no-pager -l | head -20
echo "--- process listening ---"
ss -tnlp | grep -E "(443|8443)" | head -10
'

If curl fails, the runner host has an egress problem (DNS, proxy, firewall). The runner service can be running but unable to reach the platform — and the platform will mark it offline after the heartbeat timeout.

2. Inspect the runner service log

Read-only / Safe
$ RUNNER_HOST="runner-01.example.com"
ssh "$RUNNER_HOST" '
echo "--- last 200 lines of the runner service log ---"
journalctl -u "actions.runner.*.service" --since "1 hour ago" --no-pager | tail -200
echo "--- last successful "Listening for Jobs" line ---"
journalctl -u "actions.runner.*.service" --no-pager | grep -F "Listening for Jobs" | tail -3
'

The log line Listening for Jobs is the heartbeat. If the last occurrence is older than 5 minutes, the agent is stuck. Common causes in the log: Failed to refresh configuration, Connection refused, Authentication failed, TLS handshake timeout.

3. Labels check

Read-only / Safe
$ RUNNER_NAME="runner-01"
LABEL="self-hosted-linux-large"
gh api /repos/REPLACE_WITH_ORG/REPLACE_WITH_REPO/actions/runners --paginate --jq ".runners[] | select(.name==\"$RUNNER_NAME\") | .labels[].name"
echo '--- the workflow labels ---'
grep -rh "runs-on:" .github/workflows/ | sort -u
echo '--- jobs queued with the expected label ---'
gh api /repos/REPLACE_WITH_ORG/REPLACE_WITH_REPO/actions/runs?status=queued --paginate --jq "[.workflow_runs[] | select(.runner_labels[] | contains(\"$LABEL\"))] | length"

Label drift happens when a workflow’s runs-on is updated to add a new label (e.g. gpu, large) but the runner was not re-registered with the new label. The workflow appears to be queued forever because no runner matches.

4. Capacity check

Read-only / Safe
$ RUNNER_HOST="runner-01.example.com"
ssh "$RUNNER_HOST" '
echo "--- CPU ---"
top -bn1 | head -10
echo "--- memory ---"
free -h
echo "--- disk ---"
df -h /home/runner/_work
echo "--- current job count ---"
ps -ef | grep -c "Runner.Listener"
'

If free shows <10% available, jobs are being OOM-killed. If df shows >90% on the work directory, jobs cannot checkout. The runner is online but cannot do useful work.

5. Concurrent job limit

Read-only / Safe
$ RUNNER_HOST="runner-01.example.com"
ssh "$RUNNER_HOST" '
cat /home/runner/.runner | jq ".AgentListener"
echo "--- max parallel ---"
cat /home/runner/.runner | jq ".MaxParallel"
'
echo '--- matrix size in workflows ---'
grep -h "matrix:" .github/workflows/*.yml | sort -u | head -20
grep -rh "max-parallel:" .github/workflows/*.yml | sort -u | head -20

A runner with MaxParallel=4 cannot run a 6-entry matrix in parallel. The workflow will appear to queue because no runner can claim all 6 simultaneously.

6. Self-hosted runner registration

Read-only / Safe
$ RUNNER_HOST="runner-01.example.com"
ssh "$RUNNER_HOST" '
cat /home/runner/.runner | jq ".AgentConfig | {GitHubRepository, GitHubEnterpriseUrl, RunnerGitHubUrl}"
'

A runner registered to &lt;org&gt;/&lt;repo-A&gt; but used by &lt;org&gt;/&lt;repo-B&gt; will not claim jobs. The fix is to re-register with the correct repo’s token.

7. For Kubernetes runners

Read-only / Safe
$ NAMESPACE="actions-runner-system"
SCALE_SET="prod-runners"
kubectl -n "$NAMESPACE" get pods -l "actions.github.com/scale-set-name=$SCALE_SET"
echo '--- autoscaler ---'
kubectl -n "$NAMESPACE" get hpa -l "actions.github.com/scale-set-name=$SCALE_SET" -o yaml | head -60
echo '--- controller logs ---'
kubectl -n "$NAMESPACE" logs -l app.kubernetes.io/name=actions-runner-controller --tail=100 --no-pager | head -60
echo '--- pending pods (events) ---'
kubectl -n "$NAMESPACE" describe pods -l "actions.github.com/scale-set-name=$SCALE_SET" | grep -A5 -E "(Events:|Pending)" | head -40

Pending pods with FailedScheduling events usually mean a node selector mismatch, a PVC binding failure, or a resource quota exhaustion. Pending pods with no events usually mean the controller itself is stuck; restart the controller deployment.

8. Restart the runner service

Read-only / Safe
$ RUNNER_HOST="runner-01.example.com"
ssh "$RUNNER_HOST" '
sudo systemctl restart "actions.runner.*.service"
sleep 5
journalctl -u "actions.runner.*.service" --since "1 minute ago" --no-pager -f
' &
WAIT_PID=$!
sleep 30
kill $WAIT_PID 2>/dev/null
ssh "$RUNNER_HOST" 'systemctl is-active "actions.runner.*.service"'

The -f flag follows the log. The runner should re-register within 30 seconds and show “Listening for Jobs”. If the restart fails to produce that log line, re-register (next step).

9. Re-register if restart does not help

Read-only / Safe
$ RUNNER_HOST="runner-01.example.com"
RUNNER_NAME="runner-01"
ssh "$RUNNER_HOST" '
cd /home/runner
sudo systemctl stop "actions.runner.*.service"
sudo ./config.sh remove --token "REPLACE_WITH_REGISTRATION_TOKEN_FROM_REPO_SETTINGS"
REG_TOKEN="REPLACE_WITH_NEW_TOKEN"
sudo ./config.sh --url https://github.com/REPLACE_WITH_ORG/REPLACE_WITH_REPO --token "$REG_TOKEN" --labels "self-hosted,linux,large" --name "'"$RUNNER_NAME"'" --work _work
sudo ./run.sh &
sudo systemctl enable "actions.runner.*.service"
sudo systemctl start "actions.runner.*.service"
'
gh api /repos/REPLACE_WITH_ORG/REPLACE_WITH_REPO/actions/runners --jq ".runners[] | select(.name==\"$RUNNER_NAME\") | .status"

Re-registration replaces the existing entry. Do not register twice with a different name; the platform will list both, and old tokens remain valid until manually removed.

10. Document and follow up

Read-only / Safe
$ RUNNER_NAME="runner-01"
LABEL="self-hosted-linux-large"
gh issue create --repo REPLACE_WITH_ORG/REPLACE_WITH_REPO \
--title "runner incident: $RUNNER_NAME" \
--body "Runner: $RUNNER_NAME. Label: $LABEL. Diagnosis: REPLACE_WITH_CATEGORY. Action: REPLACE_WITH_ACTION. Time-to-recovery: REPLACE_WITH_DURATION." \
--label runner --label incident

Verification

gh api .../runners --jq ".runners[] | select(.name==$RUNNER_NAME) | .status" returns online. journalctl shows a fresh “Listening for Jobs” line within the last 5 minutes. The queued job count for the runner’s labels is decreasing. A test job targeting the runner’s label completes successfully. CPU/memory/disk on the runner host are below 80% used. For Kubernetes runners, the HPA is healthy and scaling on the expected metric.

Rollback

If restarting the runner service does not bring it online, stop the service and re-register using ./config.sh. If the runner came online after restart but jobs still queue, the labels are wrong — update them in the UI or re-register with the right --labels. If scaling the runner set did not relieve the queue, the autoscaler is not the bottleneck — check upstream (PVC, network, image pull). If the runner host is irrecoverable, drain it, remove the runner from the repo, and provision a replacement. If jobs fail with “no suitable runner” intermittently, check for label-selector races in the workflow files.

References

  1. GitHub Docs — Self-hosted runners
  2. GitHub Docs — Troubleshooting self-hosted runners
  3. Actions Runner Controller (ARC)
  4. GitLab Runner — Troubleshooting
  5. systemd — service management