Skip to main content
RunBook Academy

← All runbooks in Git, CI/CD & GitOps

critical risksecurity relevant~90 min

Runbook: Respond to a Compromised Runner

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

  • · Identify the compromised runner: gh api /repos/<org>/<repo>/actions/runners --jq ".runners[] | select(.name==\\"<name>\\") | {id,name,os,status,busy,labels:[.labels[].name]}" (or the GitLab/Jenkins equivalent). The runner object has no last-activity timestamp — take last-activity evidence from the host's _diag/Runner_*.log and from job timestamps in the jobs API
  • · Capture the indicator that triggered the alert: outbound connection to unknown IP (tcpdump, ss -tnp), unexpected process (ps -ef), modified binary (rpm -V or debsums -c), secret in env (env | grep -E "(AKIA|ghp_)")
  • · Confirm the alert is not a known-good signal: scheduled jobs that legitimately use specific binaries (/usr/bin/curl to api.github.com), background workers that legitimately hold credentials, etc. The forensic evidence before disconnection is what distinguishes compromise from normal
  • · Engage security/IR: a compromised runner is a security incident. The runbook proceeds in parallel with the IR engagement, not after it
  • · Identify the blast radius: what secrets has this runner accessed? Runs are not attributed to runners (the actor filter selects the triggering user, never the runner) and the API never returns secret values. Enumerate run ids with gh api /repos/<org>/<repo>/actions/runs --paginate --jq ".workflow_runs[].id", then filter each run's jobs by runner: gh api /repos/<org>/<repo>/actions/runs/<run-id>/jobs --jq ".jobs[] | select(.runner_name==\\"<name>\\")"; the matched runs' workflow files name the secrets

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

  1. 1STEP 1 - Capture forensic evidence before taking the runner offline. On the runner host: date > /tmp/forensic-timestamp.txt, ps -efww > /tmp/forensic-processes.txt, ss -tnp > /tmp/forensic-connections.txt, ss -lnp > /tmp/forensic-listeners.txt, netstat -an > /tmp/forensic-netstat.txt (if available), find /tmp /var/tmp /dev/shm -type f -newer /etc/hostname > /tmp/forensic-recent-files.txt 2>/dev/null, last -n 50 > /tmp/forensic-logins.txt, history > /tmp/forensic-history.txt (root and runner users)
  2. 2STEP 2 - Capture network state: tcpdump -i any -w /tmp/forensic-capture.pcap -c 10000 & for 60 seconds, then kill %1 (best-effort; the runner's outbound traffic is the most useful artifact). For containerised runners: kubectl exec -n <ns> <runner-pod> -- tcpdump -i any -w /tmp/capture.pcap -c 5000 & (may require privileged context)
  3. 3STEP 3 - Capture filesystem state: sha256sum /usr/bin/* /usr/sbin/* /bin/* /sbin/* > /tmp/forensic-binaries.sha256 2>/dev/null and compare against the image's known-good hashes (the runner image's .dockerignore or the build manifest). rpm -Va (RHEL) or debsums -c (Debian) for installed package integrity
  4. 4STEP 4 - Snapshot the runner's disk before any further action: sudo fdisk -l /dev/xvda (find the device), then sudo dd if=/dev/xvda of=/tmp/forensic-disk.img bs=4M status=progress (this can take a while; consider stopping the runner process to reduce I/O contention first)
  5. 5STEP 5 - Drain the runner from active work: sudo systemctl stop actions.runner.*.service && sudo systemctl disable actions.runner.*.service (or kubectl scale deployment/runner --replicas=0 for Kubernetes runners). The runner stops claiming jobs immediately
  6. 6STEP 6 - Remove the runner from the platform: gh api -X DELETE /repos/<org>/<repo>/actions/runners/<id> (or the equivalent for GitLab/Bitbucket). The runner is no longer trusted, even if the host comes back
  7. 7STEP 7 - Audit every secret the runner had access to. Workflow-run objects carry no runner attribution — that lives on job objects. Enumerate run ids (gh api /repos/<org>/<repo>/actions/runs --paginate --jq ".workflow_runs[].id"), then keep each run id only if gh api /repos/<org>/<repo>/actions/runs/<run-id>/jobs --jq "[.jobs[] | select(.runner_name==\\"<runner-name>\\")] | length" is non-zero; collect the matches in /tmp/runs-on-this-runner.txt and wc -l /tmp/runs-on-this-runner.txt. The runs and jobs responses never include secret values; read each matched run's workflow file for the secrets it references. Treat every secret as compromised
  8. 8STEP 8 - Rotate every secret the runner had access to. For each workflow the runner executed, identify the secrets referenced (grep -rh "secrets\\." .github/workflows/) and rotate them per git-cicd-gitops-rb-08-rotate-git-credentials
  9. 9STEP 9 - Rebuild the runner from the verified image. For containerised runners: pull a fresh image from the verified registry (docker pull <image>:<tag>), re-deploy with the new image. For self-hosted: re-provision the host from the IaC (Terraform: terraform apply with the runner module) and re-register the runner
  10. 10STEP 10 - Re-add the runner to the platform with the new credentials, only after the IR review concludes the compromise is contained. Until then, the runner pool operates without this host
  11. 11STEP 11 - Add detection: a host-based IDS (OSSEC, Wazuh, Falco for Kubernetes), outbound traffic monitoring (egress firewall logging, VPC flow logs), and integrity monitoring (AIDE, Tripwire). The detection must alert on the indicator that triggered this incident, not on generic "host compromised"
  12. 12STEP 12 - Document the incident: indicator that triggered the response, evidence captured, secrets rotated, runner rebuilt, detection added. This goes into the IR ticket and the change ticket

4 · Verification

Confirm the procedure actually fixed the problem.

  • ✓The compromised runner is removed from the platform: gh api .../runners --jq ".runners[] | select(.name==\\"<name>\\")" returns empty
  • ✓Every secret the runner had access to is rotated and the new values authenticate: confirm per consumer per git-cicd-gitops-rb-08-rotate-git-credentials
  • ✓The cloud provider's audit log shows no unauthorised API calls using the rotated credentials since the rotation
  • ✓A new runner, built from the verified image, is registered and online: gh api .../runners --jq ".runners[] | select(.status==\\"online\\") | length" increases by 1
  • ✓The detection rules added in step 11 fire on the indicator that triggered this incident, verified with a synthetic test (ssh runner "touch /tmp/canary"; sleep 60; alert received)
  • ✓The IR ticket has the forensic artefacts (timestamps, process listing, network captures) preserved in the incident-response drive

5 · Rollback

If verification fails, undo the procedure in reverse order.

  • ↶If the runner was taken offline but the evidence was not captured first: the forensic value of the host is gone. Do not reboot the host before evidence is captured. If the host has already been rebooted, the in-memory state is gone; rely on the audit logs and the cloud-provider flow logs
  • ↶If the rotation was incomplete (a workflow still references the old secret): complete the rotation per git-cicd-gitops-rb-08-rotate-git-credentials before declaring the incident contained
  • ↶If the rebuilt runner cannot register or claim jobs: it is not the runner that is broken, it is the registration. Re-run git-cicd-gitops-rb-10-troubleshoot-runner against the new host
  • ↶If the detection rules added in step 11 produce too many false positives: tune the rules to the specific indicator. A noisy alert is worse than no alert because responders learn to ignore it
  • ↶If the IR review concludes the compromise is broader than this one runner: scope up. Other runners, CI workers, or developer machines may be affected. Do not stop at one host
  • ↶If the rebuilt runner is immediately compromised again: the rebuild image is the vector. Stop using the image until the supply chain is verified per git-cicd-gitops-rb-14-respond-to-compromised-dependency

6 · Escalation

When the runbook isn't enough, contact:

  • · The compromise is in a shared runner image used by many teams: this is a supply-chain attack, escalate per git-cicd-gitops-rb-14-respond-to-compromised-dependency
  • · The compromise included data exfiltration from the runner's host (database credentials, customer data): legal/PR involvement, GDPR/CCPA notification may be required. Escalate to legal
  • · The compromise is from a nation-state actor (indicators of advanced tooling, persistent presence): engage external IR, not just internal security. The compromise is likely broader than the indicator that triggered the alert
  • · The runner's host has access to production (kubectl config, AWS credentials with prod IAM): production may be compromised. Engage incident response and the application owners. Do not assume production is safe
  • · Multiple runners across the fleet show the same indicators: fleet-wide compromise, engage the platform team to take the entire fleet offline and rebuild from a known-good image

A compromised runner is the worst single-host incident in CI/CD. The runner has access to every secret the workflows it runs request; in a shared runner pool that is every secret in the platform. The response is: capture evidence, take the runner offline, remove it from the platform, audit and rotate every credential it touched, rebuild from a verified image, and add detection so the next compromise is caught earlier.

The forensic capture happens before the runner is taken offline. A runner that has been rebooted or rebuilt loses the in-memory state (network connections, process memory, recent file handles) that distinguishes compromise from misconfiguration.

1. Identify the indicator

Read-only / Safe
$ RUNNER_HOST="runner-01.example.com"
ssh "$RUNNER_HOST" '
echo "--- the indicator that triggered the alert ---"
# Outbound connection to unknown IP
ss -tnp | grep -v "127.0.0.1\|::1\|192.168.\|10.\|172.16." | head -20
# Unexpected process
ps -efww | grep -E "(curl|wget|nc|ncat|ssh|base64|python -c|perl -e)" | grep -v grep | head -20
# Modified binary
rpm -Va 2>/dev/null | grep -E "S.5" | head -20
# Secret in env
env | grep -E "(AKIA|ghp_|xox[abprs])" | head -10
# Recent files in tmp dirs
find /tmp /var/tmp /dev/shm -type f -newer /etc/hostname 2>/dev/null | head -20
'

The indicator is the hypothesis that needs verification, not the verdict. A process named python -c "import..." may be legitimate in some workflows; an outbound connection to a Tor exit node is never legitimate. Document the indicator before taking the runner offline.

2. Capture forensic evidence

Read-only / Safe
$ RUNNER_HOST="runner-01.example.com"
ssh "$RUNNER_HOST" 'sudo mkdir -p /tmp/forensic && sudo chmod 700 /tmp/forensic && sudo bash -c "
date > /tmp/forensic/timestamp.txt
ps -efww > /tmp/forensic/processes.txt
ss -tnp > /tmp/forensic/connections.txt
ss -lnp > /tmp/forensic/listeners.txt
netstat -an > /tmp/forensic/netstat.txt 2>/dev/null || true
find /tmp /var/tmp /dev/shm /var/run -type f -newer /etc/hostname > /tmp/forensic/recent-files.txt 2>/dev/null
last -n 50 > /tmp/forensic/logins.txt
history > /tmp/forensic/history-root.txt 2>/dev/null
cat ~/.bash_history > /tmp/forensic/history-runner.txt 2>/dev/null
sha256sum /usr/bin/* /usr/sbin/* /bin/* /sbin/* > /tmp/forensic/binaries.sha256 2>/dev/null
rpm -Va > /tmp/forensic/rpm-verify.txt 2>/dev/null || debsums -c > /tmp/forensic/debsums.txt 2>/dev/null
"'
echo '--- capture network for 60 seconds ---'
ssh "$RUNNER_HOST" 'sudo timeout 60 tcpdump -i any -w /tmp/forensic/capture.pcap -c 10000 2>/dev/null || true'

3. Snapshot the disk

Read-only / Safe
$ RUNNER_HOST="runner-01.example.com"
ssh "$RUNNER_HOST" '
echo "--- find the disk device ---"
lsblk
echo "--- snapshot the disk ---"
sudo dd if=/dev/nvme0n1 of=/tmp/forensic-disk.img bs=4M status=progress
echo "--- snapshot hash for chain of custody ---"
sha256sum /tmp/forensic-disk.img > /tmp/forensic-disk.sha256
'
echo '--- copy evidence off the host ---'
scp "$RUNNER_HOST:/tmp/forensic/*" /incident-drive/IR-$(date -u +%Y%m%d)/runner-01/
scp "$RUNNER_HOST:/tmp/forensic-disk.img" /incident-drive/IR-$(date -u +%Y%m%d)/runner-01/
sha256sum /incident-drive/IR-*/runner-01/forensic-disk.img | tee /incident-drive/IR-*/runner-01/forensic-disk.sha256

The disk snapshot is the durable evidence. The forensic directory is for quick triage; the disk image is for the IR review. The SHA-256 lets you prove the image was not modified after capture.

4. Take the runner offline

Read-only / Safe
$ RUNNER_HOST="runner-01.example.com"
RUNNER_ID="987654321"
ssh "$RUNNER_HOST" '
sudo systemctl stop actions.runner.*.service 2>/dev/null
sudo systemctl disable actions.runner.*.service 2>/dev/null
sudo iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
sudo iptables -A INPUT -i lo -j ACCEPT
sudo iptables -A INPUT -j DROP
sudo iptables -A OUTPUT -p icmp -j ACCEPT
sudo iptables -A OUTPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
sudo iptables -A OUTPUT -o lo -j ACCEPT
sudo iptables -A OUTPUT -j DROP
echo "host isolated - only loopback and established connections allowed"
'
gh api -X DELETE /repos/REPLACE_WITH_ORG/REPLACE_WITH_REPO/actions/runners/"$RUNNER_ID"

The iptables isolation prevents the runner from continuing to talk to the attacker while preserving existing connections for forensic capture. The platform-side deletion ensures the runner cannot re-register or claim jobs even if the host comes back.

5. Audit the secrets the runner had access to

Read-only / Safe
$ RUNNER_NAME="runner-01"
echo '--- step 1: enumerate run ids (runs carry no runner attribution) ---'
gh api /repos/REPLACE_WITH_ORG/REPLACE_WITH_REPO/actions/runs --paginate --jq ".workflow_runs[].id" > /tmp/all-runs.txt
echo '--- step 2: keep the runs whose jobs ran on this runner ---'
: > /tmp/runs.txt
while IFS= read -r RUN_ID; do
MATCHED=$(gh api /repos/REPLACE_WITH_ORG/REPLACE_WITH_REPO/actions/runs/"$RUN_ID"/jobs --paginate --jq "[.jobs[] | select(.runner_name==\"$RUNNER_NAME\")] | length")
[ "$MATCHED" -gt 0 ] && echo "$RUN_ID" >> /tmp/runs.txt
done < /tmp/all-runs.txt
wc -l /tmp/runs.txt
echo '--- workflows run on this runner ---'
while IFS= read -r RUN_ID; do
gh api /repos/REPLACE_WITH_ORG/REPLACE_WITH_REPO/actions/runs/"$RUN_ID" --jq ".path"
done < /tmp/runs.txt | sort -u > /tmp/workflows.txt
cat /tmp/workflows.txt
echo '--- secrets referenced by those workflows (the API never returns secret values) ---'
cat /tmp/workflows.txt | xargs -I{} grep -hE "secrets\.[A-Z_]+" "{}" | sort -u

Runner attribution lives on job objects (jobs[].runner_name), not on workflow-run objects — hence the two-step enumeration: run ids first, then each run’s jobs filtered by runner name. The runs and jobs responses never include secret values; the workflow file names the secrets a run could request. Every workflow run on this runner had access to the secrets the workflow referenced. Treat every referenced secret as compromised.

6. Rotate every secret

Read-only / Safe
$ echo '--- rotating every secret the runner accessed ---'
while IFS= read -r SECRET_NAME; do
echo "=== $SECRET_NAME ==="
# Each rotation follows the playbook in git-cicd-gitops-rb-08
# For GitHub PATs, regenerate via the UI; for AWS keys, iam delete + create; etc.
gh secret list --repo REPLACE_WITH_ORG/REPLACE_WITH_REPO | grep -F "$SECRET_NAME" && echo 'tracked in vault'
done < <(cat /tmp/workflows.txt | xargs -I{} grep -hoE "secrets\.[A-Z_]+" "{}" | sort -u)
echo "--- completed rotation of every secret the runner had access to ---"

The rotation happens through the standard rotation playbook per secret type. The output is the audit trail: which secret, when rotated, by whom.

7. Rebuild the runner from a verified image

Read-only / Safe
$ RUNNER_HOST="runner-01.example.com"
echo '--- for containerised runners (ARC) ---'
kubectl -n actions-runner-system scale deployment/runner --replicas=0
kubectl -n actions-runner-system set image deployment/runner runner=REPLACE_WITH_VERIFIED_IMAGE@sha256:REPLACE_WITH_DIGEST
kubectl -n actions-runner-system scale deployment/runner --replicas=1
kubectl -n actions-runner-system get pods -l app=runner

echo '--- for self-hosted runners ---'
# Decommission the host
ssh "$RUNNER_HOST" 'sudo shutdown -h now'
# Provision a new host via the IaC
cd infra/runner
terraform plan -out=tfplan
terraform apply tfplan
# Re-register
NEW_HOST=$(terraform output -raw runner_host)
ssh "$NEW_HOST" 'cd /home/runner && sudo ./config.sh --url https://github.com/REPLACE_WITH_ORG/REPLACE_WITH_REPO --token REPLACE_WITH_NEW_TOKEN --labels "self-hosted,linux" --name "runner-01" && sudo ./run.sh &'

The new image must be from a verified registry, with a digest (not a tag), and the build provenance must be auditable per git-cicd-gitops-rb-15-validate-production-artifact.

8. Add detection

Read-only / Safe
$ echo '--- for hosts ---'
sudo dnf install -y ossec-hids  # or wazuh-agent, aide
sudo aide --init
sudo cp /var/lib/aide/aide.db.new.gz /var/lib/aide/aide.db.gz
sudo aide --check  # baseline
echo '--- for Kubernetes runners ---'
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm install falco falcosecurity/falco --namespace falco --create-namespace
echo '--- egress firewall logging on the runner host ---'
sudo iptables -A OUTPUT -m state --state NEW -j LOG --log-prefix "egress: " --log-level 4
echo '--- alert routing ---'
# Send the logs to the SIEM (Splunk, ELK, Datadog) for correlation

The detection must alert on the specific indicator that triggered this incident, not on generic “host compromise”. A noisy alert trains responders to ignore; a specific alert catches the next compromise.

9. Document and follow up

Read-only / Safe
$ gh issue create --repo REPLACE_WITH_ORG/REPLACE_WITH_REPO \
--title "compromised runner $(date -u +%Y-%m-%d)" \
--body "Runner: REPLACE_WITH_NAME. Indicator: <outbound connection, modified binary, etc>. Evidence: <IR-drive path>. Secrets rotated: REPLACE_WITH_LIST. New image: REPLACE_WITH_SHA. Detection added: <Falco, AIDE>. IR ticket: REPLACE_WITH_LINK." \
--label security --label runner --label incident

Verification

The compromised runner is removed from the platform. Every secret the runner had access to is rotated and the new values authenticate. The cloud provider”s audit log shows no unauthorised API calls using the rotated credentials since the rotation. A new runner, built from the verified image, is registered and online. The detection rules added in step 8 fire on the indicator that triggered this incident, verified with a synthetic test. The IR ticket has the forensic artefacts preserved in the incident-response drive.

Rollback

If evidence was not captured before the host was taken offline, the forensic value is gone; rely on the audit logs and the cloud-provider flow logs. If the rotation was incomplete, complete it per git-cicd-gitops-rb-08-rotate-git-credentials. If the rebuilt runner cannot register or claim jobs, run git-cicd-gitops-rb-10-troubleshoot-runner. If the detection rules produce too many false positives, tune them. If the IR review concludes the compromise is broader than this one runner, scope up. If the rebuilt runner is immediately compromised again, the rebuild image is the vector — stop using it until the supply chain is verified per git-cicd-gitops-rb-14-respond-to-compromised-dependency.

References

  1. OWASP — CI/CD Security Top 10 (CSTC-4: Poisoned Pipeline Execution)
  2. GitHub Docs — Hardening self-hosted runners
  3. Falco — runtime detection for Kubernetes
  4. NIST SP 800-61r3 — Incident Response
  5. MITRE ATT&CK — Container and cloud techniques