Isolate the runner — remove from pool, drain new jobs, preserve evidence

Once scope is established, the isolation sequence is drain, snapshot, stop, deregister — in that order. Each step builds on the previous. The drain prevents new jobs from landing on a compromised runner. The snapshot captures the forensic artefact the security team will audit. The stop kills the attacker’s current access. The deregister removes the runner from the pool so it cannot re-join.

The isolation sequence

The sequence is forced by the asymmetry between what the attacker has and what the team needs:

flowchart LR
    A["compromised runner"] --> B["drain no new jobs"]
    B --> C["snapshot forensic state"]
    C --> D["stop runner process"]
    D --> E["deregister from pool"]
    E --> F["evidence handed to forensics"]

• Drain first. The runner must stop picking up new jobs before any other action. A runner that is still in the pool will be assigned new jobs every few seconds; the attacker gains new credentials and new artifacts with every job.
• Snapshot second. Capture the process tree, open sockets, file mtimes, audit log, and (where feasible) a memory image. The snapshot is the artefact the security team audits.
• Stop third. Stop the runner process or delete the pod. This kills the attacker’s current access without destroying the snapshot.
• Deregister fourth. Remove the runner from the pool so it cannot re-join after a restart. A stopped runner that is still registered will be restarted by the runner supervisor and re-pick-up jobs.

Self-hosted GitHub Actions runner

The drain step on a self-hosted runner is to remove the runner label from the workflow runs-on references that target it. The snapshot step captures the runner state. The stop step uses ./svc.sh stop. The deregister step uses ./config.sh remove --token $TOKEN.

# Stop the runner service without deregistering it
sudo ./svc.sh stop

# Capture the forensic artefact while the runner is alive
sudo ps -ef > runner-process-tree-$(date -u +%FT%TZ).txt
sudo ss -tulpn > runner-sockets-$(date -u +%FT%TZ).txt
sudo find / -newer /tmp/marker -xdev \
  > runner-modified-files-$(date -u +%FT%TZ).txt

# Deregister the runner from GitHub using the token printed
# at registration time, stored in the runner's .runner file
sudo ./config.sh remove --token "$RUNNER_TOKEN"

Kubernetes runner pod

The drain step on a Kubernetes runner is to label the pod so the runner controller no longer schedules jobs to it. The snapshot step captures the pod’s logs and the underlying node state. The stop step is kubectl delete pod. The deregister step is the runner controller’s garbage collection on the deleted pod.

# Label the pod so the runner controller drains it
kubectl label pod "$RUNNER_POD" runner.ci/drain=true \
  --namespace "$NAMESPACE"

# Capture the forensic artefact while the pod is alive
kubectl logs "$RUNNER_POD" --namespace "$NAMESPACE" \
  > runner-pod-logs-$(date -u +%FT%TZ).txt
kubectl describe pod "$RUNNER_POD" --namespace "$NAMESPACE" \
  > runner-pod-describe-$(date -u +%FT%TZ).txt

# Delete the pod; the runner controller will deregister it
kubectl delete pod "$RUNNER_POD" --namespace "$NAMESPACE"

GitLab Runner

The drain step is to set the runner to paused in the runner configuration. The snapshot step captures the runner state. The stop step is gitlab-runner stop. The deregister step is gitlab-runner unregister.

# Pause the runner so no new jobs are picked up
sudo gitlab-runner pause

# Capture the forensic artefact
sudo ps -ef > runner-process-tree-$(date -u +%FT%TZ).txt
sudo ss -tulpn > runner-sockets-$(date -u +%FT%TZ).txt

# Stop and deregister
sudo gitlab-runner stop
sudo gitlab-runner unregister --token "$RUNNER_TOKEN"

The snapshot artefact

The snapshot has five fields: process tree, open sockets, file mtimes, audit log, and (for Kubernetes pods) the pod description. Each field answers a question the forensic team will ask:

Production discipline

1. Drain before snapshot. A runner still picking up jobs is a runner the attacker is still using.
2. Snapshot before stop. The snapshot is the artefact.
3. Deregister after stop. A stopped-but-registered runner is a runner that will be restarted.
4. Copy the snapshot off-runner. The on-runner copy is evidence the attacker can tamper with.

Cross-course references

• Linux for Production Sysadmins - Part XXIX (HostIDS) covers the auditd and process accounting patterns that produce the audit log field.
• Kubernetes for Production Sysadmins - Part XXI (PodSecurity) covers the namespace patterns that constrain the blast radius of a compromised runner pod.
• Ansible for Production Sysadmins - Part XXXVIII (IncidentRunbooks) covers the runbook structure that the drain-snapshot-stop-deregister sequence extends.

Quiz