The runner incident arrives — detection, initial scope, first ten minutes

A runner compromise is a supply-chain incident, not a host incident. The runner is the boundary between untrusted pipeline input and the trusted credentials, network paths, and container registries your organisation uses. A compromised runner has read access to every secret injected during a job, write access to every artifact it produces, and the ability to reach every service the runner’s network policy permits. The first ten minutes are spent on scope, not on isolation. The questions answered in those ten minutes determine every subsequent decision.

How the alert reaches the team

The four channels are not equivalent. Each carries different evidence, different urgency, and a different starting assumption:

flowchart LR
    A["runner compromise event"] --> B["egress IDS alert"]
    A --> C["host IDS or auditd alert"]
    A --> D["runner log anomaly"]
    A --> E["vendor abuse signal"]
    B --> F["on-call paged"]
    C --> F
    D --> F
    E --> F
    F --> G["scope question set"]
    G --> H["incident commander assigned"]
    H --> I["response channel opened"]

• Egress IDS alert. The runner made a network connection to an IP or domain outside the allowed egress list. Urgency is high; an attacker is provably active.
• Host IDS or auditd alert. A file integrity monitor or audit rule on the runner host recorded a suspicious process, file modification, or ptrace syscall. Urgency is highest; the attacker is operating on the host.
• Runner log anomaly. The runner’s own logs show a workflow step that does not match the workflow file, a checkout of a branch that was never requested, or a job duration that exceeds the expected window by an order of magnitude.
• Vendor abuse signal. The cloud provider or registry observed anomalous API calls attributable to the runner’s identity.

The four scope questions

Before any isolation command, the on-call answers:

1. Which runner. Self-hosted GitHub Actions runner, GitLab Runner, Jenkins agent, or Kubernetes runner pod. Determines the isolation command set.
2. Which jobs ran. The list of workflow runs the runner picked up in the compromise window. Determines which artifacts are suspect.
3. Which credentials. The set of secrets the runner had access to: GITHUB_TOKEN, cloud access keys, registry tokens, SSH deploy keys, Vault tokens. Determines the rotation scope.
4. Which artifacts. The set of artifacts, container images, Terraform plans, and SBOMs produced by the runner in the window. Determines the rebuild scope.

Opening the response

Three artefacts are produced before isolation: a dedicated response channel, an incident commander who owns the sequence, and a timestamped timeline that becomes the artefact the security team audits.

Production discipline

1. Log everything. A timestamped entry for every action before the action is taken.
2. One commander. A single decision-maker who sequences the response.
3. No isolation before scope. Stop, drain, and snapshot commands are deferred until the four scope questions are answered.

Cross-course references

• Linux for Production Sysadmins - Part XXIX (HostIDS) covers the auditd and AIDE patterns that surface the host IDS channel.
• Kubernetes for Production Sysadmins - Part XXI (PodSecurity) covers the Pod Security Standards that constrain what a runner pod can do once isolated.
• Terraform for Production Sysadmins - Part XV (CredentialRotation) covers the rotation patterns for Terraform-managed cloud credentials the runner held.

Quiz