Runner compromise (unauthorized process detected)

Reported symptoms

●EDR (CrowdStrike, SentinelOne, Falco on the host) flags an unknown process making outbound HTTPS connections to an IP not in the allowlist
●The process tree shows `actions-runner` spawning a child that is not part of the runner documented components, or a process reading `~/.aws/credentials` and writing to a network socket
●The runner is long-lived (`actions.runner.<org>-<repo>` service registered days or weeks ago), with `ephemeral: false` in `.runner`
●The runner holds an OIDC-derived cloud credential, a long-lived cloud access key, or a kubeconfig with broad cluster access; the credential is currently valid
●Network egress logs from the runner host show `tcp/<external-ip>:443` connections from the runner UID within the last hour
●The runner `_diag/` log shows step-execution events from `jobs/build` whose steps include a `run:` block that pulled a third-party action whose digest does not match the version expected
●GitHub Actions audit log shows the runner picked up jobs for multiple repositories under the same org; the affected runner is registered at the org level
●No alert fired on the inbound supply-chain attack; the alert fired on the outbound exfiltration, by which time artifacts had already been produced

Evidence

· `ps -ef --forest | grep -A 5 actions.runner` shows a child process tree under the runner that includes `curl`, `wget`, `nc`, or a binary whose SHA-256 does not match any documented runner component
· `sudo journalctl -u actions.runner.<org>-<repo> --since "1 hour ago"` contains step logs whose `run` block executed a setup action whose `action.yml` differs from the documented version (a tag swap that landed a backdoor)
· `sudo ausearch -m EXECVE --start recent` shows the runner UID invoking a binary whose path is `/tmp/...` or `/var/tmp/...` — not the runner install path
· The EDR console shows the offending process with `image_hash: <sha256>`, `parent: actions-runner-worker`, `network_connection: <external-ip>:443`
· A previous breakfix scenario (long-lived self-hosted runner with 24-hour registration token expiry) already noted the runner is long-lived; the new evidence is the network egress
· `aws sts get-caller-identity` from the runner host returns the runner IAM role ARN; the role has `sts:GetCallerIdentity` allowed and recently issued short-lived credentials via OIDC
· GitHub Actions audit log shows the runner registered 24 days ago, last contacted less than 5 minutes ago, jobs completed in last 24 hours: 47
· Artifact attestations for the last 24 hours fail `gh attestation verify` against the `provenance` predicate, or succeed but reference a workflow whose inputs used a tag-swap
· `kubectl get pods -A --field-selector spec.serviceAccountName=<runner-sa>` shows pods deployed by Argo CD whose images were built by the affected runner; these pods are running with potentially-stolen credentials still cached on the runner

Diagnosis and resolutionclick to reveal

Root cause

A long-lived self-hosted runner is a high-value target: it holds credentials (OIDC-derived or long-lived), it executes arbitrary code from third-party actions, and it persists across jobs. A tag-swap on a third-party action (a related breakfix scenario) or a malicious composite action can land code on the runner; the code reads the runner credentials and exfiltrates them. The structural failure is that the runner is long-lived at all: an ephemeral runner that is destroyed at the end of each job cannot carry stolen credentials across jobs, and its blast radius is one job. The secondary failure is the absence of egress controls: a runner that can reach arbitrary external IPs can exfiltrate; an allowlist of registry, GitHub API, and artifact-store endpoints would have caught the egress at the network layer before the EDR did.

Remediation

Quarantine the runner immediately: `sudo systemctl stop actions.runner.<org>-<repo>`, then `sudo ./svc.sh uninstall` and remove the runner from the GitHub org via the UI or `gh api -X DELETE orgs/<org>/actions/runners/<runner-id>`. Revoke every credential the runner held: rotate the runner IAM role at the cloud provider (`aws iam delete-access-key` for long-lived, or shorten the OIDC session duration and force a re-issue), rotate the kubeconfig, rotate the registry pull token. Audit every artifact produced by the runner in the last 24 hours (or since the suspected compromise window): pull the artifacts from the registry, run `gh attestation verify` against each, and rebuild any artifact that lacks a valid provenance attestation, or whose attestation references a workflow that used a tag-pinned action. Re-deploy using the rebuilt artifacts. Then move to ephemeral runners: Actions Runner Controller on Kubernetes, where each job spins up a fresh `RunnerPod` that registers, executes one job, and is destroyed — no credentials can persist, and the blast radius is one job.

Verification

The runner is removed from the GitHub org: `gh api orgs/<org>/actions/runners/<runner-id>` returns 404. The runner host shows no `actions.runner` processes: `pgrep -fa actions.runner` returns empty. The OIDC session is expired or the IAM access key is deleted: `aws sts get-caller-identity` from the host fails. Every artifact produced by the runner in the last 24 hours has been rebuilt from the source commit, signed with a new key, and verified. New artifacts are built by ephemeral runners and each carries a valid provenance attestation. Network egress from the runner subnet is restricted to an allowlist that includes only the GitHub API, the registry, and the secret store; egress to arbitrary IPs is denied at the firewall.

Prevention

Do not run long-lived self-hosted runners in production. Ephemeral runners — Actions Runner Controller on Kubernetes, or a managed equivalent — register fresh on every job and are destroyed when the job ends. A compromised runner cannot persist credentials, cannot serve as a pivot, and the blast radius is one job. Combine ephemeral runners with: an egress allowlist on the runner subnet (only GitHub API, registry, secret store); an EDR with a custom rule for `actions-runner-worker` parent and any child not on the allowlist; a tag-swap detector on third-party actions (the same control that prevents tag-swap attacks); and a provenance attestation requirement for every deployed artifact, so a runner that lacks the signing key cannot produce a deployable artifact. The principle is that a long-lived runner is a long-lived credential, and long-lived credentials in arbitrary code execution are an incident waiting to be scheduled.

A long-lived runner is a long-lived credential that executes arbitrary code. A tag-swapped action or a malicious composite can turn it into an exfiltration endpoint in one job. Ephemeral runners reduce the blast radius to one job, and an egress allowlist prevents the exfiltration from succeeding even if the runner is compromised. The structural fix is the runner lifecycle, not the next EDR rule.