Persistence and lateral movement — what happens after the initial compromise; how to contain

The initial compromise of a runner is the beginning of the incident, not the end. The attacker persists — they want to be there tomorrow — and they pivot — they want to be somewhere else today. Persistence is how the attacker survives the job ending. Lateral movement is how the attacker reaches production from the runner. The containment procedure is how the team closes the door in the first hour. The structural changes are how the team makes the same door harder to walk through next time.

Persistence: surviving the job ending

A compromised job ends. The attacker needs a way back. The mechanisms fall into five categories:

flowchart TB
    subgraph PERS["Persistence mechanisms"]
        P1["Cron / systemd timer"]
        P2["SSH keys"]
        P3["Modified binaries"]
        P4["IAM role assumption"]
        P5["Backdoored action / dependency"]
    end
    P1 --> H["Compromised host"]
    P2 --> H
    P3 --> H
    P4 --> H
    P5 --> H

• Cron and systemd timers. A cron.d entry or systemd timer installed in /etc/cron.d/ survives container destroy only if the host’s filesystem is persistent.
• SSH keys. The attacker writes a public key to /root/.ssh/authorized_keys. The attacker can now SSH to the host from anywhere with the matching private key.
• Modified binaries. The attacker replaces /usr/local/bin/terraform (or kubectl, aws) with a wrapper that exfiltrates arguments on every invocation.
• IAM role assumption. If the runner has an IAM role, the attacker steals the role’s session and assumes it from their own host.
• Backdoored action or dependency. The attacker replaces a third-party action’s tag with a backdoored version, or adds a postinstall hook to a dependency.

The five mechanisms share one property: they outlast the job that installed them. A persistent host is the amplifier; ephemeral hosts eliminate the host-level mechanisms by tearing the host down.

Lateral movement: reaching production from the runner

The runner is the launchpad. Production is the target. The attacker pivots through three paths:

flowchart LR
    R["Compromised runner"] --> CLOUD["Cloud API\n(AWS, GCP, Azure)"]
    R --> CLUSTER["Kubernetes cluster\n(kubectl, helm)"]
    R --> INTERNAL["Internal network\n(HTTP, SSH, databases)"]
    CLOUD --> PROD["Production"]
    CLUSTER --> PROD
    INTERNAL --> PROD

• Cloud API. The attacker has either a long-lived IAM key (XLI-03) or an STS token from an assumed role. The attacker calls aws sts get-caller-identity from their own host to verify the token, then enumerates via aws iam list-users, aws ec2 describe-instances.
• Kubernetes cluster. The attacker reads ~/.kube/config or assumes a service account with cluster access. The attacker runs kubectl get nodes, kubectl get secrets --all-namespaces, kubectl exec.
• Internal network. The runner can reach an internal network that the public internet cannot. The attacker uses the runner as a jump host: ssh user@internal-host, curl http://internal-api/, psql -h internal-db ....

The three paths share one property: the attacker’s reach extends beyond the runner. The containment that stops at the runner is incomplete; the production systems are still compromised.

Containment: the first hour

The containment procedure is four steps, in order:

flowchart LR
    I["Isolate\n(disconnect)"] --> A["Audit\n(what did attacker do)"]
    A --> RV["Revoke\n(rotate creds)"]
    RV --> RB["Rebuild\n(from clean image)"]

• Step 1 — isolate. Disconnect the runner from the network. On cloud-managed runners, terminate; on self-hosted, shut down or change the security group to deny all egress.
• Step 2 — audit. Identify what the attacker did. Cloud-side logs (CloudTrail, Azure Activity Log) record every API call with stolen credentials; cluster-side logs (Kubernetes audit log) record every kubectl-equivalent call.
• Step 3 — revoke. Rotate every credential the attacker could have reached: IAM access keys, OIDC trust policies, service account tokens, database credentials.
• Step 4 — rebuild. The runner is rebuilt from a known-clean image. On self-hosted, the host is reimaged and persistent state discarded.

The four steps are sequential. Skipping step 1 gives the attacker time to exfiltrate more; skipping step 3 leaves credentials valid; skipping step 4 leaves persistence in place.

Structural changes that prevent recurrence

The first incident is the cost of learning. The second is the cost of failing to learn. The structural changes that close the door permanently:

flowchart TB
    subgraph CHG["Structural changes"]
        C1["Ephemeral runners"]
        C2["OIDC + short-lived creds"]
        C3["Scoped per-workflow creds"]
        C4["Network egress allowlist"]
        C5["Cluster via separate identity"]
    end
    C1 --> P["Persistence + lateral movement eliminated"]
    C2 --> P
    C3 --> P
    C4 --> P
    C5 --> P

• Ephemeral runners. Destroyed after each job. Cron, systemd, modified binaries, attacker-written SSH keys do not survive.
• OIDC and short-lived credentials. Static IAM keys are removed; every cloud API call uses an STS token issued per job.
• Scoped credentials per workflow. Each workflow declares the role it needs; the IAM trust policy enforces the declaration.
• Network egress allowlist. The runner reaches only the forge, the package registry, and the cloud APIs the workflow declares.
• Cluster access via a separate identity. Cluster access is granted per-job via an OIDC-issued service account token; RBAC is scoped to the workflow’s declared namespaces.

The five changes together address both halves of the incident: persistence (ephemeral) and lateral movement (OIDC, scoped creds, egress allowlist, separate cluster identity).

Production discipline

1. Isolate first, audit second, revoke third, rebuild fourth. The order is non-negotiable.
2. Ephemeral runners for untrusted code. No exceptions.
3. OIDC + scoped credentials for everything else. No static keys, no broad IAM roles, no shared service accounts.
4. Network egress allowlist for every runner. Internal destinations are not on the allowlist.
5. Quarterly purple-team exercise. A red team attempts to persist and pivot from a runner; a blue team contains. The exercise validates the controls.

Cross-course references

• Git, CI/CD & GitOps — Part XL-03 (Ephemeral runners) covers the runner class that eliminates persistence.
• Git, CI/CD & GitOps — Part XXXIV-05 (The compromised account) covers the credential-side incident response.
• Kubernetes for Production Sysadmins — Part LVI (DefenseInDepth) covers the cluster-side controls that limit lateral movement.

Quiz