Prevent recurrence — ephemeral runners, network isolation, attestation per job

The prevention controls for a runner compromise are layered: ephemeral lifetime, network isolation, and per-job attestation. Each layer catches a compromise path the previous layer did not address. A team that implements one layer is not safe; a team that implements all three has raised the cost of a successful compromise above the attacker’s expected return.

The three prevention layers

A persistent runner is a runner the attacker can dwell in. A network-permissive runner is a runner the attacker can pivot from. A non-attested runner is a runner whose artifacts the team cannot trace. Each layer addresses a different failure mode:

flowchart LR
    A["compromise prevention"] --> B["ephemeral lifetime"]
    A --> C["network isolation"]
    A --> D["per-job attestation"]
    B --> E["no persistence"]
    C --> F["no pivot"]
    D --> G["no untraceable artifact"]

• Ephemeral lifetime. The runner is destroyed after every job. A compromise that lasts the duration of one job expires with the runner.
• Network isolation. The runner can only reach the endpoints the workflow needs. A compromise cannot reach the cloud metadata service, the internal admin API, or the internet at large.
• Per-job attestation. Every artifact is signed by a job-scoped key, and the platform records which runner produced which artifact. A compromise that produces an untraceable artifact is detectable.

Ephemeral runners

An ephemeral runner is a runner that is started for one job and destroyed when the job ends. The attacker has the runtime of one job — minutes, not days — to do damage. The persistent state the attacker would otherwise have written to the runner’s disk is destroyed with the runner.

For Kubernetes runners, ephemerality is implemented by using a Job resource with restartPolicy: Never and a ttlSecondsAfterFinished of zero:

apiVersion: batch/v1
kind: Job
metadata:
  name: runner-job
spec:
  ttlSecondsAfterFinished: 0
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: runner
          image: runner-image:latest

Network isolation

A network-isolated runner can only reach the endpoints the workflow needs. The network policy is the gate:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: runner-egress-allowlist
  namespace: ci-runners
spec:
  podSelector:
    matchLabels:
      role: ci-runner
  policyTypes:
    - Egress
  egress:
    - to:
        - ipBlock:
            cidr: 10.0.0.0/16
            except:
              - 10.0.0.1/32
      ports:
        - protocol: TCP
          port: 443
    - to:
        - namespaceSelector:
            matchLabels:
              name: artifact-registry

Three rules are non-negotiable:

1. Egress allowlist. The runner cannot reach any IP or domain not on the allowlist. The metadata service (169.254.169.254) is explicitly excluded.
2. No inbound. The runner accepts no inbound connections. The build does not need to be reached from outside the namespace.
3. No metadata service. The runner cannot reach the cloud metadata service to assume the node’s IAM role.

Per-job attestation

Every artifact produced by the runner is signed by a job-scoped key, and the platform records which runner produced which artifact. The attestation is the audit trail the rebuild verification gates from lesson 05 require.

For GitHub Actions, the attestation is implemented through the built-in artifact attestation API and the Sigstore cosign keyless signing flow:

- name: Attest artifact
  uses: actions/attest-build-provenance@v1
  with:
    subject-path: '${ env.IMAGE_REF }'

For self-hosted runners, the signing key is rotated per job using cosign keyless signing, which uses the OIDC token issued by the platform. The token is job-scoped: the keypair used to sign the artifact is unique to that job, and the public key is recorded in the Sigstore transparency log.

Operational discipline

1. Ephemeral by default. Persistent runners are an exception justified by workflow class, not the default.
2. Network policy as code. The egress allowlist is checked into version control and reviewed on every change.
3. Attestation per job. The signing key is scoped to the job, not to the runner.

Cross-course references

• Linux for Production Sysadmins - Part XXXII (ContainerImages) covers the registry-side validation patterns that per-job attestation requires.
• Kubernetes for Production Sysadmins - Part XXI (PodSecurity) covers the NetworkPolicy patterns that network isolation builds on.
• Terraform for Production Sysadmins - Part XV (CredentialRotation) covers the rotation patterns that apply to the cosign signing keys.

Quiz