CI/CD architecture and runner security — recap

A CI/CD system looks like a single product but is in fact two planes: a control plane that decides what should run, and a data plane that actually runs it. The control plane holds the secrets, the workflow definitions, and the identity of every job. The data plane holds the runners that execute the workflows and the registries that store the artefacts. Confusing the two — running a job on the control plane, mounting the secrets store on the data plane — is the proximate cause of most “the CI was the breach” incidents.

The three-plane model

flowchart TB
    subgraph CP["Control plane"]
        CF[Forge: GitHub / GitLab]
        ORC[Orchestrator: Actions / GitLab CI]
        SS[Secrets store]
    end
    subgraph DP["Data plane"]
        R1[Hosted runner]
        R2[Self-hosted runner]
        R3[Ephemeral runner]
        REG[Artifact registry]
    end
    subgraph T["Targets"]
        T1[Cloud account]
        T2[Kubernetes cluster]
        T3[Terraform state]
    end
    CF -->|workflow yaml| ORC
    ORC -->|job payload| R1
    ORC -->|job payload| R2
    ORC -->|job payload| R3
    SS -->|short-lived secret| R1
    SS -->|short-lived secret| R2
    SS -->|short-lived secret| R3
    R1 --> REG
    R2 --> REG
    R3 --> REG
    R1 -->|OIDC token| T1
    R2 -->|OIDC token| T2
    R3 -->|OIDC token| T3

• Control plane. The forge (GitHub, GitLab), the orchestrator (the Actions or GitLab CI service), and the secrets store. Should be reachable from the public internet (engineers push from anywhere) but isolated from the data plane’s internal networks.
• Data plane. The runners (hosted, self-hosted, ephemeral) and the registries that hold the artifacts produced by the runners. Talks outbound to the control plane to fetch jobs and outbound to the deploy targets using credentials issued for the specific job.
• Targets. Cloud accounts, Kubernetes clusters, Terraform state buckets. Reachable from runners over the public internet or over a peering connection, with per-job, short-lived credentials only.

Runner topology

The four runner topologies, in order of blast radius:

• Hosted runners. Provided by the forge. Ephemeral per-job. No persistent state between jobs. Low blast radius because the runner is destroyed at the end of the job, but the network egress is shared with every other customer of the forge.
• Self-hosted runners. Persistent. Carry state between jobs unless explicitly cleaned. The classic foothold for lateral movement if one job is malicious.
• Ephemeral runners. Self-hosted but torn down at the end of every job. The default for any workload that handles production credentials or production deploy targets.
• Autoscaled runners. Ephemeral runners launched on demand by a controller (Actions Runner Controller, GitLab Runner autoscaler). The autoscaler decides how many runners to run; the runners themselves are destroyed when the queue empties. The production-grade default.

The three non-negotiable runner controls

The runner threat model collapses to three controls that must be in place for every runner that touches production:

1. No privileged mode without justification. A privileged container can do almost anything the host can do, including mounting the host filesystem and breaking out of its namespace. Privileged mode is sometimes required (Docker-in-Docker without rootless alternatives) but it must be justified per-job, not enabled at the runner level.
2. No mounted Docker socket. Mounting /var/run/docker.sock into a build container is equivalent to giving the container root on the host: any process inside the container can spawn a sibling container with no isolation. The only safe alternative is rootless Docker, BuildKit in rootless mode, or Kaniko in an unprivileged container.
3. Short-lived OIDC credentials, not long-lived secrets. A long-lived cloud access key in a CI secret is a key that, once leaked, works from any IP on the planet for as long as the key is valid. An OIDC token issued per-job, scoped to a single workflow, and expiring in an hour, is a token that is worthless the moment the job ends.

Production discipline

The five rules that recur across every runner configuration:

1. Runners are cattle, not pets. Ephemeral and autoscaled by default. A long-lived self-hosted runner is a foothold waiting to be taken.
2. Egress is the new perimeter. Hosted runners share egress IPs with every other customer. A workload that depends on IP allowlists cannot use hosted runners for the production deploy.
3. Privileged mode is per-job, not per-runner. A runner default of privileged: false with explicit opt-in for the one job that requires it.
4. Docker socket is never mounted. Rootless, or BuildKit, or Kaniko. Every other option is a known escape path.
5. OIDC first, secrets last. A long-lived cloud secret in CI is a control failure. OIDC federation is the default; secrets are reserved for the few cases where OIDC is not available.

Cross-course references

• Linux for Production Sysadmins — Parts on container isolation and capability dropping: the same discipline that applies to a production container applies to a build container.
• Kubernetes for Production Sysadmins — The service-account token volume projection pattern is the in-cluster analogue of OIDC federation in CI.

Quiz