The runner and its environment — hosted, self-hosted, ephemeral, persistent

The runner is the trust boundary of a CI system. The pipeline file is plain text; the commit is untrusted until proven otherwise; the secrets the job needs (cloud credentials, signing keys, deploy tokens) are sensitive. The runner is the process that turns those four things - code, secrets, execution, and output - into a result. Every choice about the runner is a security choice, because every byte the runner can read is a byte the next job can read on a persistent host, and every network the runner can reach is a network the untrusted code can reach.

Hosted vs self-hosted

flowchart TB
    subgraph H["Hosted runners"]
        H1["Forge owns the host"]
        H2["Ephemeral by default"]
        H3["Internet egress only"]
    end
    subgraph S["Self-hosted runners"]
        S1["You own the host"]
        S2["May be persistent"]
        S3["Can reach private network"]
    end

A hosted runner is a machine the forge operates. GitHub Actions runners, GitLab.com shared runners, and Jenkins controller nodes in cloud-managed setups are examples. The forge patches the host, owns the lifecycle, and tears the machine down after the job. The secrets the runner can read are the secrets the forge knows about; the network the runner can reach is the public internet (and any forge-managed egress allow-list).

A self-hosted runner is a machine you operate. You patch it, you size it, you connect it to your private network. The secrets you place on it are the secrets the runner can read; the network it can reach is the network you have given the host.

The two are not interchangeable:

gh run list --workflow=build.yml --limit 5
gh run watch 1234567890 --exit-status

gh run list and gh run watch operate against the run record regardless of which runner class executed it; the runner identity is logged in the run metadata.

Ephemeral vs persistent

flowchart LR
    E["Ephemeral runner\n(clean per job)"] --> J1["Job 1\n(fresh state)"]
    J1 --> J2["Job 2\n(fresh state)"]
    J2 --> J3["Job 3\n(fresh state)"]
    P["Persistent runner\n(state across jobs)"] --> K1["Job 1\n(left state behind)"]
    K1 --> K2["Job 2\n(reads prior state)"]

An ephemeral runner is destroyed after the job. The next job starts on a clean host. There is no shared state between runs: no leftover files, no stale caches, no lingering processes. The default for hosted runners is ephemeral.

A persistent runner stays up. The next job starts on the same host, with the same filesystem, the same package cache, the same Docker images. Caches are warm; builds are faster; but state from prior jobs is present until cleaned.

The security difference is structural. A persistent runner that ran a malicious job is a host the next job inherits. If the malicious job wrote a credential to disk, modified ~/.bashrc, planted a cron entry, or replaced terraform, the next job runs on a compromised host. Ephemeral runners remove this class of attack by definition.

The runner as the trust boundary

flowchart LR
    U["Untrusted commit"] -->|"runs on"| R["Runner\n(trust boundary)"]
    S["Secrets\n(cloud creds, signing keys)"] -->|"read by"| R
    R -->|"writes"| OUT["Artefact / status"]
    R -.->|"has network access to"| N["Internal network"]

The runner is the trust boundary because it is the place where four things meet:

1. Untrusted code. The commit on the runner’s filesystem is, by definition, code that has not been approved. A script: step is a bash -c against untrusted bytes.
2. Trusted secrets. The runner’s environment or secret store contains credentials that, if leaked, become an incident.
3. Trusted network. The runner can often reach internal systems - a private package registry, an internal API, a cloud control plane via a VPC endpoint.
4. Trusted output. The artefact the runner produces will be deployed, signed, or published.

Every control on the runner is a control on the boundary. The network allow-list, the secret scoping, the filesystem isolation, the execution user, the cleanup hooks - each is a control that limits what untrusted code can reach.

Choosing the runner class

flowchart TB
    Q["What does this job need?"]
    Q -->|public internet only| A["Hosted, ephemeral\n(default)"]
    Q -->|private network| B["Self-hosted, ephemeral\n(strict isolation)"]
    Q -->|GPU / large memory| C["Self-hosted, specialised\n(ephemeral preferred)"]
    Q -->|untrusted PR + secrets| D["Two-runner design:\nbuild on hosted,\ndeploy on isolated"]

A practical rule:

• Build and test on hosted, ephemeral. The runner is clean, the network is the public internet, and the secrets are forge-managed.
• Deploy on isolated, ephemeral runners. Whether hosted (GitHub Actions larger runners, GitLab CI/CD on Kubernetes) or self-hosted (with strict isolation), the deploy job must not share state with the build job.
• Never run untrusted PRs on the same persistent host that reads production secrets. The two operations must be on different runners, preferably different runner pools, with no shared filesystem or credentials.

Production discipline

1. Default to hosted and ephemeral. Reach for self-hosted only when the runner must reach a private network or use specialised hardware.
2. Persistent runners are a deploy-time artifact, not a build-time artifact. Caches belong in a cache store (actions/cache, S3, GCS), not on the runner’s filesystem.
3. The runner that ran an untrusted PR is not the runner that deploys to production. The trust boundary is the runner pool, not the pipeline file.

Cross-course references

• Linux for Production Sysadmins - Part XII (RepoSecurity) covers the same trust boundary at the OS level: the package build host is the trust boundary between untrusted upstream sources and the signed package in your repo.
• Ansible for Production Sysadmins - Part XXXVII (RepoArch) treats the runner as the trust boundary for playbook CI; the runner reads vault secrets, executes the playbook, and must be ephemeral.
• Terraform for Production Sysadmins - Parts IX-XII (State) cover the runner as the trust boundary for terraform apply
  • the runner holds the cloud credentials and must be isolated.

Quiz