Control plane isolation — why orchestration is separate from execution

The control plane is the part of a CI system that does not execute your code. It evaluates triggers, stores secrets, holds the pipeline file as a record, and dispatches jobs. The runner is the part that does execute your code. Separating them is what lets a CI system be both programmable and safe to expose to untrusted contributors. A system in which the control plane also ran code would be one in which a malicious pull request could rewrite the pipeline file, exfiltrate the secret store, or redirect future jobs to an attacker-controlled target.

What the control plane owns

flowchart TB
    subgraph CP["Control plane (trusted, durable)"]
        WF["Workflow file\n(record of truth)"]
        SEC["Secrets store\n(encrypted at rest)"]
        ART["Artifact store\n(versioned blobs)"]
        DIS["Dispatcher\n(job queue)"]
        OIDC["OIDC token issuer\n(short-lived identities)"]
    end
    subgraph RUN["Runner (ephemeral, untrusted)"]
        J["Job allocation\n(fresh VM or agent)"]
        STEP["Steps executing\nuser code"]
    end
    CP -->|"dispatch +\ninjected secrets"| RUN
    RUN -->|"logs, artifacts,\nstatus"| CP

The control plane owns:

• The workflow file as a record. The evaluated version, the trigger history, and the run history. A workflow in Git is the source of truth for what runs; the run history is the source of truth for what ran.
• The secrets store. Encrypted at rest. Decrypted only at the moment of injection into a job. Cleartext exists only in the runner’s memory for the duration of the job.
• The artifact store. Versioned by run ID and commit SHA; downloadable by other jobs in the same workflow.
• The dispatcher. Matches incoming jobs to runners, applies label constraints, queues work when the pool is exhausted.
• The OIDC token issuer. Mints short-lived tokens the runner exchanges with cloud IAM for a session credential. The long-lived secret never leaves the control plane’s HSM.

What the separation buys

The architectural separation buys four properties:

• The control plane cannot be compromised by a runner job. The runner job is a separate process, often on a separate machine, with a separate network identity. It can read the pipeline file and the secrets it was given, but it cannot modify the workflow definition, cannot access secrets for jobs it was not allocated, and cannot reach the secret store.
• The control plane can be hardened independently. The control plane can be a SOC2 SaaS service; the runner can be a fleet of VMs owned by the engineering team. Hardening one does not require hardening the other to the same bar.
• The runner can be ephemeral. Because the control plane holds the durable state, the runner can be a fresh VM per job and discarded at the end. A runner that was compromised last week is a runner that no longer exists.
• OIDC can replace static credentials. A control plane that can mint OIDC tokens does not need to store long-lived cloud access keys. The runner receives a token, exchanges it with cloud IAM for a session, and discards it at the end.

What the separation costs

Three costs are regularly underestimated:

• Operational complexity. Two systems means two sets of credentials, two audit trails, two sets of networking rules. A team that colocates the runner and the control plane (for example, both in the same Kubernetes cluster with the same service account) collapses the boundary and loses the security properties.
• Latency for secret retrieval. Each secret that crosses boundary 1 is decrypted at job allocation time. Hundreds of secrets per job is a measurable cost. Scope secrets per job.
• Discipline at the pipeline level. The control plane cannot enforce that the runner does not log a secret it was given. The boundary protects the secret store, not the secrets that have already left it.

OIDC and short-lived tokens

The most consequential extension of the isolation model is replacing static cloud credentials with OIDC-issued tokens:

flowchart LR
    CP["Control plane\n(OIDC issuer)"] -->|"signed JWT,\naud=sts.amazonaws.com"| RUN["Runner"]
    RUN -->|"AssumeRoleWithWebIdentity"| IAM["Cloud IAM"]
    IAM -->|"session creds,\nlifetime ~1 hour"| RUN

The runner never holds the long-lived AWS access key. It holds a JWT signed by the control plane’s OIDC issuer, exchanges it with cloud IAM for a session credential whose lifetime matches the job, and discards everything at the end. A compromised runner can use the session credential for at most an hour. A leaked session credential is invalid by the time it is read.

# Example: GitHub Actions OIDC for AWS
permissions:
  id-token: write
  contents: read
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/github-actions-deploy
          aws-region: us-east-1

No static AWS access key appears anywhere. The runner never holds a credential the cloud IAM trusts; it holds a proof that the control plane dispatched the job.

Production discipline

1. Treat the control plane as a separate system with its own audit trail. Runner logs are inputs to the run history.
2. Replace static cloud credentials with OIDC. The long-lived access key is a legacy artefact.
3. Scope secrets at the environment or job level, not the workflow level. Workflow-level secrets cross into every job, including jobs that process untrusted input.
4. Audit which secrets cross boundary 1 to which jobs. A secret that crosses to a fork-PR job has crossed into the attacker’s trust zone.

Cross-course references

• Linux for Production Sysadmins - Part XXXIV (ConfigMgmt) treats apt/dnf repository signing as a control plane isolation mechanism.
• Ansible for Production Sysadmins - Part XXXVII (RepoArch) applies the same isolation to molecule runners and AWX controllers.
• Terraform for Production Sysadmins - Parts IX-XII (State) treat Terraform Cloud’s run environment as the control plane.

Quiz