Reference architecture: delivery-pipeline identity

A delivery pipeline executes code written by people who are not administrators, on machines that hold authority over production. No other component in an estate combines those two properties. The design goal follows directly and is unusually easy to state: make the pipeline’s credential store worth nothing to whoever gets into it.

The most privileged untrusted actor

Two decades of tooling has trained teams to think of a build system as infrastructure. It is better understood as a public entry point that happens to run inside the perimeter. Three of its behaviours are routinely misread, and each has produced its own genre of incident.

The first is the event model. Code from a fork, evaluated in the usual pull request event, has no access to the repository’s secrets, which is the property most people remember. The variant event designed to give fork contributions access to the base repository behaves in the opposite way: it runs with read and write permissions and with secret access, even for a public fork. Two events with nearly identical names sit on opposite sides of the estate’s most important trust boundary.

The second is log redaction. Masking is a best-effort filter, not a guarantee, and its documented gaps are exactly the cases that occur in practice. A secret embedded in structured output defeats exact-match redaction. Values derived from a secret, such as an encoded or re-encoded form, are not masked unless they are registered separately. Only secrets used by the current job are redacted at all, and registering a mask does not apply retroactively to output already emitted. A design that depends on masking has chosen a control whose failure is silent.

The third is what a build leaves behind. A value supplied as a build argument persists in the image history and in provenance metadata, and deleting the file in a later step does not help, because the removal is recorded as a marker in an upper layer while the lower layer still carries the original bytes. The mechanism that actually works mounts the value for one step and never writes it into a layer:

# Mounted for this step only, and never written into a layer.
# required=true fails the build when the secret is absent; the
# default is false, which builds quietly without it.
RUN --mount=type=secret,id=npm_token,target=/tmp/npm_token,required=true \
    NPM_TOKEN="$(cat /tmp/npm_token)" npm ci

Infrastructure code has an equivalent. Rather than relying on values being redacted from output, current practice keeps them out of durable storage in the first place, using ephemeral values and ephemeral resources for data that exists only during an operation, and write-only arguments for values that are sent to a provider and never read back.

Federated identity instead of a stored key

The architecture that removes the problem does not protect the stored credential better. It removes the stored credential.

flowchart LR
    JOB["CI job\ncontributed code"]
    FORGE["CI platform\nidentity issuer"]
    JWKS["Published signing keys"]
    CLOUD["Cloud or secret manager\ntrust condition"]
    CRED["Temporary credential\nbounded by the job"]
    TGT["Target system"]
    JOB -->|"request a token for this job"| FORGE
    FORGE -->|"signed claims: repo, ref, environment"| JOB
    JOB -->|"present the assertion"| CLOUD
    CLOUD -->|"verify signature"| JWKS
    CLOUD -->|"match audience and subject"| CRED
    CRED --> JOB
    JOB -->|"acts, briefly"| TGT

The job asks its own platform for a token describing the work it is doing. That request has to be enabled deliberately, because a job that never needs an identity should not be able to mint one:

permissions:
  id-token: write
  contents: read

The relying party then does three things in order: fetch the issuer’s published signing keys and verify the signature, check the standard claims, and evaluate the subject against a condition you wrote. The whole security of the design lives in that last step, and it is where implementations most often fail. A condition that names only the repository is satisfied by every branch in it, every workflow file in it, and every pull request opened against it, which means anyone who can open a branch can assume the production role. The condition must pin the audience and a subject specific enough to name the environment, the reference, or the workflow that is entitled to it.

The controls that make it safe

• Keep the credential store empty of anything durable. If the store still holds a cloud key, the federation added a mechanism without removing a risk.
• Separate the identity that reads from the identity that writes. Planning a change and applying it have different blast radii, so give them different subjects and gate the second behind an environment that requires a human approval.
• Bound the credential by the job, not by a clock. The right lifetime is the shortest one the work fits into, and the exchange is cheap enough to repeat.
• Design for leaked logs. Assume every value printed will be read by someone. If what leaks has already expired, the leak is an embarrassment rather than an incident.
• Audit at the target, not at the pipeline. The pipeline is the component being constrained, so its own record of its behaviour is the least trustworthy evidence available. The cloud trail is the independent channel.
• Sign what the pipeline produces. A signature on the artefact lets a consumer verify the output later without having to trust the pipeline’s account of itself.

The six questions, answered

Part I asked six questions of every credential: which principal is being trusted, which credential proves that identity, who issued it and who checks the issuer, where the private material lives, how long it lives, and what revokes it and what enforces that. The three reference architectures in this part exist so that every one of those questions has a mechanism behind it. Run them against two credentials that could hardly be less alike.

The sixth row is the one worth defending, because it is the row where this course refuses to tell a comforting story. For a publicly trusted TLS key the honest position is that revocation is weakly enforced. Mainstream browsers do not perform online status checks by default, one large authority has switched its responder off entirely, and even a positive status response does not by itself establish that the certificate was ever issued or is currently within its validity window. What genuinely contains a compromised web-facing key is a short lifetime plus a reissuance path you have already rehearsed. In a private PKI you can do better, but only by naming the component that fetches the list and refuses when it cannot.

The pipeline identity answers the same question in a way that barely needs the word revocation, and that is the point of the architecture rather than a lucky property of the technology. The credential expires before anyone can queue up a use for it, and withdrawal is a condition you delete rather than a copy you chase.

Production discipline

1. Delete the stored key on the day federation goes live. A federated path that runs alongside a long-lived key has added a mechanism and removed nothing.
2. Review the trust condition like production code. It is the only part of this design that can be wrong, and it fails open.
3. Give the pipeline the smallest target-side authority that works. Federation controls who may ask; it says nothing about what they get, and that is a separate design decision at the target.
4. Re-run the six questions whenever a component changes. A new runner pool, a new cluster or a new managed service adds a credential, and an unexamined credential is an unbounded one.

Cross-course references

• Git, CI/CD & GitOps for Infrastructure Engineers - Part XLIII (OIDC) covers configuring the federated exchange end to end and the claim structure that the trust condition matches against.
• Docker & Containers for Production Sysadmins - Part XXXVI (Supply Chain) covers provenance and artefact signing, the controls that let a consumer verify a pipeline’s output without trusting the pipeline.
• Terraform for Production Sysadmins - Part XIX (Security: Credentials, Secrets and Audit) covers the credential the apply step runs with, which is the target-side half of this architecture.

Quiz