CI/CD Pipeline Locked Out: Credentials Not Authorised

Reported symptoms

The Thursday release is blocked. The pipeline for acme/platform-infra runs plan on the pull request and apply on merge to main; the plan job went green at 18:41 and the apply job failed at 18:52, four seconds after it started, before a single Terraform command had run.

The error is an authorisation failure on the role assumption. The release engineer reads it as “our credentials expired” and escalates to the cloud team, who read it as “your pipeline lost its permissions” and escalate back.

What was checked and ruled out in the first hour:

• Not a permissions gap on the role. Nothing in the failure names an API action against the account. The failure happens while acquiring credentials, not while using them.
• Not an IAM change. The audit log shows no update to the role’s trust policy or its attached policies in ninety days.
• Not the role ARN. The plan job and the apply job read it from the same repository variable, and the plan job worked with it eleven minutes earlier.
• Not the runner or the region. Both jobs run on the same runner image in the same region.
• Not an outage. Staging applies normally from the same workflow file, twice, while the investigation is going on.

Late in the evening a second data point arrives and is initially treated as good news: acme/payments-infra ran its production apply that night and it succeeded. The two repositories share the workflow template, so the team concludes the problem must be local to platform-infra.

The only change merged into either repository that day is a four-line workflow edit, reviewed and approved in ninety seconds, whose pull request title ends “workflow only, no infra change”.

Evidence provided

$ gh run view 1908442 --log-failed | head -8

Run aws-actions/configure-aws-credentials@v4
Assuming role with OIDC
Error: Could not assume role with OIDC: Not authorized to perform
sts:AssumeRoleWithWebIdentity

The role’s trust policy, unchanged for ninety days:

{
  "Effect": "Allow",
  "Principal": {
    "Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"
  },
  "Action": "sts:AssumeRoleWithWebIdentity",
  "Condition": {
    "StringEquals": {
      "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
      "token.actions.githubusercontent.com:sub": "repo:acme/platform-infra:ref:refs/heads/main"
    }
  }
}

A debug step added to both jobs mints the token and prints only its decoded payload. The token itself is a credential and is never echoed:

# Runs inside a job that has `permissions: id-token: write`.
# Print the PAYLOAD only. Never echo $TOKEN itself - it is a credential.
AUD="sts.amazonaws.com"

TOKEN=$(curl -sS \
  -H "Authorization: bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN" \
  "$ACTIONS_ID_TOKEN_REQUEST_URL&audience=$AUD" | jq -r '.value')

PAYLOAD=$(printf '%s' "$TOKEN" | cut -d. -f2 | tr '_-' '/+')
while [ $(( ${#PAYLOAD} % 4 )) -ne 0 ]; do PAYLOAD="${PAYLOAD}="; done

printf '%s' "$PAYLOAD" | base64 -d | jq '{sub, aud, repository, ref, environment}'

The plan job prints:

{
  "sub": "repo:acme/platform-infra:ref:refs/heads/main",
  "aud": "sts.amazonaws.com",
  "repository": "acme/platform-infra",
  "ref": "refs/heads/main",
  "environment": null
}

The apply job, in the same workflow run, prints:

{
  "sub": "repo:acme/platform-infra:environment:production",
  "aud": "sts.amazonaws.com",
  "repository": "acme/platform-infra",
  "ref": "refs/heads/main",
  "environment": "production"
}

The merged workflow diff:

   apply:
     needs: plan
     runs-on: ubuntu-latest
+    environment: production
     permissions:
       id-token: write
       contents: read

And in the repository that succeeded:

$ gh secret list --repo acme/payments-infra

AWS_ACCESS_KEY_ID       updated 2023-11-02
AWS_SECRET_ACCESS_KEY   updated 2023-11-02
TF_API_TOKEN            updated 2026-02-14

Work the evidence before reading on

Two jobs, one workflow run, one role, one repository variable holding the ARN. One is authorised and one is not.

1. Compare the two decoded payloads line by line. Exactly one field differs in a way the trust policy can see. Which one, and what does the trust policy do with it?
2. The trust policy has not changed in ninety days and the IAM permissions have not changed. Something still changed. Where does the value on the left-hand side of that string comparison come from — the role, or the caller?
3. Staging works. The staging apply job has no approval gate on it. Is that a coincidence or a control?

Before continuing: payments-infra merged the same workflow change and its production apply succeeded. Given what you now know about the subject claim, what must have happened inside that job — and which of the two repositories would you rather be running tonight?

Root cause

1. The credential was never the problem

Read the failure again: the job could not obtain credentials. It never held any, so it never used any, so no policy attached to the role was ever consulted. Every hour spent comparing IAM permissions was spent on the second half of a process that never reached its second half.

Federation splits authorisation into two questions that fail in different places and look alike in a log. Who are you? is answered by the token the CI platform mints and checked by the role’s trust policy. What may you do? is answered by the policies attached to the role. This failure is entirely in the first question.

2. An environment changes the shape of the subject claim

The CI platform does not mint one fixed identity per repository. It builds the token’s subject claim out of the job’s context, and the context includes whether the job is attached to a deployment environment. A job running against a branch asserts a subject of the ref form:

repo:acme/platform-infra:ref:refs/heads/main

A job attached to an environment asserts one of the environment form instead:

repo:acme/platform-infra:environment:production

Adding environment: production to the apply job is what added the approval gate — and it is also, unavoidably, a change to the identity that job asserts. The trust policy compares that string for equality against the ref form. The two strings are not equal. The security token service refuses.

The refusal is correct. A trust policy that accepted an assertion it was not written to accept would be a trust policy that authorised an identity nobody had reviewed.

3. Why the plan job and staging were unaffected

Nothing attached the plan job to an environment, so it still asserts the ref form and still matches. Staging’s apply job never had an approval gate added, so it still asserts the ref form too, and its own trust policy still matches.

That is why the failure looked environment-specific and was not. The correlation is not with the environment being production; it is with which jobs had the four-line change applied to them.

4. The repository that “worked” is the actual incident

payments-infra merged the same change and its OIDC exchange failed in exactly the same way. The difference is what happened next.

That repository still carries AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY from before the federation migration, exported at job level, and its credential-configuration step is marked continue-on-error: true — added during the migration so that a federation problem could not break the rollout. So when the exchange failed, the job did not stop. The Terraform steps that followed found those two variables in the environment and used them, because the provider does not care where valid credentials came from. The apply ran, the release shipped, and nobody was paged.

The two decisions that produced this are individually reasonable and lethal together: keep the old credentials until the new path is proven, and do not let the new path break the build while it is being proven. Between them they guarantee that the migration cannot be observed to fail.

It applied under an identity that was created for a different purpose years ago, is broader than the federated role, does not expire, and appears in the audit trail as a key rather than as a workflow. The team that got the failure spent an evening on it. The team that did not get a failure has no idea anything happened.

Resolution

1. Hold the release, with a named owner and a stated end time. Nothing in production is broken; the pipeline is refusing to act, which is the outcome the control exists to produce. Record the hold so that the decision is visible rather than implied by silence.
2. Confirm the diagnosis from the token rather than from reasoning. The decoded subject claim from the failing job, next to the trust policy condition, is the whole finding in two lines. Do not change IAM before that pair is in the ticket.
3. Correct the trust policy deliberately, adding the environment form of the subject for the specific environment name alongside the existing ref form. Both are needed while the plan job runs against the branch and the apply job runs against the environment.
4. Do not replace the equality condition with a repository-wide prefix match. Naming the exact environment keeps the policy a control; a prefix match authorises every workflow in the repository, on every branch, including one added by a pull request.
5. Land the trust-policy change through the same review path as any other production IAM change, and through Terraform if IAM is managed there, so the fix does not exist only as a console edit that the next converge silently reverts.
6. Treat payments-infra as the higher-severity half of the incident, and open it separately so it does not close when the release ships. Revoke the long-lived key, delete the secrets, and confirm the next run fails rather than falls back.
7. Audit what the long-lived key did. Every apply that repository ran while the key was the effective identity was executed by an identity nobody reviewed, with permissions nobody scoped for it. Reconstruct that list from the audit trail and reconcile it against the merge history.
8. Sweep every other repository using this workflow template for the same pair of secrets, before the next person adds an approval gate. The fallback is silent by construction, so its absence has to be verified rather than assumed.
9. Correct the pull request record. A change that alters the identity a production pipeline asserts was reviewed in ninety seconds because its title said it changed nothing.

Verification

1. The production apply job completes the exchange, and the job log records the decoded subject claim that was accepted. A green job with no claim recorded proves that something worked, not that the right thing worked.
2. The caller identity inside each job is an assumed role and the access key carries the temporary-session prefix, not the long-lived prefix. Check this in every repository on the template, not only in the one that failed.
3. The negative case still fails. Run the workflow from a branch, and from an environment, that the trust policy does not name, and confirm both are refused. A fix verified only by the happy path cannot distinguish a correction from a widening.
4. The trust policy in the account matches the one in version control, byte for byte. A console edit made during the incident and never committed is the most common way this recurs six months later.
5. No long-lived key prefix appears in the audit trail for these roles over the following week. This is the only check that would have caught the second repository, and it is the one nobody was running.
6. A repository with the secrets removed fails loudly when federation fails. Break the exchange deliberately on a test repository and confirm the job stops rather than proceeding under another identity.

Prevention

• Treat the execution context of a job as part of its identity. An environment, a reusable-workflow call, or a change of trigger can all rewrite the subject claim. Any of them belongs in the same review as the trust policy that authorises it.
• Keep the trust policy next to the pipeline that depends on it. When the workflow and the condition live in one repository and move through one review, a change to one that breaks the other is visible in the diff instead of at 18:52 on a Thursday.
• Keep the condition tight and explicit. Naming the repository, the branch and the environment is what makes the policy a control. Every wildcard added to make an error go away is authorisation granted to whoever finds it next.
• Never leave a second credential path in a runner. A federated pipeline that still holds a long-lived key has two identities and uses whichever one works. That turns every future authorisation failure into a silent privilege escalation, and it erases the attribution that federation was adopted to provide.
• Alert on the shape of the identity, not only on failures. A long-lived key prefix appearing where a session prefix is expected is the signal the fallback fired. It is the only signal payments-infra ever produced, and nothing was watching for it.
• Read a pull request title as a claim, not as a summary. “Workflow only, no infra change” was true about the files and false about the effect, and it bought a ninety-second review for a change to a production identity.