Lab 24: Rotate deployment credentials — AWS access key to OIDC federation · Lab · Git, CI/CD & GitOps

Objective

By the end of this lab you will have authored the artefacts that rotate the deployment credentials from a long-lived AWS access key to short-lived OIDC federated credentials: the IAM OIDC trust policy, the IAM role permission policy, the GitHub Actions workflow with OIDC, the rotation runbook, the rotation evidence, the audit trail, and the pre-rotation inventory.

The point of this lab is not the AWS IAM policy syntax — the AWS docs cover that. The point is the discipline: the inventory of systems that use the credential, the staged rotation (issue, validate, deprecate, revoke), the audit trail, and the runbook for when the credential is compromised. Without the discipline, a “rotation” becomes a new long-lived secret stored in the same place as the old one.

Architecture

The team’s CI/CD pipeline deploys to EKS via the eksctl and kubectl CLIs. The CLIs authenticate to EKS via the AWS API; the AWS API authenticates via an IAM access key (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY) stored in the GitHub repository’s secrets. The access key is long-lived (never expires), has administrator access, and is visible to anyone with read access to the repository.

The rotation replaces the access key with an OIDC federated credential: GitHub Actions presents an OIDC token to AWS, AWS validates the token against the IAM OIDC identity provider, and AWS issues short-lived credentials (default: 1 hour) scoped to the IAM role. The role’s permission policy grants the EKS deploy permissions; the trust policy restricts the role to the team’s GitHub repository and the main branch.

flowchart LR
    A["GitHub Actions\nrunner"] -- "OIDC token" --> B["AWS IAM\nOIDC provider"]
    B -- "validate" --> C["IAM role\neks-deployer"]
    C -- "issue short-lived\ncredentials" --> A
    A -- "use credentials" --> D["EKS API\nupdate kubeconfig"]
    A -- "kubectl apply" --> E["EKS cluster\nproduction"]

The OIDC token is signed by GitHub’s OIDC provider (token.actions.githubusercontent.com). AWS validates the signature, the audience (sts.amazonaws.com), and the subject (repo:runbook-academy/eks-deploy:ref:refs/heads/main). The short-lived credentials are issued for the IAM role and used by the aws, eksctl, and kubectl CLIs.

Requirements

An AWS account with IAM administrator access.
A GitHub repository with Actions enabled.
AWS CLI 2.x authenticated as the administrator.
The current AWS access key in a safe location for the rotation (the lab uses an environment variable).

Scenario

A platform team uses an AWS access key to deploy to EKS from GitHub Actions. The access key has been in use for 18 months; the team’s secret scanning detected a copy of the key in a developer’s local .env file (not in the repository). The team rotates the key to OIDC federation, eliminating the static secret. The rotation is performed in stages to avoid breaking the deploy pipeline.

Tasks

Task 1 — Build the pre-rotation inventory

# check-shell-blocks: allow-invalid
LAB="$HOME/cred-rotate-lab"
rm -rf "$LAB"
mkdir -p "$LAB"
cd "$LAB"

git init -b main
git config user.email 'ops@example.com'
git config user.name  'Ops'

cat > pre-rotation-inventory.md <<'EOF'
# Pre-rotation inventory: AWS access key AKIA...

This document is the canonical inventory of every
system and person that uses the AWS access key being
rotated. The inventory is the input to the rotation
plan; every entry must be updated or revoked before
the old key is deleted.

## Access key

- Access key ID: `AKIA...REDACTED...`
- IAM user: `eks-deployer`
- Created: 2025-02-15
- Last used: 2026-08-22
- Permissions: AdministratorAccess (managed policy)

## Systems that use the key

| System | User/Role | Usage | Plan |
|--------|-----------|-------|------|
| GitHub Actions: `eks-deploy` workflow | bot@runbook-academy | Deploy to EKS production | Switch to OIDC |
| Local `~/.aws/credentials` (developer laptop) | alice@example.com | Manual debugging | Remove; use AWS SSO |
| Local `~/.aws/credentials` (developer laptop) | bob@example.com | Manual debugging | Remove; use AWS SSO |
| Terraform Cloud workspace | tfc-agent | `terraform apply` to EKS | Switch to OIDC |
| Jenkins (legacy CI) | jenkins-deploy | Deploy to staging EKS | Migrate to GitHub Actions |
| Local laptop (ex-developer) | charlie@example.com | Last used 2024-09 | Verify revoked |

## Plan

1. **Stage 1 (this week):** Switch GitHub Actions to
   OIDC. Deploys continue to work.
2. **Stage 2 (this week):** Switch Terraform Cloud to
   OIDC. `terraform plan` and `apply` continue to work.
3. **Stage 3 (this week):** Migrate Jenkins to GitHub
   Actions. Staging deploys continue to work via the
   new workflow.
4. **Stage 4 (next week):** Remove the key from all
   developer laptops. Verify with AWS credential
   reports.
5. **Stage 5 (next week):** Delete the access key from
   IAM.

## Verification

After each stage, the team verifies:

- The system still works (deploys succeed, Terraform
  plans succeed, etc.).
- The access key is no longer used by the system
  (AWS CloudTrail `AssumeRole` or `GetSessionToken`
  events from the system are zero).

EOF

git add pre-rotation-inventory.md
git commit -m 'rotation: pre-rotation inventory'

The inventory is the rotation plan. Every system that uses the key is listed; every system has a plan (switch to OIDC, remove, migrate); every system has a verification step.

Task 2 — Build the IAM OIDC identity provider

# check-shell-blocks: allow-invalid
cd "$HOME/cred-rotate-lab"

cat > iam-oidc-provider.json <<'EOF'
{
  "Url": "https://token.actions.githubusercontent.com",
  "ClientIDList": ["sts.amazonaws.com"],
  "ThumbprintList": [
    "6938fd4d98bab03faadb97b34396831e3780aea1",
    "1c58a3a8518e8759bf075b76b750d4f2df264fcd"
  ]
}
EOF

git add iam-oidc-provider.json
git commit -m 'aws: OIDC identity provider config'

# The provider is created via AWS CLI.
aws iam create-open-id-connect-provider \
  --cli-input-json file://iam-oidc-provider.json

The OIDC identity provider is the AWS-side component that validates GitHub’s OIDC tokens. The provider is created once per AWS account; the team’s existing provider is reused.

Task 3 — Build the IAM role trust policy

# check-shell-blocks: allow-invalid
cd "$HOME/cred-rotate-lab"

cat > iam-oidc-trust-policy.json <<'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
        },
        "StringLike": {
          "token.actions.githubusercontent.com:sub": "repo:runbook-academy/eks-deploy:ref:refs/heads/main"
        }
      }
    }
  ]
}
EOF

git add iam-oidc-trust-policy.json
git commit -m 'aws: OIDC trust policy for eks-deployer role'

The trust policy is the IAM role’s gate. The policy allows the role to be assumed only by OIDC tokens from the runbook-academy/eks-deploy repository, on the main branch, and for the sts.amazonaws.com audience. Tokens from other repositories, branches, or audiences are rejected.

Task 4 — Build the IAM role permission policy

# check-shell-blocks: allow-invalid
cd "$HOME/cred-rotate-lab"

cat > iam-role-policy.json <<'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "EKSDescribeCluster",
      "Effect": "Allow",
      "Action": [
        "eks:DescribeCluster",
        "eks:ListClusters"
      ],
      "Resource": "arn:aws:eks:us-east-1:123456789012:cluster/production"
    },
    {
      "Sid": "ECRReadOnly",
      "Effect": "Allow",
      "Action": [
        "ecr:GetAuthorizationToken",
        "ecr:BatchCheckLayerAvailability",
        "ecr:GetDownloadUrlForLayer",
        "ecr:BatchGetImage",
        "ecr:DescribeImages",
        "ecr:DescribeRepositories"
      ],
      "Resource": "*"
    },
    {
      "Sid": "S3ReadDeployArtifacts",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::runbook-deploy-artifacts",
        "arn:aws:s3:::runbook-deploy-artifacts/*"
      ]
    }
  ]
}
EOF

git add iam-role-policy.json
git commit -m 'aws: EKS deployer permission policy'

The permission policy grants the role the minimum permissions needed for the EKS deploy: describe the cluster, read from ECR, and read deploy artifacts from S3. The policy does not include eks:UpdateClusterConfig, iam:PassRole, or any other write permission that the deploy does not need.

Task 5 — Create the IAM role

cd "$HOME/cred-rotate-lab"

# Create the role with the trust policy.
aws iam create-role \
  --role-name eks-deployer-oidc \
  --assume-role-policy-document file://iam-oidc-trust-policy.json \
  --max-session-duration 3600

# Attach the permission policy.
aws iam put-role-policy \
  --role-name eks-deployer-oidc \
  --policy-name eks-deployer-permissions \
  --policy-document file://iam-role-policy.json

# Verify the role.
aws iam get-role --role-name eks-deployer-oidc
aws iam list-attached-role-policies --role-name eks-deployer-oidc

The role is created with a maximum session duration of 1 hour. The team’s discipline: short sessions; if the session token is leaked, the blast radius is 1 hour.

Task 6 — Update the GitHub Actions workflow

# check-shell-blocks: allow-invalid
cd "$HOME/cred-rotate-lab"

cat > github-actions-workflow.yaml <<'EOF'
# .github/workflows/deploy.yaml — the EKS deploy
# workflow with OIDC authentication.
#
# The workflow uses aws-actions/configure-aws-credentials
# to obtain short-lived AWS credentials via OIDC. The
# OIDC token is presented by the GitHub Actions runner;
# AWS validates the token against the IAM OIDC
# identity provider and issues short-lived credentials
# for the eks-deployer-oidc role.
#
# Required:
#   - AWS IAM role: eks-deployer-oidc
#   - AWS account ID: 123456789012
#   - IAM OIDC identity provider configured for
#     token.actions.githubusercontent.com
#   - EKS cluster: production

name: deploy-to-eks

on:
  push:
    branches: [main]
    paths:
      - 'app-source/**'
      - '.github/workflows/deploy.yaml'
  workflow_dispatch:

permissions:
  id-token: write   # required for OIDC
  contents: read    # required for checkout

jobs:
  deploy:
    name: deploy to EKS production
    runs-on: ubuntu-24.04
    environment: production
    steps:
      - name: checkout
        uses: actions/checkout@v4

      - name: configure AWS credentials (OIDC)
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/eks-deployer-oidc
          aws-region: us-east-1

      - name: update kubeconfig
        run: |
          aws eks update-kubeconfig \
            --name production \
            --region us-east-1

      - name: verify cluster access
        run: |
          kubectl get nodes
          kubectl get ns

      - name: deploy manifests
        run: |
          kubectl apply -f app-source/ --prune -l app=web
EOF

git add github-actions-workflow.yaml
git commit -m 'ci: OIDC-based EKS deploy workflow'

The workflow uses OIDC for AWS authentication. The permissions.id-token: write declaration is required for the OIDC token to be issued; without it, the OIDC step fails with id_token claim is missing.

Task 7 — Test the OIDC integration

# check-shell-blocks: allow-invalid
cd "$HOME/cred-rotate-lab"

# Stage 1: Test the OIDC integration in a sandbox.
cat > sandbox-test.sh <<'EOF'
#!/usr/bin/env bash
# sandbox-test.sh — test the OIDC integration in a
# sandbox workflow before applying to production.
#
# The script runs the same steps as the production
# workflow but on a sandbox cluster. The output is
# captured in /tmp/sandbox-output.log.

set -euo pipefail

EVIDENCE="\${EVIDENCE:-/tmp/sandbox-output.log}"

{
  echo "=== OIDC sandbox test at $(date -u +%Y-%m-%dT%H:%M:%SZ) ==="

  # Step 1: configure AWS credentials via OIDC.
  echo ""
  echo "--- step 1: configure-aws-credentials ---"
  # In CI, this is the aws-actions/configure-aws-credentials
  # step. Locally, we can simulate by setting the env
  # vars to the OIDC-derived credentials.

  # Step 2: assume the role.
  echo ""
  echo "--- step 2: sts get-caller-identity ---"
  aws sts get-caller-identity

  # Step 3: list EKS clusters.
  echo ""
  echo "--- step 3: eks list-clusters ---"
  aws eks list-clusters --region us-east-1

  # Step 4: describe the production cluster.
  echo ""
  echo "--- step 4: eks describe-cluster ---"
  aws eks describe-cluster --name production --region us-east-1 \
    --query 'cluster.{name:name,status:status,endpoint:endpoint}'

  # Step 5: read a deploy artifact from S3.
  echo ""
  echo "--- step 5: s3 ls deploy artifacts ---"
  aws s3 ls s3://runbook-deploy-artifacts/web/ --recursive \
    | head -5
} | tee "$EVIDENCE"
EOF

chmod +x sandbox-test.sh

git add sandbox-test.sh
git commit -m 'rotation: OIDC sandbox test'

The sandbox test exercises the OIDC integration end to end: assume the role, list clusters, describe the cluster, read from S3. The output is the evidence that the OIDC integration works.

Task 8 — Build the rotation runbook

# check-shell-blocks: allow-invalid
cd "$HOME/cred-rotate-lab"

cat > rotation-runbook.md <<'EOF'
# Rotation runbook: AWS access key to OIDC federation

This runbook is the on-call engineer's reference for
rotating the deployment credentials from a long-lived
AWS access key to short-lived OIDC federated
credentials. The runbook covers the staged rotation:
issue, validate, deprecate, revoke.

## When to use this runbook

Use this runbook when:

- A long-lived AWS access key is in use for
  production deploys.
- The team has decided to rotate to OIDC federation
  (security review, compliance requirement, or
  proactive hardening).
- The access key is suspected of compromise (see
  the "Compromised key" section below).

Do not use this runbook when:

- The access key is for a developer laptop and the
  developer uses AWS SSO. The access key is removed
  via `aws iam delete-access-key`.
- The access key is for an IAM role and the role is
  already federated. No rotation is needed.

## Stage 1: issue the new credential

The new credential is an IAM role with an OIDC trust
policy. Create the role and attach the permission
policy:

aws iam create-role —role-name eks-deployer-oidc
—assume-role-policy-document file://iam-oidc-trust-policy.json aws iam put-role-policy —role-name eks-deployer-oidc
—policy-name eks-deployer-permissions
—policy-document file://iam-role-policy.json


The role is the new credential. The trust policy
restricts the role to the specific GitHub
repository, branch, and audience.

## Stage 2: validate

Update the GitHub Actions workflow to use OIDC. Test
in a sandbox before production:

- Run the sandbox workflow (Task 7).
- Verify the OIDC token is issued.
- Verify the role is assumed.
- Verify the deploy succeeds.

Promote the workflow to production. Monitor the
first 10 deploys for any authentication failures.

## Stage 3: deprecate

After the OIDC integration is validated in production:

- Update the pre-rotation inventory (Task 1) to mark
  each system as "switched to OIDC".
- Remove the old access key from each system
  (GitHub secrets, Terraform Cloud, Jenkins, local
  laptops).
- Verify the access key is no longer used by checking
  AWS CloudTrail for the key's last use timestamp.

## Stage 4: revoke

After all systems are switched to OIDC:

aws iam delete-access-key
—user-name eks-deployer
—access-key-id AKIA…REDACTED…


The access key is deleted from IAM. The deletion is
irreversible; the team verifies the inventory one
more time before the deletion.

## Compromised key

If the access key is suspected of compromise:

1. **Page the security team** (PagerDuty security
   escalation).
2. **Disable the key immediately** (do not delete
   yet — disable preserves the audit trail):

aws iam update-access-key
—user-name eks-deployer
—access-key-id AKIA…REDACTED…
—status Inactive

3. **Investigate** the CloudTrail logs for the key's
activity since the suspected compromise time.
4. **Rotate** to OIDC as in Stages 1-4 above.
5. **Delete** the key after the rotation is verified.
6. **Postmortem** the incident; the security team
opens a follow-up PR for the controls that
allowed the compromise.

## Verification

After each stage, the team verifies:

- The new credential (OIDC) is used by the system.
- The old credential (access key) is not used by the
system.
- The deploys succeed.
- The audit trail (CloudTrail) shows the new role
being assumed.

EOF

git add rotation-runbook.md
git commit -m 'rotation: rotation runbook'

The runbook is the on-call reference. The four stages are the spine; the AWS CLI commands are the verbs. The “Compromised key” section is the emergency path.

Task 9 — Capture the rotation evidence

# check-shell-blocks: allow-invalid
cd "$HOME/cred-rotate-lab"

cat > rotation-evidence.md <<'EOF'
# Rotation evidence: AWS access key to OIDC — 2026-08-22

This document is the canonical evidence bundle for the
rotation performed on 2026-08-22. The bundle captures
the pre-rotation state, the rotation actions, and the
post-rotation state.

## Pre-rotation state

- Access key ID: `AKIA...REDACTED...`
- IAM user: `eks-deployer`
- Permissions: AdministratorAccess
- Last used: 2026-08-22 09:00 UTC
- Systems using the key: 6 (see pre-rotation-inventory.md)

## Rotation actions

### Stage 1: issued the new credential

- Created IAM role `eks-deployer-oidc`.
- Trust policy: OIDC, scoped to
  `repo:runbook-academy/eks-deploy:ref:refs/heads/main`.
- Permission policy: EKS describe, ECR read, S3 read.
- Session duration: 3600s (1 hour).

### Stage 2: validated

- Updated GitHub Actions workflow to use OIDC.
- Ran the sandbox workflow (Task 7).
- Sandbox output: `sandbox-output.log` (annexed).
- Promoted to production.
- First 10 production deploys: 10/10 successful.

### Stage 3: deprecated the old credential

- Removed the access key from GitHub secrets.
- Removed the access key from Terraform Cloud
  workspace.
- Migrated Jenkins to GitHub Actions.
- Removed the access key from developer laptops
  (alice, bob). Verified with AWS credential reports.
- CloudTrail `lastUsed` for the access key: 2026-08-22
  09:00 UTC (no new uses since the rotation).

### Stage 4: revoked

$ aws iam delete-access-key —user-name eks-deployer
—access-key-id AKIA…REDACTED… { “ResponseMetadata”: {} }


The access key is deleted from IAM. The deletion is
irreversible.

## Post-rotation state

- Access key: deleted.
- IAM role: `eks-deployer-oidc` is the new credential.
- GitHub Actions: OIDC-based, no static secrets.
- Terraform Cloud: OIDC-based, no static secrets.
- Jenkins: migrated to GitHub Actions.
- Developer laptops: AWS SSO, no static credentials.

## CloudTrail summary

- `AssumeRoleWithWebIdentity` for `eks-deployer-oidc`:
  10 events in the last 24 hours (one per production
  deploy).
- `AccessKeyUsed` for `AKIA...REDACTED...`: 0 events
  in the last 24 hours (the access key is no longer
  used).

## Audit trail

The full audit trail is in `audit-trail.md`. The trail
includes the timestamps, the operators, and the
verification steps.

EOF

git add rotation-evidence.md
git commit -m 'rotation: evidence bundle'

The evidence bundle is the canonical record. The pre-rotation state, the rotation actions, the post-rotation state, and the CloudTrail summary are the artefacts the team reviews at the post-rotation review.

Task 10 — Build the audit trail

# check-shell-blocks: allow-invalid
cd "$HOME/cred-rotate-lab"

cat > audit-trail.md <<'EOF'
# Audit trail: AWS access key to OIDC rotation

This document is the audit trail for the credential
rotation. The trail is the canonical record for the
compliance review; every action is timestamped and
attributed.

## 2026-08-22 09:00 — inventory created

- Operator: jane.doe
- Action: created pre-rotation-inventory.md
- Verification: 6 systems listed; all have a rotation
  plan.

## 2026-08-22 09:30 — OIDC provider verified

- Operator: jane.doe
- Action: verified the OIDC identity provider exists
  in IAM.
- Verification: `aws iam list-open-id-connect-providers`
  returned 1 provider with the correct URL.

## 2026-08-22 10:00 — IAM role created

- Operator: jane.doe
- Action: created `eks-deployer-oidc` role with the
  trust policy.
- Verification: `aws iam get-role` returned the role
  with the correct trust policy.

## 2026-08-22 10:30 — permission policy attached

- Operator: jane.doe
- Action: attached the permission policy to the role.
- Verification: `aws iam list-role-policies` returned
  the policy.

## 2026-08-22 11:00 — sandbox test

- Operator: jane.doe
- Action: ran the sandbox test (sandbox-test.sh).
- Verification: `aws sts get-caller-identity` returned
  the role ARN; `aws eks describe-cluster` succeeded.

## 2026-08-22 13:00 — production deploy with OIDC

- Operator: ci-bot (GitHub Actions)
- Action: production deploy via OIDC.
- Verification: deploy succeeded; CloudTrail recorded
  `AssumeRoleWithWebIdentity` for the role.

## 2026-08-22 14:00 — old access key removed from GitHub

- Operator: jane.doe
- Action: removed the access key from the GitHub
  repository secrets.
- Verification: `gh secret list` no longer shows the
  key.

## 2026-08-22 14:30 — Terraform Cloud switched to OIDC

- Operator: jane.doe
- Action: updated the Terraform Cloud workspace to
  use OIDC.
- Verification: `terraform plan` succeeded via OIDC.

## 2026-08-22 15:00 — Jenkins migrated

- Operator: jane.doe
- Action: migrated the Jenkins pipeline to GitHub
  Actions.
- Verification: staging deploy via GitHub Actions
  succeeded.

## 2026-08-22 16:00 — developer laptops cleaned

- Operator: jane.doe
- Action: removed the access key from alice and bob's
  `~/.aws/credentials`. Verified with AWS credential
  reports.
- Verification: AWS credential report shows the key
  is not in use.

## 2026-08-22 17:00 — old access key deleted

- Operator: jane.doe
- Action: deleted the access key from IAM.
- Verification: `aws iam list-access-keys` for the
  user returned an empty list.

## 2026-08-22 17:30 — post-rotation review

- Operators: jane.doe, sre-team, security-team
- Action: reviewed the rotation evidence and the
  audit trail.
- Decision: rotation accepted; no follow-up actions
  required.

EOF

git add audit-trail.md
git commit -m 'rotation: audit trail'

The audit trail is the canonical record for the compliance review. Every action is timestamped and attributed; the trail is the answer to “who did what, when?”.

Task 11 — Validate the deliverables

cd "$HOME/cred-rotate-lab"

# Verify the inventory has all sections.
grep -c "^## " pre-rotation-inventory.md
# expected: 5 (Access key, Systems, Plan, Verification,
#              Compromised key)

# Verify the trust policy is valid JSON.
python3 -c "import json; json.load(open('iam-oidc-trust-policy.json'))" \
  && echo "trust policy: valid JSON"

# Verify the role policy is valid JSON.
python3 -c "import json; json.load(open('iam-role-policy.json'))" \
  && echo "role policy: valid JSON"

# Verify the workflow has OIDC.
grep -c "configure-aws-credentials" github-actions-workflow.yaml
# expected: 1

# Verify the workflow has id-token: write.
grep -c "id-token: write" github-actions-workflow.yaml
# expected: 1

# Verify the runbook has all four stages.
grep -c "^## Stage" rotation-runbook.md
# expected: 4

# Verify the evidence bundle has all sections.
grep -c "^## " rotation-evidence.md
# expected: 5 (Pre-rotation, Rotation actions,
#              Post-rotation, CloudTrail, Audit trail)

# Verify the audit trail has timestamps.
grep -c "^## 2026-" audit-trail.md
# expected: 11

The deliverables are validated: the inventory has all sections, the policies are valid JSON, the workflow has OIDC, the runbook has all four stages, the evidence has all sections, and the audit trail has timestamps.

Task 12 — Capture the deliverables

cd "$HOME/cred-rotate-lab"

cp pre-rotation-inventory.md \
   iam-oidc-provider.json \
   iam-oidc-trust-policy.json \
   iam-role-policy.json \
   github-actions-workflow.yaml \
   sandbox-test.sh \
   rotation-runbook.md \
   rotation-evidence.md \
   audit-trail.md \
   "$HOME/"

ls -l "$HOME"/pre-rotation-inventory.md \
       "$HOME"/iam-oidc-trust-policy.json \
       "$HOME"/github-actions-workflow.yaml \
       "$HOME"/rotation-runbook.md \
       "$HOME"/rotation-evidence.md \
       "$HOME"/audit-trail.md

The deliverables are in $HOME/.

Validation

pre-rotation-inventory.md lists every system that uses the access key with a rotation plan and a verification step.
iam-oidc-trust-policy.json is valid JSON and scopes to the specific repository, branch, and audience.
iam-role-policy.json is valid JSON and grants only the permissions the deploy needs.
github-actions-workflow.yaml uses aws-actions/configure-aws-credentials with OIDC and has permissions.id-token: write.
rotation-runbook.md documents all four stages: issue, validate, deprecate, revoke.
rotation-evidence.md captures the pre-rotation state, the rotation actions, and the post-rotation state.
audit-trail.md has timestamped entries for every action.

Expected Outcome

An IAM OIDC trust policy, a permission policy, a GitHub Actions workflow with OIDC, a rotation runbook, a rotation evidence bundle, an audit trail, and a pre-rotation inventory.

$HOME/cred-rotate-lab/
├── pre-rotation-inventory.md     # systems + plan
├── iam-oidc-provider.json        # OIDC provider config
├── iam-oidc-trust-policy.json    # role trust policy
├── iam-role-policy.json          # role permissions
├── github-actions-workflow.yaml  # OIDC-based deploy
├── sandbox-test.sh               # OIDC sandbox test
├── rotation-runbook.md           # the runbook
├── rotation-evidence.md          # the evidence bundle
└── audit-trail.md                # the audit trail

The inventory is the safety net; the policies are the security boundary; the workflow is the implementation; the runbook is the on-call reference; the evidence and the audit trail are the institutional knowledge.

Troubleshooting

The OIDC token is rejected. The trust policy audience does not match. Verify with aws sts get-caller-identity and check the token.actions.githubusercontent.com:aud condition.

The role is not assumed. The trust policy subject does not match. Verify the workflow’s sub claim and the token.actions.githubusercontent.com:sub condition (use StringLike for wildcards).

The deploy fails with AccessDenied. The permission policy does not include the required permissions. The team’s policy: every deploy is granted exactly the permissions it needs; the permissions are reviewed in code review.

The old access key is still in use. A system was missed in the inventory. Check AWS CloudTrail for AccessKeyUsed events and update the inventory.

id-token: write is missing. The OIDC token is not issued. The workflow fails with id_token claim is missing. Add id-token: write to the workflow’s permissions block.

Cleanup

LAB="$HOME/cred-rotate-lab"

cp -r "$LAB"/* "$HOME"/ 2>/dev/null
rm -rf "$LAB"

# Delete the IAM role (optional).
aws iam delete-role --role-name eks-deployer-oidc

What You Learned

Inventory is the safety net. A rotation that misses a system breaks the system silently. The inventory lists every system that uses the credential with a rotation plan.
The trust policy is the security boundary. The policy scopes the OIDC integration to the specific repository, branch, and audience. A trust policy that allows repo:org/* is too permissive.
The permission policy is the least-privilege boundary. The deploy’s permissions are exactly the permissions the deploy needs. *:* is AdministratorAccess in disguise.
The staged rotation avoids breaking the pipeline. Issue, validate, deprecate, revoke. The four stages ensure the deploys continue to work while the credential is rotated.
CloudTrail is the verification tool. The lastUsed timestamp and the AssumeRoleWithWebIdentity events are the proof that the new credential is used and the old credential is not.
The audit trail is the canonical record. Every action is timestamped and attributed. The trail is the answer to “who did what, when?”.
The compromised-key path is the emergency. The team disables the key first (preserves the audit trail), investigates, then rotates. The deletion is the last step, not the first.
id-token: write is required for OIDC. Without it, the OIDC token is not issued and the workflow fails.

Lab 24: Rotate deployment credentials — AWS access key to OIDC federation

Objectives

Prerequisites