critical risksecurity relevant~60 min

Runbook: Rotate a Deployment Identity (OIDC Reconfiguration)

1 · Prerequisites

Confirm every item is in place before any state change.

git-cicd-gitops-rb-08-rotate-git-credentials
git-cicd-gitops-rb-17-troubleshoot-argocd-auth
Access to the OIDC provider (Okta, Azure AD, Google Workspace, GitHub OIDC)
Access to the CI/CD system (GitHub Actions, GitLab CI) and the ability to register a new OIDC trust
Access to the target cluster/cloud/registry to update the trust policy
Knowledge of the current identity's trust policy and the new identity's intended trust policy

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Identify the deployment identity in use. Common patterns: GitHub Actions OIDC (token.actions.githubusercontent.com); GitLab CI JWT (gitlab.com); cloud provider service accounts (AWS IAM Role, GCP Service Account, Azure Managed Identity); Kubernetes ServiceAccount with projected SA tokens; long-lived secrets (legacy). The identity type determines the rotation procedure
· Identify every consumer of the identity. gh api repos/myorg/myrepo/actions/oidc and search the codebase for id-token: in workflows; kubectl get clusterrolebindings -A and grep for the SA; aws iam list-roles for IAM roles. Every consumer must be updated
· Capture the current trust policy. For AWS IAM OIDC: aws iam get-open-id-connect-provider --open-id-connect-provider-arn <arn> and aws iam get-role --role-name <role> --query 'Role.AssumeRolePolicyDocument'. For GitHub Actions: the workflow's permissions: id-token: write and the OIDC trust condition. The current trust policy is the source of truth for "what was allowed"
· Plan the cutover. Rotation must run old and new identities in parallel during the cutover. The new identity is provisioned, given the same permissions as the old, and tested; then the old identity is deprecated and removed. A "swap in place" without parallel run is an outage waiting to happen
· Confirm the rotation is authorized. A deployment identity rotation is a security-relevant change; it must be approved by the security team and the platform team. The decision is not unilateral
· Identify the audit and monitoring: every identity use should be logged. aws cloudtrail lookup-events for IAM; the CI/CD system's audit log for OIDC tokens; kubectl get events for the cluster. The audit confirms the old identity is no longer used after the cutover

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1STEP 1 - Provision the new identity. For GitHub Actions OIDC: create a new GitHub App or a new OIDC trust subject in the cloud provider. For AWS IAM: aws iam create-role --role-name <new-role> --assume-role-policy-document file://new-trust.json. For GCP: gcloud iam service-accounts create <new-sa>. For Kubernetes: create a new ServiceAccount in the GitOps controller namespace. The new identity starts with NO permissions
2STEP 2 - Configure the OIDC trust between the CI/CD system and the cloud/cluster. For AWS IAM + GitHub Actions: create or update the OIDC provider with the correct thumbprint (aws iam create-open-id-connect-provider); the trust policy in the new role specifies the GitHub org/repo/branch/workflow conditions. For GCP Workload Identity: bind the GSA to the KSA via gcloud iam service-accounts add-iam-policy-binding. For Kubernetes: the SA token is projected by the controller
3STEP 3 - Grant the new identity the same permissions as the old. For AWS: copy the policies from the old role to the new role (aws iam attach-role-policy --role-name <new-role> --policy-arn <arn> for each policy). For GCP: gcloud projects add-iam-policy-binding --member serviceAccount:<new-sa> --role <role> for each role. For Kubernetes: copy the ClusterRoleBindings and RoleBindings from the old SA to the new SA. The new identity is now a mirror of the old
4STEP 4 - Test the new identity WITHOUT touching production. Use a staging environment or a dry-run workflow. For GitHub Actions: run a workflow that authenticates with the new OIDC trust and lists the test bucket / test namespace. For Kubernetes: kubectl auth can-i <verb> <resource> --as=system:serviceaccount:<ns>:<new-sa>. The test must succeed
5STEP 5 - Update the CI/CD workflows to use the new identity. For GitHub Actions OIDC: update the workflow's aws-role input or azure subscription or gcp-service-account to the new identity. Test in staging. For Kubernetes ServiceAccount: update the Pod spec or the controller's workload to reference the new SA. For cloud provider service accounts: update the CI/CD variable or secret to reference the new identity
6STEP 6 - Run in parallel: cut over workflows one at a time. Each workflow uses the new identity; verify each workflow completes successfully. Monitor the cloud provider's audit log for the old identity — it should show ZERO new uses after the cutover
7STEP 7 - Update the GitOps controller's identity (Argo CD, Flux). Argo CD uses a Kubernetes ServiceAccount by default; the new SA must have the same ClusterRoleBindings. Patch the controller's StatefulSet: kubectl set serviceaccount statefulset/argocd-application-controller -n argocd <new-sa>. Restart the StatefulSet. Verify the controller can still reconcile: argocd app list
8STEP 8 - Update Kubernetes external secrets (if applicable). If the team uses External Secrets Operator or SOPS, the rotation must include the secret backend (Vault, AWS Secrets Manager, GCP Secret Manager). Rotate the IAM/GCP/Azure identity that the secrets backend uses to authenticate; the secrets backend will then re-issue the Kubernetes secrets with the new identity
9STEP 9 - Verify the new identity is in use everywhere. gh api repos/myorg/myrepo/actions/runs?per_page=10 and check the audit logs; the old identity should show zero uses in the last 24 hours. For AWS: aws cloudtrail lookup-events --lookup-attributes AttributeKey=User,AttributeValue=<old-role> returns no events. For Kubernetes: kubectl get events -n <ns> shows the new SA in the events
10STEP 10 - Deprecate the old identity. Remove the policies/permissions from the old identity. For AWS IAM: aws iam detach-role-policy --role-name <old-role> --policy-arn <arn> for each policy. For Kubernetes: delete the old ServiceAccount and its bindings. For OIDC: delete the old trust subject from the OIDC provider. The old identity can no longer be used
11STEP 11 - Delete the old identity. After a grace period (typically 7 days for production rotations), delete the old identity entirely: aws iam delete-role --role-name <old-role>; gcloud iam service-accounts delete <old-sa>; kubectl delete serviceaccount <old-sa> -n <ns>. The rotation is complete
12STEP 12 - Record the rotation. Open a ticket with: the identities rotated, the consumers updated, the cutover date, the deprecation date, the deletion date, the operator, the approver, the verification evidence. The record is the audit trail and the basis for the next rotation

4 · Verification

Confirm the procedure actually fixed the problem.

✓The new identity can authenticate against every target it needs to (cluster, cloud, registry). For AWS: aws sts assume-role-with-web-identity --role-arn <new-role> --web-identity-token <token> returns temporary credentials. For Kubernetes: kubectl auth can-i list pods --as=system:serviceaccount:<ns>:<new-sa> returns yes
✓Every CI/CD workflow runs successfully using the new identity. gh run list --workflow=<wf> --limit=10 shows no failures attributable to auth
✓The GitOps controller reconciles successfully with the new identity: argocd app list shows all Applications Synced + Healthy
✓The audit log shows zero uses of the old identity since the cutover: aws cloudtrail lookup-events --lookup-attributes AttributeKey=User,AttributeValue=<old-role> --start-time <cutover> returns no events
✓The old identity has been de-deprecated (policies removed) and is no longer authorized to perform any action
✓A test workflow that uses the old identity fails with AccessDenied or equivalent — proving the old identity is truly disabled
✓All consumers have been updated: search the codebase for the old identity reference (rg <old-role>) and verify no production code references it

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶If the new identity fails to authenticate in production: revert to the old identity. The parallel-run design means the old identity is still valid during the cutover. Revert the CI/CD workflows to the old identity, re-enable the old identity's policies if they were removed prematurely, and investigate the new identity's failure
↶If the OIDC trust configuration is wrong (the CI/CD system cannot get a token, or the token is rejected by the cloud provider): the trust policy is the suspect. Diff the new trust policy against the old (diff -u old-trust.json new-trust.json); the difference is usually a typo in the audience, the subject, or the thumbprint. Fix the trust policy; do not bypass it
↶If the new identity has the wrong permissions (a workflow that worked with the old identity fails with the new identity): the policy attachment is incomplete. Diff the old role's policies against the new role's: aws iam list-attached-role-policies --role-name <old-role> vs --role-name <new-role>. Attach the missing policies
↶If the GitOps controller fails to reconcile with the new ServiceAccount: the ClusterRoleBindings were not copied. Diff the bindings: kubectl get clusterrolebindings -o yaml | grep <old-sa> vs <new-sa>. Add the missing bindings
↶If the rotation was authorized but the cutover was not coordinated (some consumers updated, others not): the old identity is still in use by some consumers; the new identity is in use by others. This is a split-brain scenario. Either push all consumers to the new identity or revert all to the old identity; do not run in mixed mode longer than necessary
↶If the deprecation of the old identity happened before all consumers were updated: the consumers that still use the old identity will fail. Do NOT delete the old identity; re-enable the policies immediately. Investigate which consumers were missed and update them. The grace period (7 days) exists to prevent this
↶If the rotation exposed a dependency that relied on the old identity (an undocumented IAM user, a hidden webhook): the dependency must be updated too. Find it with aws iam get-principal-tag or by searching the audit log for the old identity. Update the dependency; do not bypass the rotation

6 · Escalation

When the runbook isn't enough, contact:

· The new identity is being rejected because the old identity was previously compromised: see git-cicd-gitops-rb-28-respond-to-supply-chain-compromise. The rotation is a recovery action; the security investigation continues in parallel
· The rotation requires updating the trust policy in a way that affects other teams (e.g., the OIDC provider is shared): coordinate with the security team and the platform team. A misconfigured trust policy can grant access to unintended parties
· The rotation reveals that the team's identity sprawl is unmanageable (100+ IAM roles, dozens of SAs): the fix is to consolidate identities (use one IAM role per environment, not per workflow). Engage the security team and the platform team; this is a multi-week project
· The old identity has elevated permissions that should not be in production (admin, root, wildcard *): the rotation must include a permission review. The new identity should have the minimum permissions needed; the old identity's over-permissioning is a process violation. Engage the security team
· The rotation affects a regulated system (PCI, HIPAA, SOX): the rotation must follow the regulator's change-control procedure. Engage the compliance team. The audit trail must demonstrate the rotation was authorized, executed, and verified before the old identity was deleted
· The new identity fails to authenticate due to a regional issue (the OIDC provider is in a different region than the target): the trust policy must specify the region. Engage the platform team; this is a configuration issue, not an identity issue
· The rotation cannot complete because a downstream system does not support OIDC and requires long-lived secrets: see git-cicd-gitops-rb-08-rotate-git-credentials. The long-lived secret is the legacy fallback; OIDC migration is the strategic direction

A deployment identity is the credential the CI/CD system and the GitOps controller use to authenticate to the cluster, the cloud, and the registry. Rotating that identity is a security-relevant change — it reduces the blast radius of a compromised credential and keeps the team aligned with the principle of least privilege. The rotation runs old and new identities in parallel during the cutover.

1. Identify the current identity and its consumers

Read-only / Safe

$ echo "--- GitHub Actions workflows that use OIDC ---"
gh api 'repos/myorg/myrepo/actions/workflows' --jq '.workflows[] | .path'
grep -rE 'aws-role|azure-subscription|gcp-service-account|id-token:' .github/workflows/
echo "--- Kubernetes ServiceAccounts used by GitOps ---"
kubectl get sa -A | grep -E 'argocd|flux'
echo "--- AWS IAM roles used by CI/CD ---"
aws iam list-roles --query 'Roles[?AssumeRolePolicyDocument.Statement[?Principal.Federated]].RoleName'
echo "--- audit the current identity''s recent use ---"
aws cloudtrail lookup-events --lookup-attributes AttributeKey=User,AttributeValue=REPLACE_WITH_OLD_ROLE --max-items 5

The consumer list is the rotation scope. Every consumer must be updated. The audit confirms where the identity is in use.

2. Capture the current trust policy and permissions

Read-only / Safe

$ OLD_ROLE="github-actions-deploy-prod"
echo "--- current trust policy ---"
aws iam get-role --role-name "$OLD_ROLE" --query 'Role.AssumeRolePolicyDocument'
echo "--- attached policies ---"
aws iam list-attached-role-policies --role-name "$OLD_ROLE" --query 'AttachedPolicies[].PolicyArn'
echo "--- save as the rotation baseline ---"
aws iam get-role --role-name "$OLD_ROLE" > /tmp/old-role.json
aws iam list-attached-role-policies --role-name "$OLD_ROLE" > /tmp/old-policies.json

The baseline is what the new identity must mirror. The save is the recovery path if the rotation fails.

3. Provision the new identity with the same trust policy

Read-only / Safe

$ NEW_ROLE="github-actions-deploy-prod-v2"
echo "--- create the new role with the same trust policy ---"
aws iam create-role --role-name "$NEW_ROLE" --assume-role-policy-document file://trust.json
echo "--- copy all attached policies ---"
for POLICY_ARN in $(aws iam list-attached-role-policies --role-name "github-actions-deploy-prod" --query 'AttachedPolicies[].PolicyArn' --output text); do
aws iam attach-role-policy --role-name "$NEW_ROLE" --policy-arn "$POLICY_ARN"
done
echo "--- verify ---"
aws iam list-attached-role-policies --role-name "$NEW_ROLE" --query 'AttachedPolicies[].PolicyArn'

The new identity starts as a mirror of the old. Same trust, same policies. The next step updates the trust to point to the new OIDC subject (if applicable).

4. Update the OIDC trust subject (GitHub Actions → AWS)

Read-only / Safe

$ echo "--- new trust policy with the new subject ---"
cat > /tmp/new-trust.json <<'EOF'
{
"Version": "2012-10-17",
"Statement": [{
  "Effect": "Allow",
  "Principal": {"Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"},
  "Action": "sts:AssumeRoleWithWebIdentity",
  "Condition": {
    "StringEquals": {
      "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
      "token.actions.githubusercontent.com:sub": "repo:myorg/myrepo:ref:refs/heads/main"
    }
  }
}]
}
EOF
aws iam update-assume-role-policy --role-name "$NEW_ROLE" --policy-document file:///tmp/new-trust.json
aws iam get-role --role-name "$NEW_ROLE" --query 'Role.AssumeRolePolicyDocument'

The trust subject is what binds the GitHub workflow to the IAM role. A typo here is a failed authentication.

5. Test the new identity in a non-production workflow

Read-only / Safe

$ echo "--- test workflow ---"
gh workflow run test-deploy-staging.yml -f role_arn="arn:aws:iam::123456789012:role/github-actions-deploy-prod-v2"
gh run list --workflow=test-deploy-staging.yml --limit=3
echo "--- verify the assumed identity ---"
gh run view REPLACE_WITH_RUN_ID --log | grep -iE 'assumed|arn:aws:sts' | head -5

The test must succeed and the assumed ARN must be the new role. A failure here means the trust policy is wrong; fix it before proceeding to production.

6. Update CI/CD workflows to use the new identity

Read-only / Safe

$ OLD_ROLE="github-actions-deploy-prod"
NEW_ROLE="github-actions-deploy-prod-v2"
echo "--- find all references ---"
grep -rl "$OLD_ROLE" .github/workflows/
echo "--- update each workflow ---"
sed -i "s|$OLD_ROLE|$NEW_ROLE|g" .github/workflows/deploy-prod.yml
git diff .github/workflows/deploy-prod.yml
git add -A && git commit -m "rotate(deploy-identity): use new IAM role" && git push
echo "--- verify in staging first ---"
gh workflow run deploy-staging.yml
gh run watch

The workflow update is the source of truth. Each workflow is updated one at a time; verify each before moving to the next.

7. Update the GitOps controller”s identity (Argo CD / Flux)

Read-only / Safe

$ NS="argocd"
NEW_SA="argocd-application-controller-v2"
echo "--- create the new ServiceAccount ---"
kubectl create serviceaccount "$NEW_SA" -n "$NS"
echo "--- copy all ClusterRoleBindings ---"
kubectl get clusterrolebindings -o json | jq -r '.items[] | select(.subjects[]? | .name == "argocd-application-controller") | .metadata.name' | while read CRB; do
kubectl get clusterrolebinding "$CRB" -o json | jq --arg NEW_SA "$NEW_SA" '.subjects |= map(if .name == "argocd-application-controller" then .name = $NEW_SA else . end)' | kubectl apply -f -
done
echo "--- restart the controller with the new SA ---"
kubectl set serviceaccount statefulset/argocd-application-controller -n "$NS" "$NEW_SA"
kubectl rollout status statefulset/argocd-application-controller -n "$NS" --timeout=300s
argocd app list

The controller restart is required for the new SA to take effect. The argocd app list confirms the controller can still reconcile.

8. Verify the old identity is no longer used

Read-only / Safe

$ OLD_ROLE="github-actions-deploy-prod"
CUTOVER="2026-08-20T00:00:00Z"
echo "--- old role audit (should be empty since cutover) ---"
aws cloudtrail lookup-events \
--lookup-attributes AttributeKey=User,AttributeValue="$OLD_ROLE" \
--start-time "$CUTOVER" \
--max-items 10
echo "--- new role audit (should show recent activity) ---"
aws cloudtrail lookup-events \
--lookup-attributes AttributeKey=User,AttributeValue="github-actions-deploy-prod-v2" \
--start-time "$CUTOVER" \
--max-items 10

The audit confirms the cutover. Zero events for the old identity, recent events for the new.

9. Deprecate the old identity (remove permissions)

Read-only / Safe

$ OLD_ROLE="github-actions-deploy-prod"
echo "--- detach all policies ---"
for POLICY_ARN in $(aws iam list-attached-role-policies --role-name "$OLD_ROLE" --query 'AttachedPolicies[].PolicyArn' --output text); do
aws iam detach-role-policy --role-name "$OLD_ROLE" --policy-arn "$POLICY_ARN"
done
echo "--- verify (test workflow that uses the old role must fail) ---"
gh workflow run test-old-role.yml -f role_arn="arn:aws:iam::123456789012:role/$OLD_ROLE"
gh run view REPLACE_WITH_RUN_ID --log | grep -iE 'denied|forbidden|access' | head -5

After deprecation, the old identity still exists but cannot be used. The test workflow that uses it must fail with AccessDenied.

10. Delete the old identity (after the grace period)

Read-only / Safe

$ OLD_ROLE="github-actions-deploy-prod"
OLD_SA="argocd-application-controller"
echo "--- delete the IAM role (after 7-day grace period) ---"
aws iam delete-role --role-name "$OLD_ROLE"
echo "--- delete the ServiceAccount ---"
kubectl delete serviceaccount "$OLD_SA" -n argocd
echo "--- record the rotation ---"
gh issue create --repo myorg/myorg --title "deploy-identity rotation: $OLD_ROLE -> v2" \
--body "Old identity deleted. Rotation complete. Operator: $USER. Approver: REPLACE_WITH_NAME. Audit log attached." \
--label security --label rotation --label audit

After the grace period, the old identity is deleted. The rotation record closes the loop.

Verification

The new identity can authenticate against every target it needs to. Every CI/CD workflow runs successfully using the new identity. The GitOps controller reconciles successfully with the new identity. The audit log shows zero uses of the old identity since the cutover. The old identity has been de-deprecated and is no longer authorized. A test workflow that uses the old identity fails with AccessDenied. All consumers have been updated (searched the codebase for the old identity reference).

Rollback

If the new identity fails to authenticate, revert to the old — the parallel-run design means the old is still valid during the cutover. If the OIDC trust configuration is wrong, diff the new trust against the old and fix the typo. If the new identity has wrong permissions, attach the missing policies from the old role. If the GitOps controller fails to reconcile, copy the missing ClusterRoleBindings. If the cutover was not coordinated (some consumers updated, others not), either push all to the new or revert all to the old — do not run mixed mode longer than necessary. If the deprecation happened before all consumers were updated, re-enable the policies immediately.

1 · Prerequisites

2 · Pre-checks

3 · Procedure

4 · Verification

5 · Rollback

6 · Escalation

1. Identify the current identity and its consumers

2. Capture the current trust policy and permissions

3. Provision the new identity with the same trust policy

4. Update the OIDC trust subject (GitHub Actions → AWS)

5. Test the new identity in a non-production workflow

6. Update CI/CD workflows to use the new identity

7. Update the GitOps controller”s identity (Argo CD / Flux)

8. Verify the old identity is no longer used

9. Deprecate the old identity (remove permissions)

10. Delete the old identity (after the grace period)

Verification

Rollback

References