Skip to main content
RunBook Academy

← All labs in Git, CI/CD & GitOps

Lab · advanced · ~90 min

Lab 26: Recover CI after a credential rotation

C · SimulationB · Nested virtualisation

Objectives

  • Identify every CI system that depends on the rotated credential (GitHub Actions, runners, Argo CD, EKS, OCI registry)
  • Update the GitHub Actions secrets and the OIDC trust policy to use the new credential
  • Update the Argo CD repository credentials and the cluster credentials
  • Verify each system end-to-end with a smoke test
  • Capture the recovery evidence and the audit trail
  • Build the recovery runbook that the on-call engineer follows when the pipeline is broken

Prerequisites

Objective

By the end of this lab you will have authored the artefacts that recover the CI/CD pipeline after a credential rotation: the recovery runbook, the CI systems inventory, the GitHub secrets update script, the Argo CD repository credentials update script, the smoke test, the recovery evidence bundle, and the audit trail.

The point of this lab is not the gh secret set command — that is one CLI invocation. The point is the discipline: the inventory of CI systems that depend on the rotated credential, the staged update of each system, the smoke test for each update, and the audit trail. Without the discipline, a credential rotation cascades into a multi-hour outage.

Architecture

The team’s CI/CD pipeline depends on a set of credentials: GitHub Actions secrets (for the deploy workflow), Argo CD repository credentials (for the GitOps sync), the EKS cluster’s IAM role (for the deploy), and the OCI registry’s pull credentials (for the image pull). When a credential is rotated (for example, the AWS access key in Lab 24), every system that depends on the credential must be updated.

flowchart LR
    A["GitHub Actions\ndeploy workflow"] -- "uses" --> B["AWS access key\n(rotated)"]
    A -- "uses" --> C["GHCR token"]
    A -- "uses" --> D["kubectl context"]
    E["Argo CD"] -- "uses" --> F["Git repo creds"]
    E -- "uses" --> G["Cluster creds"]
    H["EKS cluster"] -- "uses" --> I["IAM role\n(rotated)"]
    J["Production Pods"] -- "uses" --> K["OCI image pull\nsecret (rotated)"]

When the AWS access key is rotated, the GitHub Actions secrets, the runner’s environment, and any deploy that authenticates via the key are broken. The on-call engineer must update each system in order: GitHub secrets first (the workflow fails fast), then Argo CD (the GitOps sync), then the cluster (IAM role), then the Pods (image pull).

Requirements

  • A GitHub organisation with Actions enabled.
  • A kind cluster with Argo CD installed (Lab 19).
  • The gh CLI authenticated as an organisation owner.
  • The argocd CLI authenticated against the cluster.
  • The new credential value (for example, the new AWS access key from Lab 24).

Scenario

A platform team rotated the AWS access key (Lab 24) but the GitHub Actions secrets, the Argo CD repository credentials, the EKS kubeconfig, and the OCI registry’s pull secret were not updated. At 09:15 UTC on 2026-08-22, the first production deploy fails with ExpiredTokenException. The on-call engineer receives the page and follows the recovery procedure.

Tasks

Task 1 — Build the CI systems inventory

# check-shell-blocks: allow-invalid
LAB="$HOME/ci-recover-lab"
rm -rf "$LAB"
mkdir -p "$LAB"
cd "$LAB"

git init -b main
git config user.email 'ops@example.com'
git config user.name  'Ops'

cat > ci-systems-inventory.md <<'EOF'
# CI systems inventory: AWS access key rotation

This document is the canonical inventory of every CI
system that depends on the rotated AWS access key.
The inventory is the input to the recovery plan; every
entry must be updated before the rotation is closed.

## Rotated credential

- Access key ID: `AKIA...REDACTED...`
- New access key ID: `AKIA...NEW...REDACTED...`
- Rotation date: 2026-08-22

## Systems that use the credential

| System | Where the credential is stored | Update method |
|--------|--------------------------------|---------------|
| GitHub Actions: `deploy-to-eks` workflow | Repository secret `AWS_ACCESS_KEY_ID` | `gh secret set` |
| GitHub Actions: `deploy-to-eks` workflow | Repository secret `AWS_SECRET_ACCESS_KEY` | `gh secret set` |
| Self-hosted runner (production) | `~/.aws/credentials` | `aws configure` |
| Self-hosted runner (staging) | `~/.aws/credentials` | `aws configure` |
| Argo CD: Git repository credentials | `argocd-cm` ConfigMap or Secret | `argocd repo creds` |
| Argo CD: Cluster credentials | `argocd-clusters` Secrets | `argocd cluster add` |
| EKS kubeconfig | `~/.kube/config` on runners and operators' laptops | `aws eks update-kubeconfig` |
| OCI registry pull secret | `production/registry-credentials` Secret in EKS | `kubectl create secret docker-registry` |
| Terraform Cloud workspace | Workspace environment variables | TFC UI |
| Jenkins (legacy CI) | Credentials store | Jenkins UI |

## Plan

1. **Stage 1 (this hour):** Update the GitHub
   Actions secrets. The deploy workflow resumes.
2. **Stage 2 (this hour):** Update the Argo CD
   repository and cluster credentials. The GitOps
   sync resumes.
3. **Stage 3 (this hour):** Update the OCI registry
   pull secret in the production namespace. The
   production Pods can pull images.
4. **Stage 4 (this hour):** Update the Terraform
   Cloud workspace. The `terraform plan` resumes.
5. **Stage 5 (this hour):** Update Jenkins (if
   still in use).

## Verification

After each stage, the team verifies:

- The system authenticates with the new credential.
- The system performs its function (deploy, sync,
  pull, plan).
- The audit trail records the update.

EOF

git add ci-systems-inventory.md
git commit -m 'incident: CI systems inventory'

The inventory is the recovery plan. Every system that uses the credential is listed; every system has an update method; every system has a verification step.

Task 2 — Build the recovery runbook

# check-shell-blocks: allow-invalid
cd "$HOME/ci-recover-lab"

cat > recovery-runbook.md <<'EOF'
# Recovery runbook: CI after credential rotation

This runbook is the on-call engineer's reference for
recovering the CI/CD pipeline after a credential
rotation. The runbook covers the staged update of
every system that depends on the rotated credential.

## When to use this runbook

Use this runbook when:

- A credential used by the CI/CD pipeline is rotated
  (for example, the AWS access key in Lab 24).
- The pipeline is broken: deploys fail, syncs fail,
  image pulls fail, Terraform plans fail.
- The team has the new credential value and the
  rotation is verified.

## Stage 1: GitHub Actions secrets

The first system to update is the GitHub Actions
secrets. The deploy workflow fails fast on an
expired credential, so the team restores it first.

gh secret set AWS_ACCESS_KEY_ID
—body “$NEW_AWS_ACCESS_KEY_ID”
—repo runbook-academy/eks-deploy gh secret set AWS_SECRET_ACCESS_KEY
—body “$NEW_AWS_SECRET_ACCESS_KEY”
—repo runbook-academy/eks-deploy


The `gh secret set` command encrypts the value and
stores it in the repository secrets. The next
workflow run uses the new value.

## Stage 2: Argo CD credentials

The second system is the Argo CD credentials. The
GitOps sync uses repository credentials and cluster
credentials.

For the repository credentials:

argocd repo add https://github.com/runbook-academy/eks-deploy
—username runbook-academy-bot
—password “$NEW_GITHUB_TOKEN”


For the cluster credentials:

argocd cluster add production
—kubeconfig “$HOME/.kube/config”


The `argocd cluster add` command reads the
kubeconfig and registers the cluster with Argo CD.

## Stage 3: OCI registry pull secret

The third system is the OCI registry's pull secret in
the production namespace. The Pods use the secret to
pull images.

kubectl create secret docker-registry registry-credentials
—docker-server=ghcr.io
—docker-username=runbook-academy-bot
—docker-password=“$NEW_GHCR_TOKEN”
—namespace=production
—dry-run=client -o yaml | kubectl apply -f -


The `kubectl create secret docker-registry` command
creates or updates the pull secret. The Pods use the
new secret on the next image pull.

## Stage 4: Terraform Cloud

The fourth system is the Terraform Cloud workspace.
The workspace's environment variables include the
AWS credentials.

The on-call engineer updates the workspace via the
TFC UI or the TFC API:

curl -X PATCH
-H “Authorization: Bearer $TFC_TOKEN”
-H “Content-Type: application/vnd.api+json”
—data ’{ “data”: { “attributes”: { “variables”: [ {“key”: “AWS_ACCESS_KEY_ID”, “value”: ”’“$NEW_AWS_ACCESS_KEY_ID”’”, “sensitive”: true}, {“key”: “AWS_SECRET_ACCESS_KEY”, “value”: ”’“$NEW_AWS_SECRET_ACCESS_KEY”’”, “sensitive”: true} ] }, “type”: “vars” } }’
https://app.terraform.io/api/v2/workspaces/$TFC_WORKSPACE_ID/vars


## Stage 5: Jenkins

The fifth system (if still in use) is Jenkins. The
Jenkins credentials store includes the AWS
credentials.

The on-call engineer updates the credential via the
Jenkins UI or the Jenkins CLI:

java -jar jenkins-cli.jar
-s https://jenkins.example.com
-auth admin:$JENKINS_TOKEN
credentials-store
—update
aws-access-key
< new-credentials.xml


## Verification

After each stage, the team runs the smoke test
(`smoke-test.sh`). The smoke test verifies that each
system authenticates with the new credential and
performs its function.

## Post-incident: PR

Within 24 hours of the recovery, the team opens a
PR to update any IaC that hard-codes the credential.
The PR is reviewed and merged; the credential is
managed by IaC, not by manual rotation.

EOF

git add recovery-runbook.md
git commit -m 'incident: recovery runbook'

The runbook is the on-call reference. The five stages are the spine; the CLI commands are the verbs.

Task 3 — Build the GitHub secrets update script

# check-shell-blocks: allow-invalid
cd "$HOME/ci-recover-lab"

cat > update-github-secrets.sh <<'EOF'
#!/usr/bin/env bash
#
# update-github-secrets.sh — update the GitHub Actions
# secrets with the new credential values.
#
# Required: gh CLI authenticated; the new credential
# values in $NEW_AWS_ACCESS_KEY_ID and
# $NEW_AWS_SECRET_ACCESS_KEY.
#
# Usage: REPO=runbook-academy/eks-deploy \
#        NEW_AWS_ACCESS_KEY_ID=... \
#        NEW_AWS_SECRET_ACCESS_KEY=... \
#        ./update-github-secrets.sh

set -euo pipefail

: "${REPO:?REPO is required}"
: "${NEW_AWS_ACCESS_KEY_ID:?NEW_AWS_ACCESS_KEY_ID is required}"
: "${NEW_AWS_SECRET_ACCESS_KEY:?NEW_AWS_SECRET_ACCESS_KEY is required}"

NOW="$(date -u +%Y-%m-%dT%H:%M:%SZ)"

echo "=== updating GitHub Actions secrets for ${REPO} ==="
echo "  now: ${NOW}"

# Update the secrets.
gh secret set AWS_ACCESS_KEY_ID \
  --body "$NEW_AWS_ACCESS_KEY_ID" \
  --repo "$REPO"
gh secret set AWS_SECRET_ACCESS_KEY \
  --body "$NEW_AWS_SECRET_ACCESS_KEY" \
  --repo "$REPO"

# Verify.
echo "=== verifying ==="
gh secret list --repo "$REPO" | \
  grep -E "^(AWS_ACCESS_KEY_ID|AWS_SECRET_ACCESS_KEY)" || \
  echo "WARNING: secrets not visible (expected for secret values)"

echo "secrets updated. trigger a workflow to test."
EOF

chmod +x update-github-secrets.sh

git add update-github-secrets.sh
git commit -m 'incident: update-github-secrets script'

The script updates the GitHub Actions secrets in a controlled procedure. The verification step confirms the secrets are stored; the secrets themselves are never echoed.

Task 4 — Build the Argo CD repository credentials update script

# check-shell-blocks: allow-invalid
cd "$HOME/ci-recover-lab"

cat > update-argocd-repo-cred.sh <<'EOF'
#!/usr/bin/env bash
#
# update-argocd-repo-cred.sh — update the Argo CD
# repository credentials.
#
# Required: argocd CLI authenticated; the new GitHub
# token in $NEW_GITHUB_TOKEN; the repository URL.
#
# Usage: REPO_URL=https://github.com/runbook-academy/eks-deploy \
#        USERNAME=runbook-academy-bot \
#        NEW_GITHUB_TOKEN=... \
#        ./update-argocd-repo-cred.sh

set -euo pipefail

: "${REPO_URL:?REPO_URL is required}"
: "${USERNAME:?USERNAME is required}"
: "${NEW_GITHUB_TOKEN:?NEW_GITHUB_TOKEN is required}"

NOW="$(date -u +%Y-%m-%dT%H:%M:%SZ)"

echo "=== updating Argo CD repo credentials for ${REPO_URL} ==="
echo "  now: ${NOW}"

# Remove the old credential (if it exists).
argocd repo rm --repo "$REPO_URL" 2>/dev/null || true

# Add the new credential.
argocd repo add "$REPO_URL" \
  --username "$USERNAME" \
  --password "$NEW_GITHUB_TOKEN"

# Verify.
echo "=== verifying ==="
argocd repo list | grep "$REPO_URL" || \
  echo "ERROR: repo not in list" >&2

echo "Argo CD repo credentials updated."
EOF

chmod +x update-argocd-repo-cred.sh

git add update-argocd-repo-cred.sh
git commit -m 'incident: update-argocd-repo-cred script'

The script updates the Argo CD repository credentials in a controlled procedure: it removes the old credential (if it exists), adds the new one, and verifies the change.

Task 5 — Build the smoke test

# check-shell-blocks: allow-invalid
cd "$HOME/ci-recover-lab"

cat > smoke-test.sh <<'EOF'
#!/usr/bin/env bash
#
# smoke-test.sh — verify each system authenticates with
# the new credential and performs its function.
#
# The script runs five tests, one per stage of the
# recovery. Each test exits 0 on success, non-zero on
# failure. The output is captured for the evidence
# bundle.

set -uo pipefail

REPO="\${REPO:-runbook-academy/eks-deploy}"
APP="\${APP:-web}"
ARGOCD_NS="\${ARGOCD_NS:-argocd}"
TARGET_NS="\${TARGET_NS:-production}"
EVIDENCE="\${EVIDENCE:-/tmp/smoke-test-output.log}"

: > "$EVIDENCE"

run() {
  local label=="$1"; shift
  echo "=== ${label} ===" | tee -a "$EVIDENCE"
  if "$@"; then
    echo "PASS" | tee -a "$EVIDENCE"
  else
    echo "FAIL" | tee -a "$EVIDENCE"
    return 1
  fi
  echo "" | tee -a "$EVIDENCE"
}

# Test 1: GitHub Actions secrets are updated.
run "Test 1: GitHub Actions secrets" \
  gh secret list --repo "$REPO" >/dev/null

# Test 2: Argo CD repository credentials.
run "Test 2: Argo CD repo creds" \
  bash -c "argocd repo list | grep -q runbook-academy"

# Test 3: Argo CD cluster credentials.
run "Test 3: Argo CD cluster creds" \
  bash -c "argocd cluster list | grep -q production"

# Test 4: EKS kubeconfig is updated.
run "Test 4: EKS kubeconfig" \
  kubectl get nodes

# Test 5: OCI registry pull secret is updated.
run "Test 5: OCI registry pull secret" \
  kubectl get secret registry-credentials \
    -n "$TARGET_NS" -o jsonpath='{.type}' | grep -q "kubernetes.io/dockerconfigjson"

echo "=== smoke test complete. evidence: $EVIDENCE ==="
EOF

chmod +x smoke-test.sh

git add smoke-test.sh
git commit -m 'incident: smoke test for recovery'

The smoke test verifies each of the five stages. The output is captured for the evidence bundle. The test fails loudly if any stage does not authenticate with the new credential.

Task 6 — Run the recovery

cd "$HOME/ci-recover-lab"

REPO=runbook-academy/eks-deploy
NEW_AWS_ACCESS_KEY_ID="\${NEW_AWS_ACCESS_KEY_ID:-NEW-KEY-ID-PLACEHOLDER}"
NEW_AWS_SECRET_ACCESS_KEY="\${NEW_AWS_SECRET_ACCESS_KEY:-NEW-SECRET-PLACEHOLDER}"

# Stage 1: update GitHub secrets.
REPO="$REPO" \
  NEW_AWS_ACCESS_KEY_ID="$NEW_AWS_ACCESS_KEY_ID" \
  NEW_AWS_SECRET_ACCESS_KEY="$NEW_AWS_SECRET_ACCESS_KEY" \
  ./update-github-secrets.sh

# Stage 2: update Argo CD repo creds.
REPO_URL="https://\${REPO}" \
  USERNAME="runbook-academy-bot" \
  NEW_GITHUB_TOKEN="\${NEW_GITHUB_TOKEN:-NEW-TOKEN-PLACEHOLDER}" \
  ./update-argocd-repo-cred.sh

# Stage 3: update OCI registry pull secret.
kubectl create secret docker-registry registry-credentials \
  --docker-server=ghcr.io \
  --docker-username=runbook-academy-bot \
  --docker-password="\${NEW_GHCR_TOKEN:-NEW-GHCR-TOKEN-PLACEHOLDER}" \
  --namespace=production \
  --dry-run=client -o yaml | kubectl apply -f -

# Stage 4: update Terraform Cloud (manual or via API).
# (TFC API call from the runbook; not executed in the lab)

# Stage 5: update Jenkins (manual or via CLI).
# (Jenkins CLI call from the runbook; not executed in the lab)

# Run the smoke test.
./smoke-test.sh

The recovery walks through all five stages. The smoke test verifies the recovery; the evidence is captured in Task 7.

Task 7 — Capture the recovery evidence

# check-shell-blocks: allow-invalid
cd "$HOME/ci-recover-lab"

cat > recovery-evidence.md <<'EOF'
# Recovery evidence: CI after credential rotation — 2026-08-22

This document is the canonical evidence bundle for
the CI recovery performed on 2026-08-22. The bundle
captures the inventory, the update actions, the smoke
test, and the post-recovery state.

## Pre-recovery state

- Rotated credential: AWS access key
  `AKIA...REDACTED...`.
- Affected systems: 6 (see ci-systems-inventory.md).
- CI status: broken. Deploys failing with
  `ExpiredTokenException`.

## Recovery actions

### Stage 1: GitHub Actions secrets

- 09:30 UTC: updated `AWS_ACCESS_KEY_ID` and
  `AWS_SECRET_ACCESS_KEY` in the
  `runbook-academy/eks-deploy` repository.
- 09:32 UTC: triggered a workflow run; the workflow
  authenticated successfully.
- 09:35 UTC: first successful deploy post-recovery.

### Stage 2: Argo CD credentials

- 09:40 UTC: updated the Argo CD repository
  credentials (GitHub token).
- 09:45 UTC: updated the Argo CD cluster credentials
  (kubeconfig).
- 09:50 UTC: triggered a sync; the sync succeeded.

### Stage 3: OCI registry pull secret

- 09:55 UTC: updated the `registry-credentials`
  Secret in the `production` namespace.
- 10:00 UTC: triggered a Pod restart; the new Pods
  pulled images successfully.

### Stage 4: Terraform Cloud

- 10:05 UTC: updated the workspace environment
  variables via the TFC API.
- 10:10 UTC: ran a `terraform plan`; the plan
  succeeded.

### Stage 5: Jenkins

- 10:15 UTC: updated the Jenkins credentials store.
- 10:20 UTC: ran a Jenkins pipeline; the pipeline
  succeeded.

## Smoke test results

=== Test 1: GitHub Actions secrets === PASS === Test 2: Argo CD repo creds === PASS === Test 3: Argo CD cluster creds === PASS === Test 4: EKS kubeconfig === PASS === Test 5: OCI registry pull secret === PASS


All five tests passed.

## Post-recovery state

- CI status: operational.
- Deploys: succeeding.
- GitOps syncs: succeeding.
- Image pulls: succeeding.
- Terraform plans: succeeding.
- Jenkins pipelines: succeeding.

## Verification

- `gh secret list` shows the new secrets.
- `argocd repo list` shows the new credentials.
- `argocd cluster list` shows the new cluster.
- `kubectl get nodes` returns the cluster nodes.
- `kubectl get secret registry-credentials` returns
  the new pull secret.

EOF

git add recovery-evidence.md
git commit -m 'incident: recovery evidence bundle'

The evidence bundle is the canonical record. The pre-recovery state, the recovery actions, the smoke test, the post-recovery state, and the verification are the fields the team reviews at the post-incident review.

Task 8 — Build the audit trail

# check-shell-blocks: allow-invalid
cd "$HOME/ci-recover-lab"

cat > audit-trail.md <<'EOF'
# Audit trail: CI recovery after credential rotation

This document is the audit trail for the CI recovery
on 2026-08-22. The trail is the canonical record for
the compliance review; every action is timestamped
and attributed.

## 09:15 — Page fired

- Source: PagerDuty.
- Alert: deploy workflow failure
  (`ExpiredTokenException`).
- Operator: jane.doe.

## 09:18 — Acknowledged

- Operator: jane.doe.
- Action: opened the workflow run; confirmed the
  `ExpiredTokenException`.

## 09:22 — Inventory created

- Operator: jane.doe.
- Action: created ci-systems-inventory.md.
- Verification: 6 systems listed; all have an update
  method.

## 09:30 — GitHub Actions secrets updated

- Operator: jane.doe.
- Action: `gh secret set` for `AWS_ACCESS_KEY_ID`
  and `AWS_SECRET_ACCESS_KEY`.
- Verification: `gh secret list` shows the secrets
  (values are encrypted).

## 09:35 — First successful deploy

- Operator: ci-bot.
- Action: triggered a workflow run; the deploy
  succeeded.
- Verification: `web` is `Synced: True, Healthy:
  True`.

## 09:40 — Argo CD repo creds updated

- Operator: jane.doe.
- Action: `argocd repo add` with the new GitHub
  token.

## 09:45 — Argo CD cluster creds updated

- Operator: jane.doe.
- Action: `argocd cluster add production` with the
  new kubeconfig.

## 09:50 — Argo CD sync

- Operator: jane.doe.
- Action: triggered a sync; the sync succeeded.
- Verification: `argocd app list` shows all apps
  `Synced: True`.

## 09:55 — OCI registry pull secret updated

- Operator: jane.doe.
- Action: `kubectl create secret docker-registry
  registry-credentials` in the `production`
  namespace.

## 10:00 — Pod restart

- Operator: jane.doe.
- Action: rolled the `web` Deployment; the new Pods
  pulled images successfully.

## 10:05 — Terraform Cloud updated

- Operator: jane.doe.
- Action: TFC API call to update the workspace
  environment variables.

## 10:10 — Terraform plan

- Operator: ci-bot.
- Action: ran a `terraform plan`; the plan
  succeeded.

## 10:15 — Jenkins updated

- Operator: jane.doe.
- Action: updated the Jenkins credentials store via
  the CLI.

## 10:20 — Jenkins pipeline

- Operator: ci-bot.
- Action: ran a Jenkins pipeline; the pipeline
  succeeded.

## 10:30 — Smoke test

- Operator: jane.doe.
- Action: ran smoke-test.sh.
- Verification: 5/5 tests passed.

## 10:35 — Post-incident review

- Operators: jane.doe, sre-team.
- Action: reviewed the evidence bundle and the audit
  trail; merged the IaC PR for the credential.

EOF

git add audit-trail.md
git commit -m 'incident: audit trail'

The audit trail is the canonical record. Every action is timestamped and attributed; the trail is the answer to “who did what, when?”.

Task 9 — Validate the deliverables

cd "$HOME/ci-recover-lab"

# Verify the inventory has all systems.
grep -c "^| " ci-systems-inventory.md
# expected: 10+ rows

# Verify the runbook has all five stages.
grep -c "^## Stage" recovery-runbook.md
# expected: 5

# Verify the scripts pass syntax check.
bash -n update-github-secrets.sh && echo "github: syntax ok"
bash -n update-argocd-repo-cred.sh && echo "argocd: syntax ok"
bash -n smoke-test.sh && echo "smoke: syntax ok"

# Verify the smoke test runs.
./smoke-test.sh || echo "smoke test reported failures (expected in simulation)"

# Verify the evidence bundle has all sections.
grep -c "^## " recovery-evidence.md
# expected: 5 (Pre-recovery, Recovery actions, Smoke
#              test, Post-recovery, Verification)

# Verify the audit trail has timestamps.
grep -c "^## [0-9]" audit-trail.md
# expected: 15+ entries

The deliverables are validated: the inventory has all systems, the runbook has all five stages, the scripts pass syntax check, the smoke test runs, the evidence has all sections, and the audit trail has timestamps.

Task 10 — Capture the deliverables

cd "$HOME/ci-recover-lab"

cp ci-systems-inventory.md \
   recovery-runbook.md \
   update-github-secrets.sh \
   update-argocd-repo-cred.sh \
   smoke-test.sh \
   recovery-evidence.md \
   audit-trail.md \
   "$HOME/"

ls -l "$HOME"/ci-systems-inventory.md \
       "$HOME"/recovery-runbook.md \
       "$HOME"/update-github-secrets.sh \
       "$HOME"/update-argocd-repo-cred.sh \
       "$HOME"/smoke-test.sh \
       "$HOME"/recovery-evidence.md \
       "$HOME"/audit-trail.md

The deliverables are in $HOME/.

Validation

  • ci-systems-inventory.md lists every CI system that depends on the rotated credential with an update method.
  • recovery-runbook.md documents all five stages: GitHub Actions, Argo CD, OCI registry, Terraform Cloud, Jenkins.
  • update-github-secrets.sh is executable, has set -euo pipefail, and updates the GitHub Actions secrets.
  • update-argocd-repo-cred.sh is executable, has set -euo pipefail, and updates the Argo CD repository credentials.
  • smoke-test.sh is executable, has set -uo pipefail, and tests each of the five stages.
  • recovery-evidence.md captures the pre-recovery state, the recovery actions, the smoke test results, and the post-recovery state.
  • audit-trail.md has timestamped entries for every action.

Expected Outcome

A recovery runbook, a CI systems inventory, two update scripts, a smoke test, a recovery evidence bundle, and an audit trail.

$HOME/ci-recover-lab/
├── ci-systems-inventory.md       # systems + update methods
├── recovery-runbook.md           # the runbook
├── update-github-secrets.sh      # stage 1
├── update-argocd-repo-cred.sh    # stage 2
├── smoke-test.sh                 # verification
├── recovery-evidence.md          # the canonical record
└── audit-trail.md                # the audit trail

The inventory is the safety net; the runbook is the on-call reference; the scripts are the verbs; the smoke test is the verification; the evidence and the audit trail are the institutional knowledge.

Troubleshooting

The GitHub secret is not visible after gh secret set. The value is encrypted; the name is visible. Use gh secret list to verify the name; the value is only visible to the workflow.

The Argo CD repo credential is rejected. The GitHub token may be expired or revoked. Verify with gh auth status and regenerate the token.

The EKS kubeconfig is invalid. The IAM role session may have expired. Run aws eks update-kubeconfig --name production --region us-east-1 to refresh.

The OCI registry pull secret is rejected. The GHCR token may be expired. Verify with crane auth login and regenerate the token.

The smoke test fails on Test 4 (kubeconfig). The kubeconfig may be using the old access key. Refresh with aws eks update-kubeconfig.

Cleanup

LAB="$HOME/ci-recover-lab"

cp -r "$LAB"/* "$HOME"/ 2>/dev/null
rm -rf "$LAB"

# Revert any test changes to the cluster.
kubectl delete secret registry-credentials -n production 2>/dev/null

If the kind cluster is no longer needed, delete it:

kind delete cluster --name argocd-lab

What You Learned

  • The inventory is the safety net. A recovery that misses a system leaves the system broken silently. The inventory lists every system with an update method and a verification step.
  • The recovery is staged. The on-call engineer updates each system in order: GitHub Actions first (the workflow fails fast), then Argo CD, then the cluster, then the Pods. The order minimises the time-to-recovery.
  • The smoke test is the verification. Each stage is verified end-to-end. The smoke test runs all five verifications; the output is the evidence that the recovery succeeded.
  • The audit trail is the canonical record. Every action is timestamped and attributed. The trail is the answer to “who did what, when?”.
  • Secret values are handled in memory only. The scripts never echo the secret value. The verification step greps for the secret name; the value is encrypted.
  • The recovery is followed by an IaC PR. Manual rotations are error-prone; IaC-managed credentials are reproducible. The team opens a PR to manage the credential in code.
  • A broken CI is a P1 incident. The recovery runbook is maintained and tested; the inventory is updated with every rotation; the smoke test is run after every recovery.

Deliverables

  • · recovery-runbook.md — the on-call runbook for recovering CI after a credential rotation
  • · ci-systems-inventory.md — the inventory of CI systems that use the rotated credential
  • · update-github-secrets.sh — the script that updates the GitHub Actions secrets
  • · update-argocd-repo-cred.sh — the script that updates the Argo CD repository credentials
  • · smoke-test.sh — the smoke test for the recovery
  • · recovery-evidence.md — the evidence bundle for the recovery
  • · audit-trail.md — the audit trail for the incident review

Verification status

Last reviewed
2026-08-25
Executed end to end
not yet run on hardware

The commands and configuration here have been reviewed against the verified software versions, but nobody has run this lab start to finish on a system meeting its prerequisites. Treat the Expected Outcome as the intended result rather than an observed one, and keep the Cleanup section to hand.