Objective
By the end of this lab you will have authored the artefacts that perform a GitOps rollback through Argo CD: the rollback runbook, the rollback script that automates the history inspection and the validation, the evidence bundle for the scenario, the post-rollback investigation document, the decision framework, and the history inspection notes.
The point of this lab is not the argocd app rollback
command — that is one CLI invocation. The point is the
operationalisation: identifying the right revision,
validating that the rollback succeeded, capturing the
evidence, and writing the post-rollback investigation
that links the regression to the deploy.
Architecture
A kind cluster with Argo CD managing a web
application. The application has three deployment history
entries: revision 1 (initial), revision 2 (scaled to 3
replicas and updated image to 1.27-alpine), revision 3
(changed image to 1.28-alpine and added a label). The
team has detected that revision 3 introduced a
regression (the new image has a memory leak). The
on-call engineer rolls back to revision 2 (the
known-good state), validates the cluster, and writes the
post-rollback investigation.
sequenceDiagram
participant E as on-call engineer
participant A as Argo CD
participant K as cluster
participant G as Git repo
E->>A: argocd app history web
A-->>E: 3 entries: id=1, id=2, id=3
E->>A: argocd app set web --sync-policy none
E->>A: argocd app rollback web 2
A->>G: render revision 2 manifests
A->>K: apply revision 2 manifests
K-->>E: pods restart with revision 2 image
E->>A: argocd app get web
A-->>E: Synced: True, Healthy: True
E->>G: post-rollback investigation PR
E->>A: argocd app set web --sync-policy automatedThe rollback uses Argo CD’s history (which records every
sync with the source revision, the deployed manifests,
and the deploy time). The on-call engineer chooses the
known-good revision; Argo CD renders the manifests and
applies them. The history ID is a positional argument to
argocd app rollback, not a flag. Because the web
application uses automated sync, the engineer disables it
first (argocd app set web --sync-policy none) — Argo CD
refuses to roll back while automated sync is enabled — and
re-enables it only after the follow-up Git PR reconciles
the branch. The Git repository is not modified by the
rollback; the rollback is a cluster-side action.
Requirements
- A
kindcluster with Argo CD installed (Lab 19). - A sample application with at least three deployment history entries. The lab sets this up in Task 1.
- The
argocdCLI authenticated against the cluster. jqon the workstation.
Scenario
A platform team runs a web application managed by
Argo CD. The team deploys three times in a day:
revision 1 (initial, nginx:1.27-alpine, 2 replicas),
revision 2 (scaled to 3 replicas, image unchanged),
revision 3 (changed image to nginx:1.28-alpine, added
a label). After revision 3 deploys, the team notices
that the web Pods are being OOMKilled and the
application health is Degraded. The on-call engineer
inspects the history, rolls back to revision 2, and
writes the post-rollback investigation.
Tasks
Task 1 — Build three deployment history entries
LAB="$HOME/argocd-rollback-lab"
rm -rf "$LAB"
mkdir -p "$LAB"
cd "$LAB"
git init -b main
git config user.email 'ops@example.com'
git config user.name 'Ops'
mkdir -p app-source
# Revision 1: initial.
cat > app-source/web-deployment.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: web
labels:
app: web
spec:
replicas: 2
selector:
matchLabels:
app: web
template:
metadata:
labels:
app: web
spec:
containers:
- name: web
image: nginx:1.27-alpine
resources:
limits:
memory: 256Mi
ports:
- containerPort: 80
EOF
cat > app-source/web-service.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata:
name: web
spec:
selector:
app: web
ports:
- port: 80
targetPort: 80
EOF
cat > application.yaml <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: web
namespace: argocd
spec:
project: default
source:
repoURL: https://github.com/runbook-academy/argocd-rollback-lab.git
targetRevision: main
path: app-source
destination:
server: https://kubernetes.default.svc
namespace: production
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
EOF
git add application.yaml app-source/
git commit -m 'r1: initial deploy (nginx:1.27-alpine, 2 replicas)'
# Apply the Application CR and wait for sync.
kubectl apply -f application.yaml
sleep 45
argocd app sync web
sleep 30
# Revision 2: scale to 3 replicas.
cat > app-source/web-deployment.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: web
labels:
app: web
spec:
replicas: 3
selector:
matchLabels:
app: web
template:
metadata:
labels:
app: web
spec:
containers:
- name: web
image: nginx:1.27-alpine
resources:
limits:
memory: 256Mi
ports:
- containerPort: 80
EOF
git add app-source/web-deployment.yaml
git commit -m 'r2: scale to 3 replicas'
sleep 45
argocd app sync web
sleep 30
# Revision 3: change image to 1.28-alpine and add a label.
cat > app-source/web-deployment.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: web
labels:
app: web
version: v1.28
spec:
replicas: 3
selector:
matchLabels:
app: web
template:
metadata:
labels:
app: web
version: v1.28
spec:
containers:
- name: web
image: nginx:1.28-alpine
resources:
limits:
memory: 256Mi
ports:
- containerPort: 80
EOF
git add app-source/web-deployment.yaml
git commit -m 'r3: nginx 1.28-alpine (regression: OOMKilled)'
sleep 45
argocd app sync web
sleep 30
# Verify the history.
argocd app history webThe lab’s setup creates three sync history entries.
The output of argocd app history web lists each entry
with its ID, the source revision, the deployer, and
the timestamp.
Task 2 — Inspect the application history
# check-shell-blocks: allow-invalid
cd "$HOME/argocd-rollback-lab"
cat > history-inspection.md <<'EOF'
# History inspection: web — 2026-08-22
This document is the on-call engineer's record of the
Argo CD history for the `web` application at the time
of the incident. The history is the input to the
rollback decision.
## History entries
| ID | Source revision | Deployer | Deployed at | Healthy |
|----|-----------------|----------|-------------|---------|
| 3 | r3 (1.28-alpine) | ci-bot | 2026-08-22 13:45 | Degraded |
| 2 | r2 (3 replicas) | ci-bot | 2026-08-22 12:30 | Healthy |
| 1 | r1 (initial) | ci-bot | 2026-08-22 09:00 | Healthy |
## Chosen revision
The on-call engineer chooses **ID 2** (revision r2) as
the rollback target. Rationale:
- ID 2 is the last known-good state (image
`nginx:1.27-alpine`, 3 replicas, no extra label).
- ID 1 is also known-good but has only 2 replicas;
ID 2 has 3 (the team had scaled up for the
deployment).
- ID 3 is the offending state (image `nginx:1.28-alpine`
with the regression).
## Verification before rollback
The on-call engineer captures the current cluster state
before the rollback:
$ kubectl get deployment web -n production -o yaml spec.replicas: 3 spec.template.spec.containers[0].image: nginx:1.28-alpine spec.template.metadata.labels.version: v1.28 status.readyReplicas: 1 (degraded; 2 OOMKilled)
The current state is ID 3; the desired state is ID 2.
EOF
git add history-inspection.md
git commit -m 'rollback: history inspection'The history inspection document is the on-call record. The chosen revision is ID 2; the rationale is captured in the document.
Task 3 — Build the rollback runbook
# check-shell-blocks: allow-invalid
cd "$HOME/argocd-rollback-lab"
cat > rollback-runbook.md <<'EOF'
# Rollback runbook: `argocd app rollback`
This runbook is the on-call engineer's reference for
rolling back a Kubernetes application managed by Argo
CD. The runbook is intentionally short: under pressure,
the engineer follows four steps (inspect, choose,
rollback, validate) and the team has a review at the
end of the incident.
## When to use this runbook
Use this runbook when:
- A regression was introduced by a recent deploy and
the cluster state is unhealthy.
- The fix is to revert the cluster to a previous
deployment (not to forward-fix).
- The application is managed by Argo CD (otherwise,
use `kubectl rollout undo` or a Git revert).
Do not use this runbook when:
- The regression is a data integrity issue. Rollback
does not revert database state; a database
rollback or a forward-fix migration is required.
- The cluster has drifted from Git in a way that
would cause the rollback to fail. Run the drift
detection (Lab 21) first.
- The fix is permanent. Permanent fixes go through
Git (a revert or a forward-fix PR).
## Step 1: inspect the history
argocd app history <APP_NAME>
The output is a table of sync entries. Each entry has
an ID, a source revision, a deployer, and a timestamp.
The on-call engineer notes the entry that is the
"last known-good" state.
## Step 2: choose the revision
The on-call engineer chooses the ID of the last
known-good state. The decision framework
(`rollback-decision-tree.md`) covers the cases:
- The previous entry is known-good: roll back to it.
- The previous entry is also unhealthy: roll back to
the most recent known-good entry.
- No known-good entry exists: roll back is not the
right answer; consider a forward-fix or a database
rollback.
## Step 3: rollback
argocd app set <APP_NAME> —sync-policy none argocd app rollback <APP_NAME> <ID>
Argo CD refuses to roll back while automated sync is
enabled, so the first command disables it. The second
command takes the history ID as a positional argument;
it renders the manifests from the chosen revision and
applies them. The cluster is reverted without
modifying Git. The rollback is recorded in Argo CD's
history as a new entry (with the deployer "admin" or
the engineer's name). Automated sync stays off until
the post-incident Git PR reconciles the branch;
re-enabling it earlier re-applies the offending
revision.
If the rollback fails (for example, due to an
immutable field), the engineer runs
`argocd app diff <APP_NAME>` to see the diff, and
applies the change manually with `kubectl`.
## Step 4: validate
argocd app get <APP_NAME> kubectl get deployment <DEPLOYMENT_NAME> -n <NAMESPACE>
The application must reach `Synced: True, Healthy:
True`. The cluster state must match the chosen
revision (image tag, replicas, env vars). The
on-call engineer captures the before/after in the
evidence bundle (`rollback-evidence.md`).
## Post-incident: Git PR
Within 24 hours of the rollback, the team opens a Git
PR to either revert the offending commit or to fix the
regression. The PR is the permanent record; the
rollback is the immediate mitigation. After the PR
merges, re-enable automated sync:
argocd app set <APP_NAME> —sync-policy automated
EOF
git add rollback-runbook.md
git commit -m 'rollback: runbook'The runbook is the on-call reference. The four steps
are the spine; the argocd and kubectl commands
are the verbs.
Task 4 — Build the rollback script
# check-shell-blocks: allow-invalid
cd "$HOME/argocd-rollback-lab"
cat > argocd-rollback.sh <<'EOF'
#!/usr/bin/env bash
#
# argocd-rollback.sh — perform a GitOps rollback via
# `argocd app rollback` and validate the cluster state.
#
# Required: argocd CLI authenticated; jq; kubectl;
# a Slack webhook URL in $SLACK_WEBHOOK_URL.
#
# Usage: APP=web TARGET_ID=2 OPERATOR=jane.doe \
# DEPLOYMENT_NAME=web NAMESPACE=production \
# ./argocd-rollback.sh
set -euo pipefail
: "${APP:?APP is required}"
: "${TARGET_ID:?TARGET_ID is required}"
: "${OPERATOR:?OPERATOR is required}"
DEPLOYMENT_NAME="\${DEPLOYMENT_NAME:-$APP}"
NAMESPACE="\${NAMESPACE:-production}"
SLACK_WEBHOOK_URL="\${SLACK_WEBHOOK_URL:-}"
NOW="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
echo "=== rolling back ${APP} to history id ${TARGET_ID} ==="
echo " now: ${NOW}"
echo " operator: ${OPERATOR}"
# Capture before-state.
BEFORE_IMAGE="$(kubectl get deployment "$DEPLOYMENT_NAME" \
-n "$NAMESPACE" -o jsonpath='{.spec.template.spec.containers[0].image}')"
BEFORE_REPLICAS="$(kubectl get deployment "$DEPLOYMENT_NAME" \
-n "$NAMESPACE" -o jsonpath='{.spec.replicas}')"
BEFORE_READY="$(kubectl get deployment "$DEPLOYMENT_NAME" \
-n "$NAMESPACE" -o jsonpath='{.status.readyReplicas}')"
echo "before:"
echo " image: ${BEFORE_IMAGE}"
echo " replicas: ${BEFORE_REPLICAS}"
echo " ready: ${BEFORE_READY}"
# Inspect history.
echo "=== history ==="
argocd app history "$APP"
# Disable automated sync: `argocd app rollback` refuses to
# run while automated sync is enabled, and automated sync
# would re-apply the offending revision on the next tick.
argocd app set "$APP" --sync-policy none
# Roll back (the history ID is a positional argument).
argocd app rollback "$APP" "$TARGET_ID"
# Wait for sync.
echo "waiting for sync..."
sleep 30
# Wait for Healthy.
for i in $(seq 1 30); do
HEALTH="$(argocd app get "$APP" -o json | \
jq -r '.status.health.status // "Unknown"')"
SYNC="$(argocd app get "$APP" -o json | \
jq -r '.status.sync.status // "Unknown"')"
echo " attempt ${i}: sync=${SYNC} health=${HEALTH}"
if [ "$SYNC" = "Synced" ] && [ "$HEALTH" = "Healthy" ]; then
break
fi
sleep 10
done
if [ "$SYNC" != "Synced" ] || [ "$HEALTH" != "Healthy" ]; then
echo "ERROR: ${APP} did not reach Synced+Healthy in 5 minutes" >&2
exit 1
fi
# Capture after-state.
AFTER_IMAGE="$(kubectl get deployment "$DEPLOYMENT_NAME" \
-n "$NAMESPACE" -o jsonpath='{.spec.template.spec.containers[0].image}')"
AFTER_REPLICAS="$(kubectl get deployment "$DEPLOYMENT_NAME" \
-n "$NAMESPACE" -o jsonpath='{.spec.replicas}')"
AFTER_READY="$(kubectl get deployment "$DEPLOYMENT_NAME" \
-n "$NAMESPACE" -o jsonpath='{.status.readyReplicas}')"
echo "after:"
echo " image: ${AFTER_IMAGE}"
echo " replicas: ${AFTER_REPLICAS}"
echo " ready: ${AFTER_READY}"
# Announce in Slack.
if [ -n "$SLACK_WEBHOOK_URL" ]; then
curl -s -X POST -H 'Content-Type: application/json' \
--data "$(jq -n \
--arg app "$APP" \
--arg id "$TARGET_ID" \
--arg ts "$NOW" \
--arg op "$OPERATOR" \
--arg bi "$BEFORE_IMAGE" \
--arg ai "$AFTER_IMAGE" \
'{text: ("[ROLLBACK] rolled back `\` + $app + "` to history id " + $id + " at " + $ts + ". Operator: " + $op + ". Image: " + $bi + " -> " + $ai)}')" \
"$SLACK_WEBHOOK_URL" >/dev/null
fi
echo "rollback complete; ${APP} is Synced+Healthy at history id ${TARGET_ID}."
echo "NOTE: automated sync is DISABLED for ${APP}. After the Git revert/fix"
echo "PR merges, re-enable it: argocd app set ${APP} --sync-policy automated"
EOF
chmod +x argocd-rollback.sh
git add argocd-rollback.sh
git commit -m 'rollback: automated rollback script'The script automates the four steps of the runbook:
capture before-state, inspect history, rollback, and
validate. It disables automated sync before the
rollback (Argo CD refuses to roll back while automated
sync is enabled) and leaves it disabled: re-enabling
happens only after the follow-up Git PR merges. The
script fails loudly if the application does not reach
Synced+Healthy within 5 minutes.
Task 5 — Perform the rollback
cd "$HOME/argocd-rollback-lab"
APP=web
TARGET_ID=2
OPERATOR=jane.doe
DEPLOYMENT_NAME=web
NAMESPACE=production
SLACK_WEBHOOK_URL="\${SLACK_WEBHOOK_URL:-}"
APP="$APP" TARGET_ID="$TARGET_ID" OPERATOR="$OPERATOR" \
DEPLOYMENT_NAME="$DEPLOYMENT_NAME" NAMESPACE="$NAMESPACE" \
SLACK_WEBHOOK_URL="$SLACK_WEBHOOK_URL" \
./argocd-rollback.shThe script performs the rollback, validates the cluster, and announces in Slack (if configured). The output is captured in Task 6’s evidence bundle.
Task 6 — Capture the rollback evidence
# check-shell-blocks: allow-invalid
cd "$HOME/argocd-rollback-lab"
cat > rollback-evidence.md <<'EOF'
# Rollback evidence: web — 2026-08-22
This document is the canonical evidence bundle for the
rollback performed at 14:55 UTC on 2026-08-22. The
bundle captures the before-state, the rollback action,
the after-state, and the validation.
## Before-state
$ kubectl get deployment web -n production -o yaml spec.replicas: 3 spec.template.spec.containers[0].image: nginx:1.28-alpine spec.template.metadata.labels.version: v1.28 status.readyReplicas: 1 status.conditions:
- type: Progressing status: “False” reason: ProgressDeadlineExceeded
The cluster was unhealthy: 1 of 3 replicas ready, 2
OOMKilled. The image was `nginx:1.28-alpine`.
## Rollback action
$ argocd app set web —sync-policy none $ argocd app rollback web 2
Automated sync was disabled first — Argo CD refuses
to roll back while it is enabled, and automated sync
would re-apply the offending revision on the next
tick. The rollback command (history id 2 as a
positional argument) rendered the manifests from
revision r2 and applied them to the cluster. The
rollback was recorded in the history as a new entry
with deployer "admin" (jane.doe).
## After-state
$ kubectl get deployment web -n production -o yaml spec.replicas: 3 spec.template.spec.containers[0].image: nginx:1.27-alpine spec.template.metadata.labels: (no version label) status.readyReplicas: 3 status.conditions:
- type: Available status: “True”
The cluster is healthy: 3 of 3 replicas ready, image
back to `nginx:1.27-alpine`.
## Validation
$ argocd app get web Name: web Project: default Server: https://kubernetes.default.svc Namespace: production Sync Status: Synced Health Status: Healthy
$ argocd app history web ID DATE REVISION DEPLOYER 3 2026-08-22 14:55:00 r2 (rollback) jane.doe 2 2026-08-22 12:30:00 r2 ci-bot 1 2026-08-22 09:00:00 r1 ci-bot
The application is `Synced: True, Healthy: True`. The
history shows the rollback as a new entry (ID 3 with
deployer "jane.doe"). The Git repository remains at
revision r3 (the offending commit is not reverted; the
PR is opened in Task 7). Automated sync remains
disabled until that PR merges; it is then re-enabled
with `argocd app set web --sync-policy automated`.
EOF
git add rollback-evidence.md
git commit -m 'rollback: evidence bundle'The evidence bundle is the on-call record. The before/after and the validation are the artefacts the team reviews at the post-incident review.
Task 7 — Author the post-rollback investigation
# check-shell-blocks: allow-invalid
cd "$HOME/argocd-rollback-lab"
cat > post-rollback-investigation.md <<'EOF'
# Post-rollback investigation: web — 2026-08-22
This document is the team's investigation of the
regression that triggered the rollback. The
investigation links the regression to the offending
commit and identifies the root cause.
## Summary
The team rolled back the `web` application from
revision r3 (`nginx:1.28-alpine`) to revision r2
(`nginx:1.27-alpine`) at 14:55 UTC on 2026-08-22.
The regression was a memory leak in `nginx:1.28-alpine`
that caused the Pods to OOMKill under load. The
rollback restored the application to a known-good
state.
## Timeline
- 13:45 — r3 deployed (`nginx:1.28-alpine`).
- 14:30 — first OOMKill observed.
- 14:35 — PagerDuty page fired.
- 14:40 — on-call engineer (jane.doe) acknowledged.
- 14:50 — investigation complete; regression linked to
image change.
- 14:55 — rollback to history id 2 performed.
- 15:00 — cluster healthy; incident resolved.
- 15:30 — post-rollback investigation opened.
## Root cause
The `nginx:1.28-alpine` image has a known memory leak
in the gzip module (CVE-2026-1234). The leak is
triggered when the module is enabled and the request
rate is high. The team's `web` application enables the
gzip module by default; the production traffic rate
triggered the leak within 4 hours of deploy.
## Detection
The detection was within the team's SLO (60 seconds
from first OOMKill to the page). The alert is
`kube_pod_container_status_terminated_reason ==
"OOMKilled"`. The alert has been in place since
2026-01; the team has a runbook for it.
## Response
The on-call engineer chose rollback over forward-fix
because:
- The fix (image bump to `nginx:1.28.1-alpine`) was
not yet built or signed.
- The customer impact was 35% error rate; rollback
was the fastest path to recovery.
- The data integrity question was not relevant
(the regression was HTTP 5xx, not data loss).
The rollback was performed in 5 minutes (history
inspection, rollback command, validation). The team's
SLO for rollback is 15 minutes; the actual time was
within SLO.
## Permanent fix
The team opens a PR to update the image to
`nginx:1.28.1-alpine` (which fixes the memory leak).
The PR is reviewed and merged; Argo CD syncs the new
image; the history records the new revision.
## Lessons learned
1. The team's pre-prod load test should have caught
the memory leak. The test runs at 2× production
traffic; the leak triggered at 4×. The test
threshold is increased to 5×.
2. The team did not have a `pinned` image tag; the
`1.28-alpine` tag moved without notice. The team
pins image tags to specific digests.
3. The team's CVE subscription did not include
nginx. The team subscribes to the nginx security
announcements.
## Action items
- [x] jane.doe: rollback (done at 14:55).
- [ ] platform-team: increase pre-prod load test
threshold to 5× (due 2026-08-29).
- [ ] platform-team: pin all image tags to digests
(due 2026-09-12).
- [ ] sre-team: subscribe to nginx security
announcements (due 2026-08-25).
EOF
git add post-rollback-investigation.md
git commit -m 'rollback: post-rollback investigation'The post-rollback investigation is the team’s record. The root cause, the response, the permanent fix, and the lessons learned are the fields the team reviews at the post-incident review.
Task 8 — Build the rollback decision tree
# check-shell-blocks: allow-invalid
cd "$HOME/argocd-rollback-lab"
cat > rollback-decision-tree.md <<'EOF'
# Rollback decision tree
This document is the team's reference for choosing
between rollback, revert, and forward-fix. The decision
is made under pressure; the tree is the on-call
engineer's guide.
## Question 1: is the cluster state recoverable by
rollback?
If the cluster state is recoverable by re-applying a
previous Git revision, rollback is on the table. If
not (for example, a PersistentVolume was resized and
cannot be shrunk), rollback is not the right answer.
## Question 2: is the data integrity at risk?
Rollback does not revert database state. If the
regression involved a database migration or a data
corruption, rollback is not the right answer. The
on-call engineer considers:
- Forward-fix migration: apply a new migration that
undoes the offending one.
- Database snapshot restore: restore the database
from a pre-deploy snapshot.
- Manual data repair: identify and repair the
corrupted data.
## Question 3: is the fix in a known-good image or
manifest?
If the fix is in a new image (for example, a
patched `nginx:1.28.1-alpine`) and the image is built,
signed, and pushed, forward-fix is faster than
rollback. The on-call engineer:
- Builds the fix in the CI pipeline.
- Promotes the image through staging → production.
- The new deploy replaces the offending revision.
If the fix is not yet built, rollback is the path.
## Question 4: is the regression customer-visible?
If the regression is customer-visible (HTTP 5xx, data
corruption), rollback is the priority. The team's SLO
for customer-visible incidents is 15 minutes; the
rollback path is faster than the forward-fix path.
If the regression is internal-only (for example, a
metric that is not collected), the team can take
the forward-fix path.
## Decision matrix
| Cluster recoverable | Data integrity | Fix available | Customer-visible | Decision |
|---------------------|----------------|---------------|------------------|----------|
| Yes | No | Yes | Yes | Forward-fix (fast) |
| Yes | No | Yes | No | Forward-fix |
| Yes | No | No | Yes | Rollback |
| Yes | No | No | No | Rollback or forward-fix |
| Yes | Yes | (any) | (any) | Database snapshot + forward-fix |
| No | (any) | (any) | (any) | Forward-fix or restore |
## Default
The team's default is rollback when in doubt. The
rollback is faster (5 minutes vs 30 minutes for
forward-fix) and the Git PR is opened in parallel.
The team's discipline: rollback first, postmortem
second, PR third.
EOF
git add rollback-decision-tree.md
git commit -m 'rollback: decision tree'The decision tree is the on-call reference. The matrix at the bottom is the answer to “what do I do when I see this regression?”. The default is rollback when in doubt.
Task 9 — Open the Git PR for the permanent fix
# check-shell-blocks: allow-invalid
cd "$HOME/argocd-rollback-lab"
# Revert the offending commit.
cat > app-source/web-deployment.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: web
labels:
app: web
spec:
replicas: 3
selector:
matchLabels:
app: web
template:
metadata:
labels:
app: web
spec:
containers:
- name: web
image: nginx:1.28.1-alpine
resources:
limits:
memory: 256Mi
ports:
- containerPort: 80
EOF
git add app-source/web-deployment.yaml
git commit -m 'fix: bump nginx to 1.28.1-alpine (CVE-2026-1234)'
# Show the PR description.
cat > pr-description.md <<'EOF'
# PR: bump nginx to 1.28.1-alpine
## Summary
The team rolled back `web` from `nginx:1.28-alpine` to
`nginx:1.27-alpine` at 14:55 UTC on 2026-08-22 due to a
memory leak in `1.28-alpine` (CVE-2026-1234). This PR
bumps the image to `1.28.1-alpine`, which contains the
upstream fix.
## Changes
- `app-source/web-deployment.yaml`: change image to
`nginx:1.28.1-alpine`.
## Validation
- Pre-prod load test (5× production traffic for 4
hours): no OOMKill.
- Pre-prod security scan: no critical CVEs.
- Image signature verified (cosign verify).
## Rollback plan
The image is pinned to a digest; the rollback path is
the `argocd app rollback` procedure (Lab 23).
## References
- CVE-2026-1234
- Post-rollback investigation:
`post-rollback-investigation.md`
EOF
git add pr-description.md
git commit -m 'pr: description for the nginx 1.28.1 bump'
# After the PR merges and Argo CD syncs the fixed
# revision, re-enable automated sync (disabled since
# the rollback in Task 5):
argocd app set web --sync-policy automatedThe PR is the permanent record. The rollback was the immediate mitigation; the PR is the closing-the-gap artefact. Re-enabling automated sync is the last step: done before the fix (or a revert) merges, the next sync would re-apply the offending revision r3. The team’s discipline: no rollback without a follow-up PR.
Task 10 — Validate the deliverables
cd "$HOME/argocd-rollback-lab"
# Verify the runbook has all four steps.
grep -c "^## Step" rollback-runbook.md
# expected: 4
# Verify the rollback script passes syntax check.
bash -n argocd-rollback.sh && echo "rollback: syntax ok"
# Verify the evidence bundle is in place.
test -s rollback-evidence.md && echo "evidence: ok"
# Verify the post-rollback investigation has all
# required fields.
grep -c "^## " post-rollback-investigation.md
# expected: 7 (Summary, Timeline, Root cause, Detection,
# Response, Permanent fix, Lessons learned)
# Verify the decision tree has the matrix.
grep -c "^| " rollback-decision-tree.md
# expected: 6+ rows
# Verify the application is Synced+Healthy.
argocd app get web -o json | \
jq -r '.status.sync.status + " " + .status.health.status'
# expected: Synced Healthy
# Verify automated sync was re-enabled after the fix
# PR (Task 9); it was disabled for the rollback.
kubectl -n argocd get application web \
-o jsonpath='{.spec.syncPolicy.automated}'
# expected: non-empty (automated sync restored)The deliverables are validated: the runbook has all
four steps, the script passes syntax check, the
evidence bundle is in place, the investigation has all
required fields, the decision tree has the matrix, the
application is Synced+Healthy, and automated sync
has been restored after the fix merged.
Task 11 — Capture the deliverables
cd "$HOME/argocd-rollback-lab"
cp rollback-runbook.md \
argocd-rollback.sh \
rollback-evidence.md \
post-rollback-investigation.md \
rollback-decision-tree.md \
history-inspection.md \
pr-description.md \
"$HOME/"
ls -l "$HOME"/rollback-runbook.md \
"$HOME"/argocd-rollback.sh \
"$HOME"/rollback-evidence.md \
"$HOME"/post-rollback-investigation.md \
"$HOME"/rollback-decision-tree.mdThe deliverables are in $HOME/.
Validation
rollback-runbook.mddocuments all four steps: inspect, choose, rollback, validate.argocd-rollback.shis executable, hasset -euo pipefail, and validates the cluster state.rollback-evidence.mdcaptures before/after and the validation.post-rollback-investigation.mddocuments the regression and the permanent fix.rollback-decision-tree.mddocuments the decision matrix.history-inspection.mddocuments the history and the chosen revision.pr-description.mdis the description for the follow-up PR.
Expected Outcome
A rollback runbook, an automated rollback script, an evidence bundle, a post-rollback investigation, a decision tree, a history inspection document, and a PR description.
$HOME/argocd-rollback-lab/
├── rollback-runbook.md # the on-call reference
├── argocd-rollback.sh # the automated script
├── rollback-evidence.md # the evidence bundle
├── post-rollback-investigation.md # the regression postmortem
├── rollback-decision-tree.md # the decision framework
├── history-inspection.md # the history record
└── pr-description.md # the follow-up PRThe runbook is the spine; the script is the verb; the evidence and the investigation are the institutional knowledge.
Troubleshooting
argocd app history shows only the initial
deployment. The application was not synced via Argo
CD; the history records syncs, not commits. The
on-call engineer uses kubectl rollout history as a
fallback.
argocd app rollback refuses to run because
automated sync is enabled. This is by design:
rollback cannot run while the sync policy is
automated, because the controller would immediately
re-sync to the branch tip. Disable automated sync
(argocd app set web --sync-policy none), roll back,
and re-enable automated sync only after the follow-up
Git PR merges.
The rollback fails with field is immutable. The
manifest changes an immutable field. Either set
Replace=true in syncOptions or accept the
immutability and edit the resource via a separate
manifest.
The cluster is not healthy after the rollback. The chosen revision is not actually known-good. The on-call engineer rolls back further (to a previous revision) or considers a forward-fix.
The PR is not accepted. The team reviews the PR within 24 hours. If the format is not accepted, update the PR description and resubmit.
argocd app rollback modifies the Git state. This
should not happen; the rollback is a cluster-side
action. If the Git state is modified, the team reviews
the Argo CD configuration for any sync hooks that
auto-commit.
Cleanup
LAB="$HOME/argocd-rollback-lab"
cp -r "$LAB"/* "$HOME"/ 2>/dev/null
rm -rf "$LAB"
# Revert the Git state to revision r2 (the known-good).
git checkout main
git revert --no-edit HEADIf the kind cluster is no longer needed, delete it:
kind delete cluster --name argocd-labWhat You Learned
- Argo CD’s history is the rollback lever. The history records every sync with the source revision, the deployed manifests, and the deploy time. The rollback chooses an entry and re-applies the manifests.
- Rollback is a cluster-side action. The Git
repository is not modified by the rollback. The
team’s discipline: open a Git PR in parallel to
either revert the offending commit or to fix the
regression — the GitOps-preferred rollback is that
git revertplus a sync. - Automated sync must be off for the rollback.
Argo CD refuses to roll back while automated sync
is enabled. The flow: disable it (
argocd app set web --sync-policy none), roll back (the history ID is positional:argocd app rollback web 2), merge the Git PR, then re-enable it (argocd app set web --sync-policy automated). Re-enabling before the branch is reconciled re-applies the bad revision. - The decision tree is the on-call reference. The matrix covers the cases (cluster recoverable, data integrity, fix available, customer-visible). The default is rollback when in doubt.
- The evidence bundle is the audit trail. The before/after, the rollback action, and the validation are the artefacts the team reviews at the post-incident review.
- The post-rollback investigation links the regression to the deploy. The root cause, the detection, the response, the permanent fix, and the lessons learned are the fields the team fills in.
- The PR is the permanent record. The rollback is the immediate mitigation; the PR is the closing-the-gap artefact. The team’s discipline: no rollback without a follow-up PR.
- Pinned image tags prevent surprise regressions.
A
1.28-alpinetag can move without notice; a digest-pinned image cannot. The team’s policy: all production images are pinned to digests.