Lab 28: Recover GitOps after a controller failure

Objective

By the end of this lab you will have authored the artefacts that recover the GitOps pipeline after the Argo CD application controller fails: the diagnosis procedure, the restore procedure, the recovery runbook, the controller health check, the restore script, the recovery evidence bundle, the audit trail, and the post-incident policy.

The point of this lab is not the kubectl rollout restart command — that is one CLI invocation. The point is the operationalisation: the diagnosis that identifies the failure mode, the restore procedure that brings the controller back, the re-sync that re-establishes the GitOps pipeline, and the policy update that prevents the failure from recurring.

Architecture

The team’s GitOps control plane is Argo CD: the argocd-application-controller runs as a StatefulSet (one replica in the standard install) in the argocd namespace, the argocd-repo-server serves the Git repositories, and the argocd-server exposes the API. The team’s cluster is managed exclusively through Argo CD; if the controller fails, the cluster is in an unmanaged state — no syncs, no self-heal, no notifications.

flowchart LR
    A["Git repo"] --> B["argocd-repo-server"]
    B --> C["argocd-application-controller"]
    C --> D["in-cluster apps"]
    E["argocd-server"] --> C
    F["notifications controller"] --> C
    G["Velero backup"] --> C
    H["etcd"] --> C
    C -- "reconcile" --> D

The failure mode is one of: OOMKilled (the controller is over memory limit), evicted (node pressure), crashed (bug in the controller), deadlocked (cache invalidation), or storage corruption (etcd or the CRD). Each mode has a different recovery procedure.

Requirements

• A kind cluster with Argo CD installed (Lab 19).
• A sample application managed by Argo CD (Lab 19).
• Velero 1.13+ for backup/restore.
• kubectl 1.36.x teaching target (1.29+ minimum) and the argocd CLI 3.5.x teaching target (3.0+ minimum).

Scenario

A platform team uses Argo CD to manage a production cluster. On 2026-08-22 at 03:00 UTC, the argocd-application-controller is OOMKilled. The cluster’s apps are OutOfSync and Degraded. The on-call engineer is paged and follows the recovery procedure.

Tasks

Task 1 — Build the diagnosis procedure

# check-shell-blocks: allow-invalid
LAB="$HOME/controller-recover-lab"
rm -rf "$LAB"
mkdir -p "$LAB"
cd "$LAB"

git init -b main
git config user.email 'ops@example.com'
git config user.name  'Ops'

cat > controller-diagnosis.md <<'EOF'
# Controller diagnosis procedure

This document is the on-call engineer's reference for
diagnosing an Argo CD application controller failure.
The diagnosis is the first step of the recovery; the
correct procedure depends on the failure mode.

## Symptom: controller pods not running

The on-call engineer checks the controller's status:

kubectl -n argocd get pods -l app.kubernetes.io/name=argocd-application-controller

If no pods are running, the controller is down. The
engineer checks the events:

kubectl -n argocd describe pod -l app.kubernetes.io/name=argocd-application-controller

The events show the cause: `OOMKilled`, `Evicted`,
`FailedScheduling`, `CrashLoopBackOff`, or
`ImagePullBackOff`.

## Failure mode 1: OOMKilled

**Symptom:** The controller's last `Reason` is
`OOMKilled`. The pod is restarted by the
`StatefulSet` but the same OOMKill recurs.

**Cause:** The controller's memory usage exceeds the
limit. The cause is one of:

- Large number of `Application` CRs (the controller
  caches every CR in memory).
- Large manifests (the controller parses every
  manifest).
- Memory leak in the controller (rare; the
  controller is well-tested).

**Recovery:**

1. Increase the memory limit in the controller's
   `StatefulSet`.
2. Roll the controller.
3. If OOMKill recurs, scale the `StatefulSet` to
   multiple replicas (HA mode) and set
   `ARGOCD_CONTROLLER_REPLICAS` to match.

## Failure mode 2: Evicted

**Symptom:** The controller's last `Reason` is
`Evicted`. The pod is rescheduled to another node.

**Cause:** Node pressure (memory, disk, or PID
exhaustion) on the node where the controller was
running.

**Recovery:**

1. Investigate the node pressure:
   `kubectl describe node <node-name>`.
2. Resolve the pressure (drain other workloads,
   add resources, or evict the offending process).
3. The controller is automatically rescheduled.

## Failure mode 3: CrashLoopBackOff

**Symptom:** The controller's `Restart Count` is
increasing; the `Last State` is `Error`.

**Cause:** A bug in the controller, a bad
configuration, or a corrupt Git repository.

**Recovery:**

1. Check the controller's logs:
   `kubectl -n argocd logs -l app.kubernetes.io/name=argocd-application-controller --previous`.
2. Identify the error.
3. If the error is a configuration issue, fix the
   configuration and roll the controller.
4. If the error is a bug, file an issue with the
   Argo CD project.

## Failure mode 4: Deadlocked

**Symptom:** The controller's pods are running, but
the apps are `OutOfSync` indefinitely. The logs show
no errors.

**Cause:** A deadlock in the controller's cache
invalidation. The controller has a stale view of the
cluster state.

**Recovery:**

1. Restart the controller:
   `kubectl -n argocd rollout restart statefulset argocd-application-controller`.
2. The restart clears the cache; the controller
   re-reads the cluster state.

## Failure mode 5: storage corruption

**Symptom:** The controller's pods fail to start
with `FailedMount` or `FailedAttachVolume` errors.
The events show storage issues.

**Cause:** The etcd cluster (or the CRD storage) is
corrupt.

**Recovery:**

1. Restore the etcd cluster from a backup (Velero or
   `etcdctl snapshot`).
2. Roll the controller.
3. Verify the `Application` CRs are present.

## Triage decision tree

controller pods not running? ├── OOMKilled → Failure mode 1 ├── Evicted → Failure mode 2 ├── CrashLoopBackOff → Failure mode 3 └── (no errors, OutOfSync) → Failure mode 4 (FailedMount/FailedAttachVolume) → Failure mode 5

EOF

git add controller-diagnosis.md
git commit -m 'controller: diagnosis procedure'

The diagnosis procedure is the on-call reference. The five failure modes are the most common; the triage decision tree is the entry point.

Task 2 — Build the controller health check

# check-shell-blocks: allow-invalid
cd "$HOME/controller-recover-lab"

cat > controller-health-check.sh <<'EOF'
#!/usr/bin/env bash
#
# controller-health-check.sh — diagnose the Argo CD
# application controller's state.
#
# The script reports:
#   - the number of running controller pods
#   - the pod's last reason (OOMKilled, Evicted, etc.)
#   - the pod's restart count
#   - the application's sync status
#   - the application's health status
#
# The output is the input to the recovery procedure.

set -uo pipefail

ARGOCD_NS="\${ARGOCD_NS:-argocd}"

echo "=== controller health check at $(date -u +%Y-%m-%dT%H:%M:%SZ) ==="
echo ""

# Pod status.
echo "--- pods ---"
kubectl -n "$ARGOCD_NS" get pods \
  -l app.kubernetes.io/name=argocd-application-controller \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.phase}{"\t"}{.status.containerStatuses[0].restartCount}{"\t"}{.status.containerStatuses[0].lastState.terminated.reason}{"\n"}{end}'

# Events.
echo ""
echo "--- recent events ---"
kubectl -n "$ARGOCD_NS" get events \
  --field-selector involvedObject.kind=Pod \
  --sort-by='.lastTimestamp' | tail -10

# Application status.
echo ""
echo "--- application status ---"
argocd app list -o json 2>/dev/null | \
  jq -r '.[] | "\(.metadata.name)\tsync=\(.status.sync.status)\thealth=\(.status.health.status)"' || \
  echo "argocd CLI not authenticated"

# Controller logs (last 20 lines).
echo ""
echo "--- controller logs (last 20) ---"
kubectl -n "$ARGOCD_NS" logs \
  -l app.kubernetes.io/name=argocd-application-controller \
  --tail=20 --timestamps=true 2>&1 || echo "no logs available"
EOF

chmod +x controller-health-check.sh

git add controller-health-check.sh
git commit -m 'controller: health check script'

The health check script automates the diagnosis. The output is the input to the recovery procedure.

Task 3 — Build the restore procedure

# check-shell-blocks: allow-invalid
cd "$HOME/controller-recover-lab"

cat > controller-restore.md <<'EOF'
# Controller restore procedure

This document is the on-call engineer's reference for
restoring the Argo CD application controller from a
Velero backup. The procedure is used when the
controller's state is corrupt (etcd corruption, lost
CRDs, or unrecoverable deadlock).

## Phase 1: identify the backup

The on-call engineer identifies the most recent
successful Velero backup:

velero backup get

The engineer selects the most recent backup that
includes the `argocd` namespace:

velero backup describe <backup-name> —details

## Phase 2: scale down the controller

The on-call engineer scales down the controller to
prevent it from interfering with the restore:

kubectl -n argocd scale statefulset argocd-application-controller —replicas=0

The controller is scaled to 0 replicas; the
`argocd-repo-server` and `argocd-server` remain
running (they do not need to be scaled down).

## Phase 3: restore from backup

The on-call engineer restores the `argocd` namespace
from the Velero backup:

velero restore create —from-backup <backup-name>
—include-namespaces argocd

The restore takes 5-15 minutes. The on-call engineer
monitors the progress:

velero restore describe <restore-name> —details

## Phase 4: scale up the controller

The on-call engineer scales the controller back up:

kubectl -n argocd scale statefulset argocd-application-controller —replicas=1

The engineer waits for the controller to be ready:

kubectl -n argocd rollout status statefulset argocd-application-controller —timeout=300s

## Phase 5: verify the restore

The on-call engineer verifies the restore:

- `argocd app list` shows the expected `Application`
  CRs.
- `argocd app get <name>` shows `Synced: True,
  Healthy: True` for each app.
- `kubectl -n argocd get application -o yaml` shows
  the expected CRs.

## Phase 6: re-sync the apps

The on-call engineer re-syncs the apps:

for app in $(argocd app list -o name); do argocd app sync “$app” —prune —timeout 300 done

The sync ensures the cluster matches the Git state;
the `prune` flag removes any resources that are not
in Git.

## Verification

After each phase, the team verifies:

- The backup is identified.
- The controller is scaled down.
- The restore completes.
- The controller is scaled up and ready.
- The `Application` CRs are restored.
- The apps are re-synced and healthy.

EOF

git add controller-restore.md
git commit -m 'controller: restore procedure'

The restore procedure is the recovery for severe failure modes (storage corruption, lost CRDs). The six phases are the spine; the Velero and kubectl commands are the verbs.

Task 4 — Build the recovery runbook

# check-shell-blocks: allow-invalid
cd "$HOME/controller-recover-lab"

cat > recovery-runbook.md <<'EOF'
# Recovery runbook: Argo CD controller failure

This runbook is the on-call engineer's reference for
recovering from an Argo CD application controller
failure. The runbook covers the staged recovery:
diagnose, restore, re-sync, verify.

## When to use this runbook

Use this runbook when:

- The `argocd-application-controller` is not running
  (no pods, OOMKilled, Evicted, CrashLoopBackOff).
- The applications are `OutOfSync` indefinitely.
- The cluster is in an unmanaged state.

## Phase 1: diagnose

The on-call engineer runs the health check
(`controller-health-check.sh`):

./controller-health-check.sh

The output identifies the failure mode (OOMKilled,
Evicted, CrashLoopBackOff, Deadlocked, Storage
corruption).

## Phase 2: choose the recovery

The on-call engineer chooses the recovery based on
the failure mode:

| Failure mode | Recovery |
|--------------|----------|
| OOMKilled | Increase memory limit, roll controller |
| Evicted | Resolve node pressure, controller is rescheduled |
| CrashLoopBackOff | Fix configuration or roll controller |
| Deadlocked | Restart controller (`kubectl rollout restart`) |
| Storage corruption | Restore from Velero backup (Task 3) |

## Phase 3: apply the recovery

The on-call engineer applies the recovery:

- **OOMKilled:** Edit the controller's `StatefulSet`
  to increase the memory limit; roll the controller.
- **Evicted:** Investigate the node pressure;
  resolve; the controller is rescheduled.
- **CrashLoopBackOff:** Investigate the logs; fix
  the configuration; roll the controller.
- **Deadlocked:** `kubectl -n argocd rollout
  restart statefulset argocd-application-controller`.
- **Storage corruption:** Follow the restore
  procedure (Task 3).

## Phase 4: re-sync the apps

After the controller is back, the on-call engineer
re-syncs the apps:

for app in $(argocd app list -o name); do argocd app sync “$app” —prune —timeout 300 done

## Phase 5: verify

The on-call engineer verifies the recovery:

- `argocd app list` shows all apps `Synced: True,
  Healthy: True`.
- `kubectl get pods -A` shows the expected Pods.
- The synthetic check is passing.

## Post-incident: review

Within 24 hours of the recovery, the team reviews:

1. What was the failure mode?
2. What was the root cause?
3. Could the failure have been prevented? (Memory
   limit, node pressure, configuration error.)
4. Is the recovery procedure adequate?
5. Are there policy gaps? (Backup frequency,
   monitoring, HA mode.)

EOF

git add recovery-runbook.md
git commit -m 'controller: recovery runbook'

The recovery runbook is the on-call reference. The five phases (diagnose, choose, apply, re-sync, verify) are the spine; the kubectl and argocd commands are the verbs.

Task 5 — Build the restore script

# check-shell-blocks: allow-invalid
cd "$HOME/controller-recover-lab"

cat > controller-restore.sh <<'EOF'
#!/usr/bin/env bash
#
# controller-restore.sh — restore the Argo CD
# application controller from a Velero backup.
#
# Required: velero CLI; kubectl; jq.
#
# Usage: BACKUP=<backup-name> ARGOCD_NS=argocd \
#        ./controller-restore.sh

set -euo pipefail

: "${BACKUP:?BACKUP is required}"
ARGOCD_NS="\${ARGOCD_NS:-argocd}"
OPERATOR="\${OPERATOR:-$(whoami)}"

NOW="$(date -u +%Y-%m-%dT%H:%M:%SZ)"

echo "=== restoring controller from backup ${BACKUP} ==="
echo "  now: ${NOW}"
echo "  operator: ${OPERATOR}"

# Step 1: confirm the backup exists.
if ! velero backup get "$BACKUP" >/dev/null 2>&1; then
  echo "ERROR: backup ${BACKUP} not found" >&2
  exit 1
fi

# Step 2: scale down the controller.
echo "scaling down controller"
kubectl -n "$ARGOCD_NS" scale statefulset \
  argocd-application-controller --replicas=0

# Step 3: restore from backup.
echo "restoring from backup"
velero restore create --from-backup "$BACKUP" \
  --include-namespaces "$ARGOCD_NS" \
  --wait

# Step 4: scale up the controller.
echo "scaling up controller"
kubectl -n "$ARGOCD_NS" scale statefulset \
  argocd-application-controller --replicas=1
kubectl -n "$ARGOCD_NS" rollout status \
  statefulset argocd-application-controller --timeout=300s

# Step 5: re-sync the apps.
echo "re-syncing apps"
for app in $(argocd app list -o name 2>/dev/null | sed 's/^app://'); do
  echo "syncing $app"
  argocd app sync "$app" --prune --timeout 300
done

# Step 6: verify.
echo "verifying"
argocd app list -o json | \
  jq -r '.[] | "\(.metadata.name)\tsync=\(.status.sync.status)\thealth=\(.status.health.status)"'

echo "controller restore complete"
EOF

chmod +x controller-restore.sh

git add controller-restore.sh
git commit -m 'controller: restore script'

The restore script automates the Velero restore. It scales the controller down, restores from backup, scales the controller up, and re-syncs the apps.

Task 6 — Simulate the failure and recover

# check-shell-blocks: allow-invalid
cd "$HOME/controller-recover-lab"

# Diagnose the controller.
./controller-health-check.sh

# Simulate the failure: scale down the controller.
kubectl -n argocd scale statefulset \
  argocd-application-controller --replicas=0

# Wait for the cluster to be unmanaged.
sleep 60

# Run the health check to confirm the failure.
./controller-health-check.sh

# Restore the controller from a backup.
# (The Velero backup must exist; the lab assumes it does.)
BACKUP="\${BACKUP:-argocd-daily-20260822}"
BACKUP="$BACKUP" ./controller-restore.sh

# Verify the recovery.
argocd app list

The simulation walks through the entire recovery. The health check diagnoses the failure; the restore script recovers the controller; the verification confirms the recovery.

Task 7 — Capture the recovery evidence

# check-shell-blocks: allow-invalid
cd "$HOME/controller-recover-lab"

cat > recovery-evidence.md <<'EOF'
# Recovery evidence: controller failure — 2026-08-22

This document is the canonical evidence bundle for
the controller recovery on 2026-08-22. The bundle
captures the detection, the diagnosis, the recovery,
and the post-recovery state.

## Detection

- 03:00 UTC: PagerDuty page fired (controller
  OOMKilled).
- 03:01 UTC: on-call engineer (jane.doe)
  acknowledged.
- 03:02 UTC: ran the health check
  (`controller-health-check.sh`).
- 03:03 UTC: confirmed the failure mode (OOMKilled).

## Diagnosis

- Controller pods: 0 running.
- Last reason: `OOMKilled`.
- Restart count: increasing.
- Application status: `OutOfSync, Degraded`.
- Controller logs: `runtime: out of memory`.

## Root cause

The controller's memory usage exceeded the 1 GB
limit. The cause was a sudden increase in the number
of `Application` CRs (the team had added 20 new apps
in the previous week). The controller's memory
footprint is proportional to the number of CRs.

## Recovery

- 03:05 UTC: increased the controller's memory limit
  to 2 GB.
- 03:06 UTC: rolled the controller.
- 03:08 UTC: controller is ready.
- 03:10 UTC: re-synced the apps.
- 03:15 UTC: all apps are `Synced: True, Healthy:
  True`.

## Post-recovery state

- Controller: running with 2 GB memory limit.
- Apps: all `Synced: True, Healthy: True`.
- Cluster: managed; drift detection active.
- Notifications: active.

## Verification

- `kubectl -n argocd get pods` shows the controller
  running.
- `argocd app list` shows all apps `Synced: True`.
- The synthetic check is passing.

EOF

git add recovery-evidence.md
git commit -m 'controller: recovery evidence bundle'

The evidence bundle is the canonical record. The detection, the diagnosis, the root cause, the recovery, the post-recovery state, and the verification are the fields the team reviews at the post-incident review.

Task 8 — Build the audit trail

# check-shell-blocks: allow-invalid
cd "$HOME/controller-recover-lab"

cat > audit-trail.md <<'EOF'
# Audit trail: controller failure — 2026-08-22

This document is the audit trail for the controller
recovery on 2026-08-22. The trail is the canonical
record for the compliance review; every action is
timestamped and attributed.

## 03:00 — PagerDuty page

- Source: PagerDuty.
- Alert: controller OOMKilled.
- Operator: jane.doe.

## 03:01 — Acknowledged

- Operator: jane.doe.
- Action: opened the alert; confirmed the controller
  was OOMKilled.

## 03:02 — Health check

- Operator: jane.doe.
- Action: ran `controller-health-check.sh`.
- Verification: pods not running; last reason
  `OOMKilled`.

## 03:03 — Diagnosis

- Operator: jane.doe.
- Action: identified the failure mode (OOMKilled).
- Verification: controller logs showed
  `runtime: out of memory`.

## 03:05 — Memory limit increased

- Operator: jane.doe.
- Action: edited the controller's `StatefulSet` to
  set the memory limit to 2 GB.

## 03:06 — Controller rolled

- Operator: jane.doe.
- Action: `kubectl -n argocd rollout restart
  statefulset argocd-application-controller`.

## 03:08 — Controller ready

- Operator: jane.doe.
- Action: waited for the controller to be ready.
- Verification: `kubectl rollout status` reported
  `rolling update complete`.

## 03:10 — Re-sync

- Operator: jane.doe.
- Action: re-synced the apps via
  `argocd app sync`.

## 03:15 — Verified

- Operator: jane.doe.
- Action: verified all apps are `Synced: True,
  Healthy: True`.

## 03:30 — Post-incident review

- Operators: jane.doe, sre-team, platform-team.
- Action: reviewed the evidence bundle; opened a
  policy update to add HA mode (Task 9).

EOF

git add audit-trail.md
git commit -m 'controller: audit trail'

The audit trail is the canonical record. Every action is timestamped and attributed.

Task 9 — Build the post-incident policy

# check-shell-blocks: allow-invalid
cd "$HOME/controller-recover-lab"

cat > post-incident-policy.md <<'EOF'
# Post-incident policy update: Argo CD controller HA

## Background

On 2026-08-22 at 03:00 UTC, the
`argocd-application-controller` was OOMKilled. The
root cause was a sudden increase in the number of
`Application` CRs (the team had added 20 new apps in
the previous week), which exceeded the controller's
1 GB memory limit. The recovery took 15 minutes
(diagnose, increase limit, roll, re-sync). The
underlying gap — single-replica controller with a
fixed memory limit — must be closed.

## Proposal

The team adopts the following policy:

1. **The controller runs in HA mode** (3 replicas) in
   production. The `StatefulSet` replica count and
   the `ARGOCD_CONTROLLER_REPLICAS` environment
   variable are set to the same number; the
   controller shards managed clusters across the
   replicas. A single replica OOMKilled is no
   longer a P1 incident; the remaining replicas
   continue to reconcile.
2. **The controller's memory request and limit are
   increased** to 4 GB. The memory footprint is
   proportional to the number of `Application` CRs;
   4 GB accommodates up to 500 CRs.
3. **The controller's CPU request and limit are
   increased** to 2 cores. The CPU footprint is
   proportional to the reconciliation rate.
4. **A capacity alert is added** to the controller's
   memory usage. The alert fires when the controller
   reaches 80% of the limit.
5. **The Velero backup frequency is increased** to
   hourly. The previous frequency was daily.

## Roll-out

- **Phase 1 (this week):** Increase the controller's
  memory and CPU limits; add the capacity alert.
- **Phase 2 (this week):** Scale the controller to
  HA mode (3 replicas).
- **Phase 3 (next week):** Increase the Velero backup
  frequency to hourly.

## Success criteria

- No controller OOMKill in the 90 days after the
  roll-out.
- The capacity alert fires before the limit is
  reached.
- The Velero backup is verified hourly.

## Owners

- platform-team (roll-out)
- sre-team (monitoring and alerting)
- security-team (backup verification)

EOF

git add post-incident-policy.md
git commit -m 'controller: post-incident policy'

The policy update is the closing-the-gap artefact. The proposal links the incident to the systemic change; the team reviews the proposal at the post-incident review and merges it to the policy repository.

Task 10 — Validate the deliverables

# check-shell-blocks: allow-invalid
cd "$HOME/controller-recover-lab"

# Verify the diagnosis has all five failure modes.
grep -c "^## Failure mode" controller-diagnosis.md
# expected: 5

# Verify the recovery runbook has all five phases.
grep -c "^## Phase" recovery-runbook.md
# expected: 5

# Verify the restore procedure has all six phases.
grep -c "^## Phase" controller-restore.md
# expected: 6

# Verify the scripts pass syntax check.
bash -n controller-health-check.sh && echo "health: syntax ok"
bash -n controller-restore.sh && echo "restore: syntax ok"

# Verify the health check runs.
./controller-health-check.sh

# Verify the evidence bundle has all sections.
grep -c "^## " recovery-evidence.md
# expected: 5 (Detection, Diagnosis, Root cause,
#              Recovery, Post-recovery, Verification)

# Verify the audit trail has timestamps.
grep -c "^## [0-9]" audit-trail.md
# expected: 8+ entries

# Verify the policy has all five items.
grep -c "^[0-9]\." post-incident-policy.md
# expected: 5

The deliverables are validated.

Task 11 — Capture the deliverables

# check-shell-blocks: allow-invalid
cd "$HOME/controller-recover-lab"

cp controller-diagnosis.md \
   controller-health-check.sh \
   controller-restore.md \
   recovery-runbook.md \
   controller-restore.sh \
   recovery-evidence.md \
   audit-trail.md \
   post-incident-policy.md \
   "$HOME/"

ls -l "$HOME"/controller-diagnosis.md \
       "$HOME"/controller-health-check.sh \
       "$HOME"/controller-restore.md \
       "$HOME"/recovery-runbook.md \
       "$HOME"/controller-restore.sh \
       "$HOME"/recovery-evidence.md \
       "$HOME"/audit-trail.md \
       "$HOME"/post-incident-policy.md

The deliverables are in $HOME/.

Validation

• controller-diagnosis.md documents all five failure modes with symptoms and recovery.
• controller-health-check.sh is executable, has set -uo pipefail, and reports the controller’s state.
• controller-restore.md documents all six phases: identify, scale down, restore, scale up, verify, re-sync.
• recovery-runbook.md documents all five phases: diagnose, choose, apply, re-sync, verify.
• controller-restore.sh is executable, has set -euo pipefail, and automates the Velero restore.
• recovery-evidence.md captures detection, diagnosis, root cause, recovery, post-recovery, and verification.
• audit-trail.md has timestamped entries for every action.
• post-incident-policy.md proposes HA mode and increased resource limits with roll-out phases.

Expected Outcome

A diagnosis procedure, a controller health check, a restore procedure, a recovery runbook, a restore script, a recovery evidence bundle, an audit trail, and a post-incident policy.

$HOME/controller-recover-lab/
├── controller-diagnosis.md       # five failure modes
├── controller-health-check.sh    # the diagnosis script
├── controller-restore.md         # the restore procedure
├── recovery-runbook.md           # the runbook
├── controller-restore.sh         # the restore script
├── recovery-evidence.md          # the canonical record
├── audit-trail.md                # the audit trail
└── post-incident-policy.md       # the closing-the-gap

The diagnosis is the first step; the runbook is the on-call reference; the script is the verb; the evidence and the audit trail are the institutional knowledge; the policy update is the closing-the-gap.

Troubleshooting

The health check returns “no logs available”. The controller’s pods are not running. Follow the diagnosis procedure (Task 1) and the recovery runbook (Task 4).

The Velero restore fails with “backup not found”. The backup may have been deleted or expired. Verify with velero backup get. The team retains 30 days of backups.

The controller is ready but the apps are still OutOfSync. The re-sync may have failed. Run argocd app sync manually for each app.

The memory limit increase does not fix the OOMKill. The controller may have a memory leak. Enable the controller’s profiling endpoint and capture a heap dump.

Apps go unreconciled after scaling to HA. The StatefulSet replica count and the ARGOCD_CONTROLLER_REPLICAS environment variable must be set to the same number; a mismatch leaves some managed clusters assigned to shards that no replica serves. The HA scaling is for the production cluster; the lab’s kind cluster is single-replica.

Cleanup

# check-shell-blocks: allow-invalid
LAB="$HOME/controller-recover-lab"

cp -r "$LAB"/* "$HOME"/ 2>/dev/null
rm -rf "$LAB"

# Revert the controller's memory limit (optional).
kubectl -n argocd set resources statefulset \
  argocd-application-controller \
  --limits=memory=1Gi

If the kind cluster is no longer needed, delete it:

# check-shell-blocks: allow-invalid
kind delete cluster --name argocd-lab

What You Learned

• Diagnosis is the first step. A restore without a diagnosis may fix the symptom but not the cause. The five failure modes (OOMKilled, Evicted, CrashLoopBackOff, Deadlocked, Storage corruption) cover the common cases.
• The health check automates the diagnosis. The script reports the pod status, the events, the application status, and the logs. The output is the input to the recovery procedure.
• The recovery is staged. Diagnose, choose, apply, re-sync, verify. The order minimises the time-to-recovery and the risk of cascading failures.
• The restore is for severe failure modes only. OOMKilled, Evicted, and Deadlocked are recovered via restart or resource adjustment. Storage corruption is recovered via Velero.
• The re-sync restores the cluster to the Git state. After the controller is back, the re-sync ensures the cluster matches Git. The prune flag removes any drift.
• The audit trail is the canonical record. Every action is timestamped and attributed.
• The policy update closes the gap. HA mode, increased resource limits, hourly backups. The incident is the symptom; the policy update is the cure.
• The Velero backup is verified weekly. A backup that has never been restored is a liability. The team restores a sample backup weekly to verify the procedure.