Lab 22: Handle an emergency manual change — disable self-heal, fix, document, restore

Objective

By the end of this lab you will have authored the artefacts that handle an emergency manual change in a GitOps-managed cluster: the break-glass runbook, the disable-selfHeal script, the postmortem template, the populated postmortem for the scenario, the re-converge script, the policy update proposal, and the incident timeline.

The point of this lab is not the argocd command to disable selfHeal — that is one flag. The point is the discipline: the break-glass procedure that the on-call engineer follows under pressure, the documentation that captures the imperative change, the reconciliation that restores the GitOps invariant, and the policy update that closes the gap. Without the discipline, an emergency manual change becomes an undocumented mutation that lingers in the cluster.

Architecture

A production cluster is under load. An on-call engineer discovers that the cluster is failing health checks because the web Deployment is OOMKilled under traffic. The Git manifest commits replicas: 2 and resources.limits.memory: 256Mi. The on-call engineer needs to scale the Deployment to 4 replicas and raise the memory limit to 512Mi to absorb the spike. The change must be applied immediately, but Argo CD’s selfHeal: true will revert it on the next reconciliation. The on-call engineer disables selfHeal, applies the fix, documents the change, re-enables selfHeal, and reconverges the cluster to Git after the incident.

sequenceDiagram
    participant E as engineer
    participant K as kubectl
    participant A as Argo CD
    participant G as Git repo

    E->>A: argocd app set web no_self_heal
    A-->>E: selfHeal disabled
    E->>K: kubectl scale deployment web --replicas=4
    E->>K: kubectl set resources deployment web memory_512Mi
    K-->>E: fix applied, cluster stable
    E->>G: postmortem and policy update
    Note over E,G: incident resolved, cluster remains in fix state
    E->>A: argocd app set web self_heal_true
    E->>A: argocd app sync web_request
    A->>G: pull Git state
    A->>K: apply Git state from latest commit_sha
    Note over A,K: cluster reconverged to Git state

The sequence has four phases: disable, fix, document, re-converge. Each phase has a defined entry condition and a defined exit condition. The on-call engineer does not improvise between phases.

Requirements

• A kind cluster with Argo CD installed (Lab 19).
• A sample application with syncPolicy.automated and selfHeal: true (Lab 20).
• The argocd CLI authenticated against the cluster.
• jq on the workstation.

Scenario

A platform team runs a production cluster with Argo CD managing the web application. At 14:32 UTC on 2026-08-22, PagerDuty pages the on-call engineer: the web Deployment is OOMKilled under traffic. The on-call engineer investigates and confirms that the cluster is failing because the manifest’s replicas: 2 and resources.limits.memory: 256Mi are insufficient for the current load. The engineer needs to scale to 4 replicas and raise the memory limit to 512Mi to absorb the spike while a proper fix (horizontal pod autoscaling, right-sizing) is designed. The change must be applied immediately, but Argo CD’s selfHeal: true will revert it within 30 seconds. The engineer follows the break-glass procedure.

Tasks

Task 1 — Build the break-glass runbook

# check-shell-blocks: allow-invalid
LAB="$HOME/emergency-drift-lab"
rm -rf "$LAB"
mkdir -p "$LAB"
cd "$LAB"

git init -b main
git config user.email 'ops@example.com'
git config user.name  'Ops'

cat > break-glass-runbook.md <<'EOF'
# Break-glass runbook: emergency manual change in GitOps

This runbook is the on-call engineer's reference for an
emergency manual change in a GitOps-managed cluster. The
runbook is intentionally short: under pressure, the
engineer follows four phases (disable, fix, document,
re-converge) and the team has a review at the end of
the incident.

## When to use this runbook

Use this runbook when:

- A production system is failing and a fix must be
  applied to the cluster within minutes.
- The fix is a change to a resource that Argo CD
  manages with `selfHeal: true` (the default for the
  team's production fleet).
- The fix is temporary (a mitigation, not a permanent
  design change).

Do not use this runbook when:

- The fix is a permanent change. Permanent changes go
  through Git, not through the break-glass path.
- The cluster is not managed by Argo CD (for example,
  a development cluster without GitOps). Edit the
  cluster directly.
- The fix can wait 30 minutes. If it can wait, fix it
  through Git and let `selfHeal` apply it.

## Phase 1: disable `selfHeal`

The first action is to disable `selfHeal` for the
affected `Application` CR. Without this step, any
manual change is reverted on the next reconciliation.

argocd app set $APP_NAME —self-heal=false

The flag is `false` (not `no` or `0`). Verify with:

argocd app get “ -o jsonpath={.spec.syncPolicy.automated.selfHeal}

Expected output: `false`.

Announce in `#incidents`:

[BREAK-GLASS] disabled selfHeal for $APP_NAME at <TIMESTAMP>. Reason: ONE-LINE_INCIDENT_DESCRIPTION. Planned re-enable: TIMESTAMP_PLUS_60m. Operator: <YOUR NAME>.

## Phase 2: apply the fix

Apply the imperative change. The on-call engineer uses
`kubectl` directly. The change is recorded in the
incident timeline (Task 5).

Common imperative changes:

- **Scale a Deployment:** `kubectl scale deployment <name> -n <ns> --replicas=<N>`
- **Change a resource limit:** `kubectl set resources deployment <name> -n <ns> --limits=cpu=<X>,memory=<Y>`
- **Set an env var:** `kubectl set env deployment <name> -n <ns> <KEY>=<VALUE>`
- **Patch a field:** `kubectl patch <kind> <name> -n <ns> --type json -p='<PATCH>'`

Verify the change:

kubectl get <kind> <name> -n <ns> -o yaml | head -50

## Phase 3: document

Open the postmortem document (Task 3) and fill in the
required fields:

- Change author: your name and PagerDuty ID.
- Change reason: the incident and the blast radius.
- Change applied: the exact `kubectl` commands and the
  observed before/after.
- Rollback plan: how to revert to the Git state.
- Planned re-enable: timestamp for the `selfHeal`
  re-enable.

Save the postmortem in the team's incident repository:
`https://github.com/runbook-academy/incidents/<YEAR>/<DATE>-<APP>.md`

## Phase 4: re-converge

After the incident is resolved (or after 60 minutes,
whichever is first), re-enable `selfHeal` and
reconverge the cluster to Git.

argocd app set $APP_NAME —self-heal=true argocd app sync $APP_NAME

If the Git state is the desired production state, the
sync applies the Git state and reverts the imperative
change. If the imperative change is permanent, update
Git first (PR + review) and then sync.

## Post-incident: review

Within 24 hours of the incident, the team holds a
review. The review covers:

1. Was the break-glass procedure followed? If not, why?
2. Was the imperative change documented? If not, the
   team updates the runbook.
3. Should the imperative change become permanent? If
   yes, the team opens a PR to update Git.
4. Are there policy gaps? The policy update proposal
   (Task 6) is reviewed and merged.

EOF

git add break-glass-runbook.md
git commit -m 'incident: break-glass runbook'

The runbook is the on-call reference. The four phases are the spine; the argocd and kubectl commands are the verbs. The team updates the runbook every quarter based on the reviews.

Task 2 — Build the disable-selfHeal script

# check-shell-blocks: allow-invalid
cd "$HOME/emergency-drift-lab"

cat > argocd-disable-selfheal.sh <<'EOF'
#!/usr/bin/env bash
#
# argocd-disable-selfheal.sh — disable selfHeal for an
# Application CR and announce in Slack.
#
# Required: argocd CLI authenticated; jq; curl; a Slack
# webhook URL in $SLACK_WEBHOOK_URL.
#
# Usage: APP=web TARGET_NS=production INCIDENT=INC-12345 \
#        REENABLE_AT="2026-08-22T15:32:00Z" REASON="OOMKilled" \
#        OPERATOR="jane.doe" \
#        ./argocd-disable-selfheal.sh

set -euo pipefail

: "${APP:?APP is required}"
: "${REASON:?REASON is required}"
: "${OPERATOR:?OPERATOR is required}"
: "${REENABLE_AT:?REENABLE_AT is required}"
INCIDENT="\${INCIDENT:-UNSPECIFIED}"
TARGET_NS="\${TARGET_NS:-production}"
SLACK_WEBHOOK_URL="\${SLACK_WEBHOOK_URL:-}"

NOW="$(date -u +%Y-%m-%dT%H:%M:%SZ)"

echo "=== disabling selfHeal for ${APP} ==="
echo "  incident:  ${INCIDENT}"
echo "  operator:  ${OPERATOR}"
echo "  reason:    ${REASON}"
echo "  now:       ${NOW}"
echo "  re-enable: ${REENABLE_AT}"

# Disable selfHeal.
argocd app set "$APP" --self-heal=false

# Verify.
HEAL="$(argocd app get "$APP" -o json | jq -r '.spec.syncPolicy.automated.selfHeal // "unset"')"
if [ "$HEAL" != "false" ]; then
  echo "ERROR: selfHeal is ${HEAL}, expected false" >&2
  exit 1
fi

# Record in the incident timeline.
TIMELINE_FILE="\${TIMELINE_FILE:-$HOME/incident-timeline.md}"
{
  echo "## ${NOW} — ${OPERATOR} — ${INCIDENT}"
  echo "- action: disabled selfHeal for ${APP}"
  echo "- reason: ${REASON}"
  echo "- planned re-enable: ${REENABLE_AT}"
  echo ""
} >> "$TIMELINE_FILE"

# Announce in Slack.
if [ -n "$SLACK_WEBHOOK_URL" ]; then
  curl -s -X POST -H 'Content-Type: application/json' \
    --data "$(jq -n \
      --arg app "$APP" \
      --arg ts "$NOW" \
      --arg re "$REENABLE_AT" \
      --arg op "$OPERATOR" \
      --arg in "$INCIDENT" \
      --arg rs "$REASON" \
      '{text: ("[BREAK-GLASS] disabled selfHeal for `\` + $app + "` at " + $ts + ". Reason: " + $rs + ". Planned re-enable: " + $re + ". Operator: " + $op + ". Incident: " + $in)}')" \
    "$SLACK_WEBHOOK_URL" >/dev/null
fi

echo "selfHeal disabled; announcement posted."
EOF

chmod +x argocd-disable-selfheal.sh

git add argocd-disable-selfheal.sh
git commit -m 'incident: disable-selfheal script'

The script wraps the argocd app set command in a controlled procedure: it verifies the disable, records the action in a timeline file, and announces in Slack (if the webhook is configured). The script fails loudly if the disable does not take effect.

Task 3 — Build the postmortem template

# check-shell-blocks: allow-invalid
cd "$HOME/emergency-drift-lab"

cat > postmortem-template.md <<'EOF'
# Postmortem: <APP> — <DATE>

## Summary

A one-paragraph description of the incident, the
imperative change, and the resolution.

## Timeline (UTC)

- <TIMESTAMP> — pager fired.
- <TIMESTAMP> — on-call engineer acknowledged.
- <TIMESTAMP> — investigation complete; mitigation
  identified.
- <TIMESTAMP> — break-glass disabled selfHeal.
- <TIMESTAMP> — imperative change applied.
- <TIMESTAMP> — cluster stable; monitoring continues.
- <TIMESTAMP> — selfHeal re-enabled; cluster reconverged.

## Imperative change

- Change author: <NAME> (<PAGERDUTY ID>).
- Change reason: ONE-LINE_INCIDENT_DESCRIPTION.
- Commands applied:

<KUBECTL COMMANDS>

- Before/after:
- <FIELD>: <BEFORE> → <AFTER>
- <FIELD>: <BEFORE> → <AFTER>

## Blast radius

What was affected and what was not. Include the
percentage of traffic, the user-visible impact, and
the duration of degradation.

## Detection

How was the incident detected? PagerDuty, customer
report, synthetic check, etc. Was the detection within
the team's SLO?

## Rollback plan

How was the imperative change reverted? The team
follows the re-converge procedure (Task 5).

## Lessons learned

What did the team learn? What would the team do
differently? What policy gaps were exposed?

## Action items

- [ ] <OWNER>: <ACTION>.
- [ ] <OWNER>: <ACTION>.

EOF

git add postmortem-template.md
git commit -m 'incident: postmortem template'

The template is the team’s standard. Every imperative change has a postmortem; the template enforces the required fields.

Task 4 — Populate the postmortem

# check-shell-blocks: allow-invalid
cd "$HOME/emergency-drift-lab"

cat > postmortem-2026-08-22.md <<'EOF'
# Postmortem: web — 2026-08-22

## Summary

At 14:32 UTC on 2026-08-22, the `web` Deployment
began OOMKilling under traffic. The on-call engineer
identified insufficient replicas and memory limits as
the root cause and applied a temporary mitigation
(scale to 4 replicas, raise memory limit to 512Mi)
via the break-glass procedure. The cluster stabilised
at 14:38 UTC. The mitigation was reverted at 15:32
UTC after the permanent fix (HPA + right-sizing) was
designed and merged to Git.

## Timeline (UTC)

- 14:32 — PagerDuty page: web Deployment OOMKilled.
- 14:33 — on-call engineer (jane.doe) acknowledged.
- 14:36 — investigation complete; root cause: 2
  replicas × 256Mi insufficient for 4× traffic spike.
- 14:37 — break-glass: selfHeal disabled; Slack
  announcement posted.
- 14:38 — imperative change applied (scale to 4,
  memory to 512Mi); cluster stable.
- 14:55 — HPA + right-sizing PR opened.
- 15:18 — HPA + right-sizing PR merged.
- 15:32 — selfHeal re-enabled; cluster reconverged
  to Git.

## Imperative change

- Change author: jane.doe (PD-12345).
- Change reason: OOMKilled under traffic; need
  temporary capacity boost.
- Commands applied:

kubectl scale deployment web -n production —replicas=4 kubectl set resources deployment web -n production
—limits=memory=512Mi

- Before/after:
- `spec.replicas`: 2 → 4
- `spec.template.spec.containers[0].resources.limits.memory`:
  256Mi → 512Mi

## Blast radius

- 100% of `web` traffic affected for ~6 minutes
(14:32–14:38).
- Error rate: 35% of requests returned 5xx during the
incident.
- Customer-visible: yes; status page updated.
- Data loss: none (the failure was an HTTP 5xx, not a
data integrity issue).

## Detection

PagerDuty fired on the OOMKilled alert
(`kube_pod_container_status_terminated_reason`). The
detection was within the team's SLO (60 seconds from
the first OOMKill to the page).

## Rollback plan

The re-converge procedure was followed at 15:32 UTC.
`selfHeal` was re-enabled and `argocd app sync web`
was applied. The cluster reverted to the Git state
(2 replicas, 256Mi memory) and the HPA took over
scaling. The HPA's minimum was set to 2 (matching
Git) and the maximum was set to 8 (absorbing spikes).

## Lessons learned

1. The 2-replica × 256Mi configuration was correct
 for the average load but had no headroom for the
 4× spike. The team adds a load test to the
 pre-prod pipeline.
2. The `selfHeal: true` configuration would have
 reverted the mitigation within 30 seconds. The
 break-glass procedure was the only way to apply
 the imperative change.
3. The HPA was not in place; the team had been
 deferring it. The HPA was added in the same PR as
 the right-sizing.

## Action items

- [x] jane.doe: open HPA + right-sizing PR (done at
14:55; merged at 15:18).
- [ ] platform-team: add load test to pre-prod
pipeline (due 2026-08-29).
- [ ] platform-team: add HPA to all production
deployments in the fleet (due 2026-09-12).

EOF

git add postmortem-2026-08-22.md
git commit -m 'incident: postmortem for 2026-08-22'

The populated postmortem is the team’s record. The timeline, the imperative change, the blast radius, and the lessons learned are the fields the team reviews at the post-incident review.

Task 5 — Build the re-converge script

# check-shell-blocks: allow-invalid
cd "$HOME/emergency-drift-lab"

cat > argocd-reconverge.sh <<'EOF'
#!/usr/bin/env bash
#
# argocd-reconverge.sh — re-enable selfHeal and re-sync
# the Application to reconverge the cluster to Git.
#
# Required: argocd CLI authenticated; jq; curl; a Slack
# webhook URL in $SLACK_WEBHOOK_URL.
#
# Usage: APP=web TARGET_NS=production OPERATOR=jane.doe \
#        POSTMORTEM="postmortem-2026-08-22.md" \
#        ./argocd-reconverge.sh

set -euo pipefail

: "${APP:?APP is required}"
: "${OPERATOR:?OPERATOR is required}"
POSTMORTEM="\${POSTMORTEM:-}"
TARGET_NS="\${TARGET_NS:-production}"
SLACK_WEBHOOK_URL="\${SLACK_WEBHOOK_URL:-}"

NOW="$(date -u +%Y-%m-%dT%H:%M:%SZ)"

echo "=== reconverging ${APP} to Git ==="
echo "  now:       ${NOW}"
echo "  operator:  ${OPERATOR}"
echo "  postmortem: ${POSTMORTEM}"

# Confirm the Git state is the desired state.
if [ -z "$POSTMORTEM" ] || [ ! -f "$POSTMORTEM" ]; then
  echo "ERROR: postmortem file ${POSTMORTEM} is required for audit" >&2
  exit 1
fi

# Re-enable selfHeal.
argocd app set "$APP" --self-heal=true

# Verify.
HEAL="$(argocd app get "$APP" -o json | jq -r '.spec.syncPolicy.automated.selfHeal // "unset"')"
if [ "$HEAL" != "true" ]; then
  echo "ERROR: selfHeal is ${HEAL}, expected true" >&2
  exit 1
fi

# Sync to Git.
argocd app sync "$APP" --prune --timeout 300

# Wait for Healthy.
echo "waiting for ${APP} to be Healthy..."
for i in $(seq 1 30); do
  HEALTH="$(argocd app get "$APP" -o json | jq -r '.status.health.status // "Unknown"')"
  SYNC="$(argocd app get "$APP" -o json | jq -r '.status.sync.status // "Unknown"')"
  echo "  attempt ${i}: sync=${SYNC} health=${HEALTH}"
  if [ "$SYNC" = "Synced" ] && [ "$HEALTH" = "Healthy" ]; then
    break
  fi
  sleep 10
done

if [ "$SYNC" != "Synced" ] || [ "$HEALTH" != "Healthy" ]; then
  echo "ERROR: ${APP} did not reach Synced+Healthy in 5 minutes" >&2
  exit 1
fi

# Announce in Slack.
if [ -n "$SLACK_WEBHOOK_URL" ]; then
  curl -s -X POST -H 'Content-Type: application/json' \
    --data "$(jq -n \
      --arg app "$APP" \
      --arg ts "$NOW" \
      --arg op "$OPERATOR" \
      --arg pm "$POSTMORTEM" \
      '{text: ("[RECONVERGE] re-enabled selfHeal and re-synced `\` + $app + "` to Git at " + $ts + ". Operator: " + $op + ". Postmortem: " + $pm)}')" \
    "$SLACK_WEBHOOK_URL" >/dev/null
fi

echo "reconverge complete; ${APP} is Synced+Healthy."
EOF

chmod +x argocd-reconverge.sh

git add argocd-reconverge.sh
git commit -m 'incident: re-converge script'

The script wraps the re-enable and the sync in a controlled procedure: it verifies the Git state, the postmortem is required, the sync is timed out, and the final state is verified. The script fails loudly if the cluster does not reach Synced+Healthy within 5 minutes.

Task 6 — Build the policy update proposal

# check-shell-blocks: allow-invalid
cd "$HOME/emergency-drift-lab"

cat > policy-update-proposal.md <<'EOF'
# Policy update proposal: HPA for production deployments

## Background

On 2026-08-22 at 14:32 UTC, the `web` Deployment was
OOMKilled under traffic. The on-call engineer applied
a temporary mitigation via the break-glass procedure.
The mitigation worked, but the underlying gap is that
the `web` Deployment had no HorizontalPodAutoscaler
(HPA). The 2-replica × 256Mi configuration was correct
for the average load but had no headroom for spikes.

## Proposal

Add an HPA to every production Deployment in the
fleet. The HPA's `minReplicas` matches the Git
manifest's `replicas`; the `maxReplicas` is 4× the
`minReplicas` (the team's standard headroom). The
HPA's `metrics` use CPU at 70% target average
utilisation.

## Manifest example

```yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: web
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web
  minReplicas: 2
  maxReplicas: 8
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

Roll-out

• Phase 1 (week 1): Add HPA to the web Deployment. Monitor for one week.
• Phase 2 (week 2): Add HPA to the top 10 Deployments by traffic.
• Phase 3 (week 4): Add HPA to all production Deployments.

Success criteria

• No break-glass incidents for capacity reasons in the 90 days after the roll-out.
• All production Deployments have an HPA declared in Git.

Owners

• jane.doe (incident commander)
• platform-team (roll-out)
• sre-team (review)

EOF

git add policy-update-proposal.md git commit -m ‘incident: policy update proposal’

The policy update is the closing-the-gap artefact. The
proposal links the incident to the systemic change; the
team reviews the proposal at the post-incident review
and merges it to the policy repository.

### Task 7 — Build the incident timeline

```bash
# check-shell-blocks: allow-invalid
cd "$HOME/emergency-drift-lab"

cat > incident-timeline.md <<'EOF'
# Incident timeline: web OOMKilled — 2026-08-22

This document is the minute-by-minute record of the
incident. The timeline is the source of truth for the
postmortem; every entry is timestamped and attributed.

## 14:32 — PagerDuty page

- kube_pod_container_status_terminated_reason ==
  "OOMKilled" for web-7c5b9-xyz, web-7c5b9-abc.
- PagerDuty fires on jane.doe.

## 14:33 — Acknowledged

- jane.doe acknowledges the page.
- Initial investigation: `kubectl get pods -n
  production` shows 2/2 OOMKilled.

## 14:36 — Root cause identified

- `kubectl describe deployment web -n production`
  shows `replicas: 2`, `memory: 256Mi`.
- Traffic graph shows 4× spike starting at 14:30.
- Root cause: insufficient capacity for the spike.

## 14:37 — Break-glass

- `argocd app set web no_self_heal` (jane.doe).
- Slack announcement posted to `#incidents`.
- `argocd app get web -o json | jq
  .spec.syncPolicy.automated.selfHeal` returns
  `false`.

## 14:38 — Imperative change

- `kubectl scale deployment web -n production
  --replicas=4`.
- `kubectl set resources deployment web -n production
  --limits=memory=512Mi`.
- `kubectl get deployment web -n production` shows
  `replicas: 4`, `memory: 512Mi`.
- Pods restart with new resources; cluster stable.

## 14:55 — HPA PR opened

- jane.doe opens PR #1234: "production(web): add HPA
  and right-size resources".
- PR includes HPA + resources change.

## 15:18 — HPA PR merged

- platform-team reviews and merges PR #1234.
- Argo CD detects the change; sync is paused
  (`selfHeal: false` is still active).

## 15:32 — Re-converge

- `argocd app set web self_heal_true` (jane.doe).
- `argocd app sync web` applies the Git state (HPA +
  right-sized resources).
- `argocd app get web` shows `Synced: True, Healthy:
  True`.
- Incident resolved; postmortem opened.

EOF

git add incident-timeline.md
git commit -m 'incident: timeline for 2026-08-22'

The timeline is the minute-by-minute record. The on-call engineer writes the timeline during the incident; the postmortem is the polished version.

Task 8 — Simulate the incident

cd "$HOME/emergency-drift-lab"

APP=web
TARGET_NS=production
INCIDENT=INC-12345
REENABLE_AT="2026-08-22T15:32:00Z"
REASON="OOMKilled under traffic"
OPERATOR="jane.doe"
SLACK_WEBHOOK_URL="\${SLACK_WEBHOOK_URL:-}"

# Phase 1: disable selfHeal.
APP="$APP" REASON="$REASON" OPERATOR="$OPERATOR" \
  REENABLE_AT="$REENABLE_AT" INCIDENT="$INCIDENT" \
  TARGET_NS="$TARGET_NS" SLACK_WEBHOOK_URL="$SLACK_WEBHOOK_URL" \
  ./argocd-disable-selfheal.sh

# Phase 2: apply the fix.
kubectl scale deployment "$APP" -n "$TARGET_NS" --replicas=4
kubectl set resources "deployment/${APP}" -n "$TARGET_NS" \
  --limits=memory=512Mi

# Verify the fix is in place.
kubectl get deployment "$APP" -n "$TARGET_NS" \
  -o jsonpath='{.spec.replicas}{" "}{.spec.template.spec.containers[0].resources.limits.memory}'
echo ""

# Phase 3: document. (postmortem-2026-08-22.md is the artefact.)

# Phase 4: re-converge.
APP="$APP" OPERATOR="$OPERATOR" \
  POSTMORTEM="$LAB/postmortem-2026-08-22.md" \
  TARGET_NS="$TARGET_NS" SLACK_WEBHOOK_URL="$SLACK_WEBHOOK_URL" \
  ./argocd-reconverge.sh

# Verify the cluster is back to Git state (2 replicas, 256Mi).
kubectl get deployment "$APP" -n "$TARGET_NS" \
  -o jsonpath='{.spec.replicas}{" "}{.spec.template.spec.containers[0].resources.limits.memory}'
echo ""
# expected: 2 256Mi (after re-converge; HPA is now active)

The simulation walks through all four phases. The lab records the output of each phase in the incident timeline.

Task 9 — Validate the deliverables

cd "$HOME/emergency-drift-lab"

# Verify the runbook has all four phases.
grep -c "^## Phase" break-glass-runbook.md
# expected: 4

# Verify the postmortem has all required fields.
grep -c "^## " postmortem-2026-08-22.md
# expected: 8 (Summary, Timeline, Imperative, Blast,
#              Detection, Rollback, Lessons, Actions)

# Verify the scripts pass syntax check.
bash -n argocd-disable-selfheal.sh && echo "disable: syntax ok"
bash -n argocd-reconverge.sh && echo "reconverge: syntax ok"

# Verify the timeline is in place.
test -s incident-timeline.md && echo "timeline: ok"

The deliverables are validated: the runbook has all four phases, the postmortem has all required fields, the scripts pass syntax check, and the timeline is in place.

Task 10 — Capture the deliverables

cd "$HOME/emergency-drift-lab"

cp break-glass-runbook.md \
   argocd-disable-selfheal.sh \
   postmortem-template.md \
   postmortem-2026-08-22.md \
   argocd-reconverge.sh \
   policy-update-proposal.md \
   incident-timeline.md \
   "$HOME/"

ls -l "$HOME"/break-glass-runbook.md \
       "$HOME"/argocd-disable-selfheal.sh \
       "$HOME"/postmortem-2026-08-22.md \
       "$HOME"/argocd-reconverge.sh \
       "$HOME"/policy-update-proposal.md

The deliverables are in $HOME/.

Validation

• break-glass-runbook.md documents all four phases: disable, fix, document, re-converge.
• argocd-disable-selfheal.sh is executable, has set -euo pipefail, and verifies the disable.
• postmortem-template.md has all required fields.
• postmortem-2026-08-22.md is populated for the scenario.
• argocd-reconverge.sh is executable, has set -euo pipefail, and verifies the re-enable and the sync.
• policy-update-proposal.md links the incident to a systemic change.
• incident-timeline.md has minute-by-minute entries for the entire incident.

Expected Outcome

A break-glass runbook, two scripts that automate the disable and re-enable, a postmortem template, a populated postmortem, a policy update proposal, and an incident timeline.

$HOME/emergency-drift-lab/
├── break-glass-runbook.md       # the on-call reference
├── argocd-disable-selfheal.sh   # the disable script
├── postmortem-template.md       # the template
├── postmortem-2026-08-22.md     # the populated postmortem
├── argocd-reconverge.sh         # the re-enable script
├── policy-update-proposal.md    # the closing-the-gap
└── incident-timeline.md         # the minute-by-minute record

The runbook is the spine; the scripts are the verbs; the postmortem and the policy update are the institutional knowledge.

Troubleshooting

argocd app set --self-heal=false does not take effect. The application controller may be in the middle of a reconciliation. Wait 30 seconds and re-verify with argocd app get.

The re-converge script times out. The cluster state may differ from the Git state in a way that Argo CD cannot resolve automatically (for example, an immutable field). Run argocd app diff $NAME to see the diff, and resolve manually.

The Slack webhook fails. The script logs the announcement to the timeline file even if the webhook fails. Manually announce in #incidents with the timeline entry.

The postmortem is not accepted by the team. The postmortem is the team’s standard; the team reviews it within 24 hours. If the format is not accepted, update the template (Task 3) and resubmit.

The imperative change persists after re-enable. The Git state was not updated to match the imperative change. Either update Git (if the change is permanent) or accept the revert (if the change was a temporary mitigation).

Cleanup

LAB="$HOME/emergency-drift-lab"

cp -r "$LAB"/* "$HOME"/ 2>/dev/null
rm -rf "$LAB"

# Restore selfHeal if it was disabled.
argocd app set web self_heal_true 2>/dev/null || true

If the kind cluster is no longer needed, delete it:

kind delete cluster --name argocd-lab

What You Learned

• The break-glass path has four phases. Disable, fix, document, re-converge. Skipping a phase leaves the cluster in an undocumented state.
• selfHeal is a controlled invariant. Disabling it is the only way to apply an imperative change in a GitOps-managed cluster. The disable is paired with a timer (60 minutes) and a mandatory postmortem.
• The postmortem is the audit trail. Without the postmortem, the imperative change is unrecorded. The team’s policy: no reconciliation without documentation.
• The re-converge script enforces the discipline. The script refuses to proceed without the postmortem; it verifies the re-enable; it waits for the cluster to reach Synced+Healthy; it announces in Slack.
• The policy update closes the gap. The incident is the symptom; the policy update is the cure. The team opens a PR to the policy repository within 24 hours of the incident.
• The timeline is the source of truth. The postmortem is the polished version; the timeline is the minute-by-minute record. Both are committed to the incident repository.
• HPA + right-sizing is the systemic fix. A break-glass mitigation is a temporary patch; the permanent fix is the HPA and the right-sized resources. The PR is opened in the same incident.