Incident — a prune deleted production — the recovery procedure

The incident begins with a chart typo and ends with a Git commit that restores the deleted resource. The chain — typo, merge, automated sync, deletion, consumer failure — is the canonical production prune failure. The recovery is canonical too: classify the deletion, restore the resource as a Git commit, let the next sync apply it. The GitOps promise is preserved because the recovery is itself a Git commit.

The incident timeline

flowchart LR
    A["Typo in chart removes a manifest"] --> B["PR merged with single reviewer"]
    B --> C["Automated sync triggers"]
    C --> D["Controller deletes resource"]
    D --> E["Consumer fails"]
    E --> F["Alert fires"]
    F --> G["On-call investigates"]

The timestamps tell the story:

• T0: a developer opens a PR that renames a ConfigMap from payments-config to payments-cfg.
• T+12m: a reviewer approves. The PR is merged to main.
• T+13m: the automated sync runs. The diff between the previous render and the new render contains a delete (payments-config) and a create (payments-cfg).
• T+13m 4s: the controller executes both actions. The ConfigMap is gone from the cluster.
• T+14m: a downstream Deployment that mounts payments-config fails its readiness probe. The pods restart in a crash loop.
• T+18m: an alert fires on the readiness probe failure rate. The on-call engineer is paged.

The total elapsed time from PR merge to alert is six minutes. The total elapsed time from the deletion to consumer failure is less than ninety seconds.

The diagnostic steps

The on-call engineer’s first three commands:

argocd app manifests "$APP_NAME" | grep payments
kubectl get configmap -n "$NAMESPACE" payments-config
kubectl get configmap -n "$NAMESPACE" payments-cfg

The first command shows the controller’s view: only payments-cfg is in the manifest. The second command shows the cluster’s view: payments-config is gone. The third command shows the new name exists. The diff is exactly the prune-and-create pair the chart produced.

The second set of commands trace the deletion to a sync:

kubectl get events -n "$NAMESPACE" \
  --field-selector involvedObject.name=payments-config
argocd app history "$APP_NAME"

The events show the delete action with the controller’s service account as the actor. The history shows the sync at T+13m, the revision it applied, and the diff it executed. The audit trail is intact: the Git commit, the PR, the reviewer, the sync, and the deletion are all visible.

The GitOps-native recovery

The recovery procedure:

1. Restore the manifest as a Git commit. Add the missing ConfigMap back to the chart. The contents can be the same as the deleted version (recovered from the previous commit) or the contents of the new payments-cfg ConfigMap (if a rename was the intent).
2. Push and let the sync apply the recreation. The next automated sync sees the manifest in the render, diffs against the cluster, and creates the resource. The audit trail records the recreation as a commit.
3. Verify the consumer recovers. The Deployment’s readiness probe passes when the ConfigMap is mounted. The alert clears.
4. Investigate why the rename reached production. A rename that touched a consumer’s mount path was a delete-and-create from the consumer’s point of view. The next lesson (the decision framework) is the prevention.

The recovery is auditable because every step is a Git operation. The recreation is reproducible because the Git commit is the source of truth.

The post-incident prevention

Three changes close the gap:

1. Add a namespace backup before re-enabling prune. A Velero or equivalent backup is the recovery path for the next unintended delete.
2. Require a second reviewer on any PR that removes a manifest. The deletion is destructive; the review is the prevention.
3. Treat rename PRs as migration PRs. A two-step migration
   • add the new resource, migrate the consumer, remove the old resource only after the consumer is updated - prevents the prune-driven deletion of a resource that is still in use.

Cross-course references

• Git, CI/CD & GitOps for Infrastructure Engineers - Part LXXX-04 (Prune safety mechanisms) is the procedure the incident reveals; Part LXIV (Deploy evidence) is the audit trail.
• Kubernetes for Production Sysadmins - Part XXI (Disaster Recovery) covers Velero and the namespace backup pattern; Part XXII (Postmortem) is the formal incident analysis.

Quiz