Auto-prune deletes a critical resource

Reported symptoms

●A resource that was running in the cluster disappeared with no Deployment or StatefulSet rollback
●`kubectl get <resource> -n <ns>` returns `NotFound`; the resource was healthy seconds earlier
●`git log -- overlays/prod/<env>/<app>/` shows a recent commit that removed the resource from the manifest tree
●`argocd app history <app>` shows a sync operation (ID and date) whose revision matches that commit; the controller log for that sync records the resource being pruned
●The Application spec has `syncPolicy.automated.prune: true` and no `prunePropagationPolicy` override
●The deleted resource has a `finalizer` set (typical for `PersistentVolumeClaim`, `Service` of type `LoadBalancer`) but Argo CD did not wait for it — the deletion was immediate
●No human approved the change; the commit was on `main` and Argo CD's automated sync ran within `timeout.seconds`
●A pre-sync hook in the Application ran `kubectl get pvc` and logged `NotFound` before the prune proceeded

Evidence

· `argocd app history <app>` shows the sync operation (ID, date, revision) that landed the removal commit, and `kubectl get application <app> -n argocd -o yaml` shows that sync's `status.operationState.syncResult.resources` entry for the deleted resource with a `pruned` message
· `git log -p -- overlays/prod/<env>/<app>/<resource>.yaml` shows the file deletion in the most recent merge to `main`
· `kubectl get events -n <ns> --sort-by=.lastTimestamp` shows a `Delete` event followed by `NotFound` events for dependents
· The kube audit log shows `verb=delete, user.username=argocd-application-controller` for the resource, with `responseStatus.code=200`
· The PVC had `persistentvolumeclaim.kubernetes.io/finalizer: kubernetes.io/pvc-protection` set, but the deletion succeeded because `prunePropagationPolicy` defaulted to `foreground` with no wait
· `kubectl get application <app> -n argocd -o yaml` returns `spec.syncPolicy.automated` as `{prune: true}`
· `kubectl get application <app> -n argocd -o yaml` shows `spec.syncPolicy.syncOptions` does NOT include `Prune=false` or `PruneLast=true`
· A search of the Argo CD repo-server log for the deletion timestamp shows `delete resource` entries with no `propagationPolicy: orphan` override

Diagnosis and resolutionclick to reveal

Root cause

Argo CD''s `automated.prune: true` means "if a resource declared in git is no longer declared, delete it from the cluster". The intent is to keep the cluster tidy: removing a manifest should remove the resource it managed. The danger is that the intent is applied uniformly, including to resources where deletion is destructive in a way the developer did not intend: a `PVC` is bound to a `PersistentVolume` whose data is the database; a `Service` of type `LoadBalancer` has an associated cloud load balancer that disappears; a `ConfigMap` may be consumed by other Applications; a `CustomResource` may own child resources. None of these are "tidy-up" candidates, and the developer who removed them from git was probably removing a copy-paste mistake or a test fixture, not intending to delete production data. The structural failure is that prune is a single boolean applied to the whole Application, with no per-resource opt-out and no review requirement for deletions.

Remediation

Recover the deleted resource from backups or from a peer cluster. For a PVC, the underlying `PersistentVolume` may still exist (`kubectl get pv`) and can be re-bound by recreating the PVC with the same `storageClassName`, `accessModes`, and `volumeName` referencing the PV. For a LoadBalancer `Service`, the cloud load balancer recreation is automatic once the `Service` is reapplied. Once the resource is restored, immediately disable pruning on the Application: `kubectl patch application <app> -n argocd --type merge -p ''{"spec":{"syncPolicy":{"automated":{"prune":false}}}}''` and re-add the resource to git. Then introduce a per-resource opt-out: the Application spec can include an `ignoreDifferences` block to keep certain fields from being reconciled, but for deletion control the right tool is a separate set of resources tagged with the `argocd.argoproj.io/compare-options=IgnoreExtraneous` annotation, or — more cleanly — splitting the Application into multiple Applications by lifecycle: a "deploy" Application that has `prune: false` and manages Deployments/StatefulSets, and an "infrastructure" Application that has `prune: true` and manages only resources whose deletion is non-destructive. Finally, add a `syncOptions: Prune=confirm` and a required reviewer for any PR that deletes a manifest from a Production overlay, so the deletion is reviewable before it lands on `main`.

Verification

The deleted resource is restored from the underlying PV or recreated. `kubectl get <resource> -n <ns>` returns the resource in `Running` or `Bound` or `Ready` state. `argocd app get <app>` shows `Sync Status: Synced` with no pruning pending. A test that adds and then removes a non-destructive resource (e.g. a `ConfigMap`) to the same overlay prunes it within `timeout.seconds`, while a similar change to the destructive resource does NOT prune — the Application is split, or the destructive resource is in a separate overlay. `git log` shows the destructive-resource manifest is still present, and the PR that removed it has been closed without merge.

Prevention

Prune is destructive. Default to `prune: false` on every Application that manages a `PersistentVolumeClaim`, a `LoadBalancer` Service, a `CustomResource` that owns children, or any resource whose deletion is non-reversible. Where prune is needed, structure the Application so destructive resources are in a separate Application with explicit `prune: true` and with a pre-sync hook that snapshots the resource set (`kubectl get <kind>.<group> -n <ns> -o yaml > /tmp/before.json`) before each prune, so a bad deletion is at least recoverable from the snapshot. Add a CI check that flags deletions in a Production overlay (`git diff main..HEAD -- overlays/prod/ | grep "^-" | grep -E "^-\s+kind:"`) and fails the PR unless a designated reviewer approves. Add an Argo CD `resource.exclusions` rule (`apiVersion: argoproj.io/v1alpha1, kind: AppProject`) that lists resources whose deletion should never be automated (PV, PVC, Service of type LoadBalancer), and route those resources through a separate review-and-apply process. The principle is that deletion is a load-bearing action and must be reviewable at the same level as the code that created the resource.

prune: true is a powerful default, and the default is applied to everything. Destructive resource kinds — PVC, LoadBalancer Service, custom resources that own children — should never be pruned by the same controller that prunes Deployments and ConfigMaps. Split the Application by lifecycle, default prune: false on the destructive side, and make every deletion a reviewed change.