Skip to main content
RunBook Academy

← All runbooks in Git, CI/CD & GitOps

critical riskcluster affecting~60 min

Runbook: Recover a GitOps Controller

1 · Prerequisites

Confirm every item is in place before any state change.

  • git-cicd-gitops-rb-18-troubleshoot-argocd-reconcile
  • kubectl configured against the affected cluster context
  • For Argo CD HA: knowledge of the Redis/Postgres backend (and credentials)
  • Access to the GitOps repo for manifest recovery
  • Helm installed locally (helm version) for re-installation scenarios

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

  • · Identify the controller type and version. Argo CD: kubectl get statefulset -n argocd -l app.kubernetes.io/name=argocd-application-controller -o jsonpath="{.items[0].spec.template.spec.containers[0].image}". Flux: flux --version (run from a workstation with cluster access) or kubectl get deploy -n flux-system -o jsonpath="{.items[*].spec.template.spec.containers[*].image}". The version determines the recovery procedure (HA vs non-HA)
  • · Capture the current state of the controller namespace. Argo CD: kubectl get all,cm,secret -n argocd -o yaml > /tmp/argocd-pre-recovery.yaml. Flux: kubectl get all,cm,secret -n flux-system -o yaml > /tmp/flux-pre-recovery.yaml. The capture is the baseline for the recovery
  • · Capture the controller logs. Argo CD: kubectl logs -n argocd -l app.kubernetes.io/name=argocd-application-controller --previous --tail=300. Flux: kubectl logs -n flux-system -l app=kustomize-controller --previous --tail=300. The "previous" logs are from the crashed instance and contain the failure cause
  • · Check the controller's events. kubectl get events -n argocd --sort-by=.lastTimestamp | tail -50 (Argo CD) or kubectl get events -n flux-system --sort-by=.lastTimestamp | tail -50 (Flux). Events name the failure (image pull failure, OOMKilled, probe failure)
  • · Check the controller's database (Argo CD only). Argo CD HA uses Postgres or Redis; non-HA uses in-memory state. kubectl get pods -n argocd -l app=redis -o wide and kubectl get pods -n argocd -l app.kubernetes.io/name=argocd-server. The database must be healthy for the controller to recover
  • · Verify the cluster's connectivity to the Git repository. The controller's failure may be unrelated to the controller itself; the Git provider may be unreachable. See git-cicd-gitops-rb-17-troubleshoot-argocd-auth and git-cicd-gitops-rb-21-recover-git-hosting-dependency

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

  1. 1STEP 1 - Classify the controller failure. Five classes: (a) single-pod CrashLoopBackOff; (b) all-pod CrashLoopBackOff; (c) controller pods running but not reconciling; (d) database backend (Redis/Postgres) down; (e) controller namespace lost (cluster recovery scenario)
  2. 2STEP 2 - For class (a) single-pod CrashLoopBackOff: the StatefulSet is healthy but a single pod is bad. kubectl delete pod -n argocd -l app.kubernetes.io/name=argocd-application-controller (replace the label for Flux). Kubernetes will create a new pod. If the new pod also crashes, escalate to class (b)
  3. 3STEP 3 - For class (b) all-pod CrashLoopBackOff: the StatefulSet is broken. Inspect the controller logs from every pod: kubectl logs -n argocd -l app.kubernetes.io/name=argocd-application-controller --previous --tail=100 --all-containers. Common causes: image pull failure (registry outage, image missing), RBAC denied (ServiceAccount misconfigured), config map invalid (argocd-cm corruption), resource limit hit (OOMKilled)
  4. 4STEP 4 - For class (b) image pull failure: kubectl describe pod -n argocd -l app.kubernetes.io/name=argocd-application-controller | grep -A5 "Failed\\|ImagePull". If the registry is down, see git-cicd-gitops-rb-22-recover-registry-outage. If the image tag is wrong, patch the StatefulSet: kubectl set image statefulset/argocd-application-controller -n argocd argocd-application-controller=<registry>/argocd:<correct-tag>
  5. 5STEP 5 - For class (b) OOMKilled: the controller ran out of memory. kubectl describe pod -n argocd -l app.kubernetes.io/name=argocd-application-controller | grep -A2 "Last State". Increase the memory limit in the StatefulSet: kubectl set resources statefulset/argocd-application-controller -n argocd --limits=memory=2Gi. The default limit is too low for large clusters (1000+ Applications)
  6. 6STEP 6 - For class (b) config map invalid: the argocd-cm or argocd-cmd-params-cm is corrupted. Diff against a known-good version: kubectl get cm -n argocd argocd-cm -o yaml > /tmp/argocd-cm-broken.yaml. Restore from the GitOps repo (where the manifest is the source of truth) or from a backup. kubectl apply -f /path/to/gitops-repo/argocd-cm.yaml
  7. 7STEP 7 - For class (c) pods running but not reconciling: the controller is alive but not doing work. Check the controller's metrics endpoint (kubectl port-forward -n argocd svc/argocd-metrics 8082:8082 then curl :8082/metrics). A controller with argocd_app_reconcile_count stuck at 0 is deadlocked. Common causes: shard/replica misconfiguration (HA), Git source unreachable, RBAC denied on managed namespaces
  8. 8STEP 8 - For class (c) shard/replica misconfiguration: in HA mode, the controller shards managed clusters across StatefulSet replicas — if the replica count and the shard count disagree, Applications on some clusters never reconcile. Compare kubectl get statefulset argocd-application-controller -n argocd -o jsonpath="{.spec.replicas}" with the ARGOCD_CONTROLLER_REPLICAS env var, and check kubectl logs -n argocd -l app.kubernetes.io/name=argocd-application-controller --tail=100 | grep -i shard for the shard assignment. Make the two values match, then kubectl rollout restart statefulset/argocd-application-controller -n argocd
  9. 9STEP 9 - For class (d) database backend down (Argo CD HA): the controller cannot start without its database. Argo CD HA uses Redis (cache) and Postgres (source of truth). kubectl get pods -n argocd -l app=redis. If Redis is down, see kubernetes runbooks for StatefulSet recovery. If Postgres is down, see the database runbook. The controller cannot function without its database; restore the database first
  10. 10STEP 10 - For class (e) controller namespace lost (cluster recovery): the entire argocd/flux-system namespace is gone. This is the worst case. Reinstall the controller via Helm: Argo CD helm install argocd argo-cd/argo-cd --namespace argocd --create-namespace; Flux flux install --namespace flux-system. Then re-import every Application or re-apply every Kustomization/HelmRelease from the GitOps repo
  11. 11STEP 11 - For Argo CD re-import after reinstall: argocd app create <app> --repo <url> --path <path> --dest-server <server> --dest-namespace <ns> --revision <branch> for every Application. The Applications must be re-imported from the GitOps repo; they cannot be recovered from the cluster
  12. 12STEP 12 - For Flux re-apply after reinstall: every Kustomization and HelmRelease CR must be re-applied from the GitOps repo. kubectl apply -f <path-to-cr> for each. The source-controller will re-establish the Git connection; the kustomize-controller/helm-controller will reconcile
  13. 13STEP 13 - Verify the controller recovers. argocd app list shows all Applications (re-imported, Synced, Healthy). flux get kustomization -A shows all kustomizations (Ready=True). A forced refresh produces a new sync: argocd app get <app> --refresh. The controller is recovered

4 · Verification

Confirm the procedure actually fixed the problem.

  • ✓kubectl get pods -n argocd -l app.kubernetes.io/name=argocd-application-controller shows all pods Running (HA: 3, non-HA: 1). No CrashLoopBackOff. No OOMKilled in kubectl describe
  • ✓kubectl get pods -n argocd -l app=redis shows Redis running (HA mode only)
  • ✓argocd app list returns the same Applications that were registered before the failure
  • ✓argocd app list reports every Application as Synced + Healthy
  • ✓flux get kustomization -A reports Ready=True for every kustomization
  • ✓A forced refresh (argocd app get <app> --refresh) produces a new sync attempt and succeeds
  • ✓The controller metrics endpoint shows argocd_app_reconcile_count increasing (controller is doing work)
  • ✓End-to-end: a test manifest change in Git results in the cluster being updated within the sync interval

5 · Rollback

If verification fails, undo the procedure in reverse order.

  • ↶If the recovery fails and the controller is in a worse state than before: revert to the previous known-good state. For Helm-installed controllers: helm rollback argocd <previous-revision> -n argocd. For Flux: re-apply the previous known-good set of CRs
  • ↶If the database was restored from a backup but the controller is using a newer schema: the controller will fail to start. Run the database migrations: Argo CD argocd-db-migration job or kubectl create job --from=cronjob/argocd-db-migration argocd-db-migration-once -n argocd. The job upgrades the database schema to match the controller version
  • ↶If Applications cannot be re-imported because the GitOps repo is unavailable: see git-cicd-gitops-rb-21-recover-git-hosting-dependency. The recovery is blocked on Git access
  • ↶If the controller is recovered but Applications remain OutOfSync: the manifest sync is independent of the controller recovery. See git-cicd-gitops-rb-18-troubleshoot-argocd-reconcile for the application-side reconciliation
  • ↶If the controller's HA setup was non-HA and the single-pod failure was caused by a node failure: the recovery is temporary until the node is fixed. Restart the controller on a different node with kubectl delete pod -n argocd -l app.kubernetes.io/name=argocd-application-controller and a nodeSelector patch. The node must be investigated separately
  • ↶If the recovery required re-importing every Application and the GitOps repo is large (>500 Applications): the recovery is slow. Scale out the repo server for manifest-generation throughput (kubectl scale deploy/argocd-repo-server -n argocd --replicas=3) and drive argocd app create from a script. Document the recovery time in the incident postmortem
  • ↶If the controller is recovered but the cluster was the failure cause (etcd corruption, network partition): the controller recovery is not enough. See kubernetes runbooks for cluster recovery

6 · Escalation

When the runbook isn't enough, contact:

  • · The controller is recovered but applications are still Degraded in a way that suggests a cluster-wide issue (etcd, CNI, ingress): the cluster itself is broken. Engage the platform team immediately
  • · The controller database (Postgres) is down and cannot be recovered from a backup: the controller's source of truth is lost. Reinstall the controller and re-import every Application from the GitOps repo. This is a multi-hour recovery; engage the platform team and inform the application owners
  • · The controller namespace was lost and the GitOps repo is also lost: this is the worst case. The cluster has no GitOps coverage. Engage the platform team to re-establish GitOps and re-bootstrap the cluster from any remaining infrastructure-as-code
  • · The controller is recovered but the cluster is still in an unmanaged state (no Applications registered, no resources reconciled): the GitOps onboarding for this cluster must be re-done. This is a multi-day recovery; engage the platform team and the application owners
  • · The controller recovery triggered a mass reconciliation that is overwhelming the cluster: scale the controller back (kubectl scale statefulset/argocd-application-controller -n argocd --replicas=1, keeping the ARGOCD_CONTROLLER_REPLICAS env var in step with the replica count whenever you scale), set --status-processors=1 and --kubectl-parallelism-limit=1, and let the reconciliation drain. Do not increase parallelism to "fix" the slowness
  • · The controller is recovering but the GitOps repo is being actively rewritten (a force-push incident): see git-cicd-gitops-rb-06-respond-to-force-push-incident. The controller recovery must wait for Git to stabilize

The GitOps controller is the cluster”s brain. When the controller is down, the cluster stops reconciling — Applications stay in the state they were in when the controller died. The recovery depends on the failure class: a single-pod crash is a delete-and-recreate; a full controller outage is a database/restore problem; a namespace loss is a full reinstall.

1. Capture pre-recovery state

Read-only / Safe
$ echo "--- Argo CD namespace state ---"
kubectl get all,cm,secret -n argocd -o yaml > /tmp/argocd-pre-recovery.yaml
kubectl get pods -n argocd -o wide
echo "--- controller logs (previous instance) ---"
kubectl logs -n argocd -l app.kubernetes.io/name=argocd-application-controller --previous --tail=200
echo "--- events ---"
kubectl get events -n argocd --sort-by=.lastTimestamp | tail -30

The pre-recovery capture is the baseline. The previous-instance logs contain the failure cause. Events name the failure class (image pull, OOMKilled, probe failure).

2. Single-pod recovery

Read-only / Safe
$ NS="argocd"
LABEL="app.kubernetes.io/name=argocd-application-controller"
kubectl delete pod -n "$NS" -l "$LABEL"
kubectl rollout status statefulset/argocd-application-controller -n "$NS" --timeout=120s
echo "--- new pod ---"
kubectl get pods -n "$NS" -l "$LABEL"
echo "--- new pod logs ---"
kubectl logs -n "$NS" -l "$LABEL" --tail=100

A single-pod CrashLoopBackOff is usually a transient resource contention. Deleting the pod forces Kubernetes to create a new one on a different node (or with a new PID). If the new pod also crashes, escalate.

3. All-pod recovery: fix the cause, not the symptom

Read-only / Safe
$ NS="argocd"
LABEL="app.kubernetes.io/name=argocd-application-controller"
echo "--- common causes ---"
kubectl describe pod -n "$NS" -l "$LABEL" | grep -E 'Failed|ImagePull|OOMKilled|BackOff' | head -20
echo "--- recent logs (all pods) ---"
kubectl logs -n "$NS" -l "$LABEL" --previous --tail=100 --all-containers | head -100
echo "--- events ---"
kubectl get events -n "$NS" --sort-by=.lastTimestamp | tail -30

The describe output names the failure class. The previous-instance logs give the stack trace. Events give the kubelet”s perspective. All three together identify the cause.

4. Fix common causes

Read-only / Safe
$ NS="argocd"
echo "--- image pull failure: check the image ---"
kubectl get statefulset argocd-application-controller -n "$NS" -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
echo "--- OOMKilled: bump the memory limit ---"
kubectl set resources statefulset/argocd-application-controller -n "$NS" --limits=memory=2Gi,cpu=2 --requests=memory=1Gi,cpu=500m
echo "--- RBAC: verify the SA ---"
kubectl get statefulset argocd-application-controller -n "$NS" -o jsonpath='{.spec.template.spec.serviceAccountName}{"\n"}'
kubectl auth can-i list applications --as=system:serviceaccount:argocd:argocd-application-controller -n argocd
echo "--- rollout restart after fix ---"
kubectl rollout restart statefulset/argocd-application-controller -n "$NS"
kubectl rollout status statefulset/argocd-application-controller -n "$NS" --timeout=300s

Each cause has a different fix. Image pull: patch the image. OOMKilled: bump the limit. RBAC: patch the ClusterRole. After the fix, rollout restart and wait for healthy.

5. Recover the database backend (Argo CD HA)

Read-only / Safe
$ NS="argocd"
echo "--- redis ---"
kubectl get pods -n "$NS" -l app=redis
echo "--- postgres ---"
kubectl get pods -n "$NS" -l app.kubernetes.io/name=argocd-server,role=server -o jsonpath='{.items[*].metadata.labels.app}'
echo "--- database migrations if needed ---"
kubectl create job --from=cronjob/argocd-db-migration argocd-db-migration-once -n "$NS" --dry-run=client -o yaml | kubectl apply -f -
kubectl wait --for=condition=complete job/argocd-db-migration-once -n "$NS" --timeout=600s

Argo CD HA uses Redis (cache) and Postgres (source of truth). The controller cannot start without both. Restore the database first; the controller will recover automatically once the database is healthy.

6. Reinstall (namespace-loss scenario)

Read-only / Safe
$ echo "--- Argo CD ---"
helm repo add argo-cd https://argoproj.github.io/argo-helm
helm repo update
helm install argocd argo-cd/argo-cd --namespace argocd --create-namespace --version REPLACE_WITH_CHART_VERSION
echo "--- wait for controller ---"
kubectl rollout status statefulset/argocd-application-controller -n argocd --timeout=300s
echo "--- Flux ---"
flux install --namespace flux-system --export > /tmp/flux-install.yaml
kubectl apply -f /tmp/flux-install.yaml

A namespace loss requires a full reinstall. The Helm chart (Argo CD) or flux install (Flux) re-creates the controllers. After reinstall, re-import every Application or re-apply every Kustomization/HelmRelease.

7. Re-import applications (Argo CD)

Read-only / Safe
$ GITOPS_REPO="/path/to/gitops-prod"
cd "$GITOPS_REPO"
echo "--- enumerate Applications in Git ---"
find . -name '*.yaml' | xargs grep -l 'kind: Application' | head
echo "--- re-import each Application ---"
for app_file in $(find . -name '*.yaml' | xargs grep -l 'kind: Application'); do
argocd app create -f "$app_file"
done
echo "--- verify ---"
argocd app list

Applications are not recovered from the cluster — they are recovered from the GitOps repo. Iterate over every Application manifest and argocd app create -f.

Verification

kubectl get pods -n argocd -l app.kubernetes.io/name=argocd-application-controller shows all pods Running with no CrashLoopBackOff. Redis running (HA mode). argocd app list returns the same Applications that were registered before the failure and reports Synced + Healthy. A forced refresh produces a new sync attempt and succeeds. The controller metrics endpoint shows argocd_app_reconcile_count increasing.

Rollback

If the recovery fails and the controller is in a worse state, revert to the previous known-good Helm revision. If the database was restored from a backup but the controller is using a newer schema, run the database migrations job. If Applications cannot be re-imported because the GitOps repo is unavailable, the recovery is blocked on Git access. If Applications remain OutOfSync, see git-cicd-gitops-rb-18-troubleshoot-argocd-reconcile. If the controller”s HA setup was non-HA and the single-pod failure was caused by a node failure, restart the controller on a different node. If the recovery required re-importing every Application and the GitOps repo is large, expect a slow recovery — document the time.

References

  1. Argo CD — Operator Manual
  2. Argo CD — High Availability
  3. Argo CD — Installation
  4. Argo CD — Disaster Recovery
  5. Flux — Installation
  6. Flux — Recovery Procedures