high riskservice affecting~45 min

Runbook: Troubleshoot GitOps Reconciliation (OutOfSync, Degraded)

1 · Prerequisites

Confirm every item is in place before any state change.

git-cicd-gitops-rb-17-troubleshoot-argocd-auth
kubectl configured against the affected cluster context
Access to the GitOps controller UI/CLI (argocd, flux)
Read access to the Git repository containing the manifests

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Identify which controller is reporting the issue. Argo CD: argocd app list -o wide. Flux: flux get kustomization -A; flux get helmrelease -A. List all degraded applications, not just the one reported — a single root cause can affect many
· Capture the current sync status and the desired state. Argo CD: argocd app get <app> -o yaml. Flux: kubectl get kustomization/<name> -n <ns> -o yaml and kubectl get helmrelease/<name> -n <ns> -o yaml. The full YAML is required to see the source of the divergence
· Capture the controller's recent logs filtered by the failing app: Argo CD kubectl logs -n argocd -l app.kubernetes.io/name=argocd-application-controller --tail=300 | grep -i <app>; Flux kubectl logs -n flux-system -l app=kustomize-controller --tail=300 | grep -i <name>. The error messages identify the failure class
· Verify Git auth is working. If authentication is broken, the reconciliation is broken regardless of the manifest content. See git-cicd-gitops-rb-17-troubleshoot-argocd-auth
· Capture the diff between desired (Git) and live (cluster) state. Argo CD: argocd app diff <app>. Flux: kubectl get kustomization/<name> -n <ns> -o jsonpath='{.status}' | jq. The diff is the source of truth for what the controller wants to change

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1STEP 1 - Classify the failure. Three classes: (a) OutOfSync with a known diff — manifests changed but not applied; (b) Degraded with sync OK — manifests applied but health check failing; (c) Degraded with sync Failed — manifests could not be applied. Each class has a different fix
2STEP 2 - For class (a) OutOfSync with a known diff: review the diff with argocd app diff <app> or flux diff kustomization <name>. If the diff is intentional (a new commit was pushed and not yet applied), trigger a sync: argocd app sync <app> or flux reconcile kustomization <name>. If the diff is unintentional (someone modified a resource directly in the cluster), see git-cicd-gitops-rb-19-reconcile-emergency-manual-change
3STEP 3 - For class (a) with no diff: the application is OutOfSync but the controller cannot determine the diff. This is usually a transient state during a sync. Wait 60 seconds and refresh. If still OutOfSync, check the source repo (argocd app get <app> → Status.ComparisonResult). The revision field will show whether the controller is looking at the right commit
4STEP 4 - For class (b) Degraded with sync OK: manifests are applied but the health check is failing. Argo CD health checks are Lua-based and run against the live cluster. Get the health details: argocd app get <app> → Status.Health. Inspect the failing resource: kubectl get <resource>/<name> -n <ns> -o yaml. Common causes: a Deployment with no Ready replicas (image pull failure, readiness probe failing), a Service with no endpoints, a PVC unbound
5STEP 5 - For class (b) Lua health check errors: Argo CD custom health checks live in argocd-cm ConfigMap under resource.customizations.health.<api-group>_<kind>. A typo in the Lua causes every resource of that kind to be Degraded. kubectl get cm -n argocd argocd-cm -o yaml | yq '.data["resource.customizations.health"]' | head. Roll back to the previous known-good Lua
6STEP 6 - For class (c) Degraded with sync Failed: manifests could not be applied. Get the sync details: argocd app get <app> → Status.Sync.Result. The error message names the resource and the failure. Common causes: missing CRD (a dependency was uninstalled), RBAC denied (the controller's ServiceAccount lacks permission), webhook failure (an admission webhook is rejecting the resource), validation schema mismatch
7STEP 7 - For class (c) missing CRD: the CRD is owned by another Application or Helm release that is not yet installed. kubectl get crd | grep <crd-name>. If missing, identify the owner in the GitOps repo and sync it first: argocd app sync <crd-owner> && argocd app sync <failing-app>. For Flux: flux reconcile kustomization <crd-owner> then flux reconcile kustomization <failing-app>
8STEP 8 - For class (c) RBAC denied: the controller's ServiceAccount lacks the required permission. kubectl get clusterrolebinding,rolebinding -A | grep argocd. Test with kubectl auth can-i <verb> <resource> --as=system:serviceaccount:argocd:argocd-application-controller. Fix by patching the ClusterRole (Argo CD default is argocd-application-controller ClusterRole) and restart the controller
9STEP 9 - For class (c) webhook failure: an admission webhook is rejecting the resource. Get the webhook URL from the error message. Test the webhook: kubectl get validatingwebhookconfiguration,mutatingwebhookconfiguration -o yaml | grep -B2 -A5 <webhook-name>. Common cause: the webhook Service is unreachable (cert expired, Service down, NetworkPolicy blocks). See kubernetes runbooks for webhook recovery
10STEP 10 - For class (c) validation schema mismatch: the manifest references an API version the cluster does not know. kubectl api-resources | grep <kind>. If missing, the CRD is uninstalled or the manifest references an outdated version. Update the manifest to match the cluster's API version
11STEP 11 - Reset the controller's state if it is permanently corrupted. Argo CD: first make sure the resources-finalizer.argocd.argoproj.io finalizer is ABSENT — its presence makes the controller cascade-delete every resource the app manages when the Application is deleted: kubectl patch application/<app> -n argocd --type merge -p '{"metadata":{"finalizers":null}}'. Then delete only the Application object (argocd app delete <app> --cascade=false) and re-create it from the manifest; the workloads survive because no resources-finalizer is present. Use only as a last resort; this drops the controller's memory of the application
12STEP 12 - For Flux: hard-reset with flux reconcile kustomization <name> --with-source to also force a Git refresh. For deeper reset, kubectl annotate kustomization/<name> -n <ns> reconcile.fluxcd.io/requestedAt=$(date -u +%FT%TZ) --overwrite

4 · Verification

Confirm the procedure actually fixed the problem.

✓argocd app list reports every Application as Synced and Healthy (no OutOfSync or Degraded status)
✓flux get kustomization -A reports Ready=True and ReadyReason="" for every kustomization
✓A forced refresh produces a new sync attempt and the new sync succeeds: argocd app get <app> --refresh; flux reconcile kustomization <name>
✓The controller logs no longer contain the error class from the original failure: kubectl logs -n argocd -l app.kubernetes.io/name=argocd-application-controller --tail=200 | grep -i <error-class> returns nothing
✓End-to-end: a manifest change in Git results in the cluster being updated within the sync interval (Argo CD 3m default, Flux 1m default)

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶If a sync applied a bad manifest: revert the Git commit (the rollback is GitOps; see git-cicd-gitops-rb-16-roll-back-deployment). Do not try to undo the sync from the cluster side
↶If the controller was restarted or its state was reset and applications are now stuck: re-import them. Argo CD: argocd app create <app> --repo <url> --path <path> --dest-server <server> --dest-namespace <ns>. Flux: re-apply the Kustomization/HelmRelease CR
↶If a Lua health check was rolled back to fix a class (b) failure but the actual resource is still unhealthy: the rollback unmasked the real problem. Re-investigate the resource, not the health check
↶If the controller's RBAC was patched and now it has too much permission: roll back the ClusterRole to the previous known-good. The principle of least privilege still applies; the fix should not over-grant
↶If the controller state reset deleted an Application and re-creation is not possible (manifest lost): the cluster is now unmanaged for that Application. Engage the application owner; re-create the manifest from the GitOps repo before re-creating the Application
↶If reconciliation runs in an infinite loop (sync → fail → reset → sync): there is a structural problem (e.g., the controller cannot satisfy the manifest, the webhook is broken). Halt the reconciliation (argocd app set <app> --sync-policy none, re-enabling automated sync after the fix, or pause Flux with flux suspend kustomization <name>) and engage the platform team

6 · Escalation

When the runbook isn't enough, contact:

· Multiple applications are Degraded simultaneously with the same error class: a platform-level issue (CRD removal, controller RBAC, webhook outage). Engage the platform team immediately
· The controller is reporting Healthy but the cluster is not behaving as expected: the controller is not the source of truth; the cluster is. Inspect the cluster directly with kubectl get and kubectl describe. The controller may be lying
· The reconciliation succeeds but the application is not reachable from end users: a service mesh, ingress, or DNS issue. The GitOps side is healthy; the connectivity side is broken. Engage the networking team
· A namespace is completely unmanaged (no Application, no Kustomization): the cluster may have been provisioned without GitOps coverage. Engage the platform team to onboard the namespace
· A managed resource is missing from the cluster but is present in Git: the resource was deleted manually (kubectl delete) and the controller has not resynced. This is a class (a) variant — trigger a sync, but also investigate who deleted the resource and why. Manual deletion of GitOps-managed resources is a process violation
· The controller is running but the application's reconciliation is stuck on a network call (e.g., Helm chart download from a private registry that is unreachable): see git-cicd-gitops-rb-22-recover-registry-outage

A GitOps controller reports OutOfSync when the live cluster has diverged from Git, and Degraded when sync succeeds but health checks fail. Both are normal transient states that resolve themselves within the sync interval. The runbook is for when they do not — when an Application stays OutOfSync or Degraded across multiple reconcile cycles.

1. Enumerate all degraded applications

Read-only / Safe

$ echo "--- Argo CD applications ---"
argocd app list -o wide
echo "--- Flux kustomizations ---"
flux get kustomization -A
echo "--- Flux helm releases ---"
flux get helmrelease -A

A single root cause (RBAC, CRD, webhook) can affect many applications. Enumerate first; do not chase one application at a time.

2. Class A: OutOfSync with a known diff

Read-only / Safe

$ APP="checkout-api"
echo "--- diff between Git and cluster ---"
argocd app diff "$APP" | head -100
echo "--- sync status ---"
argocd app get "$APP" -o jsonpath='{.status.sync.status}{"\n"}'
echo "--- live resources ---"
kubectl get deploy "$APP" -n prod -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
echo "--- desired resources (from Git) ---"
argocd app manifests "$APP" | grep image: | head

The diff is the gap between Git and the cluster. If the diff is intentional (a new commit was pushed), trigger a sync. If the diff is unintentional (someone ran kubectl edit), the cluster has drifted from Git — the cluster must be reconciled to Git, not the other way around.

3. Class B: Degraded with sync OK (health check failure)

Read-only / Safe

$ APP="checkout-api"
echo "--- application health ---"
argocd app get "$APP" -o jsonpath='{.status.health.status}{"\n"}'
echo "--- failing resource ---"
argocd app get "$APP" -o jsonpath='{.status.resources[?(@.health.status=="Degraded")].name}{"\n"}'
echo "--- inspect the failing resource ---"
kubectl get deploy "$APP" -n prod -o yaml | head -40
echo "--- pod status ---"
kubectl get pods -n prod -l app="$APP"
kubectl describe pods -n prod -l app="$APP" | grep -A5 'Conditions:|Events:' | head -40

Sync OK means manifests were applied. Degraded means health is failing. The health is failing because the resource is not Ready (image pull failure, probe failing, etc.). Fix the resource, not the sync.

4. Class C: Degraded with sync Failed (manifest could not apply)

Read-only / Safe

$ APP="checkout-api"
echo "--- sync error ---"
argocd app get "$APP" -o jsonpath='{.status.conditions[?(@.type=="ComparisonError")].message}{"\n"}'
argocd app get "$APP" -o jsonpath='{.status.conditions[?(@.type=="SyncError")].message}{"\n"}'
echo "--- last sync result ---"
argocd app get "$APP" -o jsonpath='{.status.operationState.message}{"\n"}'
echo "--- relevant controller logs ---"
kubectl logs -n argocd -l app.kubernetes.io/name=argocd-application-controller --tail=300 | grep -A2 "$APP" | head -40

The sync error names the failure class: missing CRD, RBAC denied, webhook rejected, validation error. The class determines the fix.

5. Fix the four common sync-failed causes

Read-only / Safe

$ APP="checkout-api"
echo "--- missing CRD? ---"
kubectl get crd | grep -i REPLACE_WITH_CRD_NAME || echo "CRD missing"
echo "--- RBAC denied? ---"
kubectl auth can-i create deploy --as=system:serviceaccount:argocd:argocd-application-controller -n prod
echo "--- webhook rejected? ---"
kubectl get validatingwebhookconfiguration -o jsonpath='{.items[*].metadata.name}' | tr ' ' '\n' | grep -i REPLACE_WITH_WEBHOOK
echo "--- validation error? ---"
kubectl apply --dry-run=server -f /tmp/manifest.yaml 2>&1 | head -10
echo "--- then sync ---"
argocd app sync "$APP"

Each cause has a different fix: CRD missing → install the CRD owner first; RBAC denied → patch the controller”s ClusterRole; webhook rejected → see kubernetes webhook runbook; validation error → fix the manifest.

6. Force a hard refresh

Read-only / Safe

$ APP="checkout-api"
echo "--- Argo CD: hard refresh ---"
argocd app get "$APP" --refresh --hard-refresh
echo "--- Argo CD: re-sync ---"
argocd app sync "$APP" --force
echo "--- Flux: reconcile with source refresh ---"
flux reconcile kustomization "$APP" --with-source
echo "--- verify ---"
argocd app get "$APP"
flux get kustomization -A | grep "$APP"

--hard-refresh (Argo CD) and --with-source (Flux) re-pull the source repo and re-evaluate the manifest. --force overrides deletion protections. Use these only after the underlying fix; do not use force as the primary fix.

Verification

argocd app list reports every Application as Synced and Healthy (no OutOfSync or Degraded). flux get kustomization -A reports Ready=True for every kustomization. A forced refresh produces a new sync attempt and the new sync succeeds. The controller logs no longer contain the error class from the original failure. End-to-end: a manifest change in Git results in the cluster being updated within the sync interval.

Rollback

If a sync applied a bad manifest, revert the Git commit (the rollback is GitOps). If the controller state was reset and applications are now stuck, re-import them. If a Lua health check was rolled back but the actual resource is still unhealthy, the rollback unmasked the real problem — re-investigate the resource. If RBAC was patched to fix the issue but the patch over-grants, roll back to least privilege. If reconciliation runs in an infinite loop, halt it by disabling automated sync (argocd app set APP --sync-policy none) or suspending the Flux Kustomization, and engage the platform team.

1 · Prerequisites

2 · Pre-checks

3 · Procedure

4 · Verification

5 · Rollback

6 · Escalation

1. Enumerate all degraded applications

2. Class A: OutOfSync with a known diff

3. Class B: Degraded with sync OK (health check failure)

4. Class C: Degraded with sync Failed (manifest could not apply)

5. Fix the four common sync-failed causes

6. Force a hard refresh

Verification

Rollback

References