high riskcluster affecting~60 min

Runbook: Validate GitOps After Cluster Recovery

1 · Prerequisites

Confirm every item is in place before any state change.

git-cicd-gitops-rb-20-recover-argocd-controller
git-cicd-gitops-rb-18-troubleshoot-argocd-reconcile
git-cicd-gitops-rb-19-reconcile-emergency-manual-change
kubectl access to the recovered cluster
Access to the GitOps repository (manifests)
Knowledge of which Applications/Kustomizations/HelmReleases belong to the cluster

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Confirm the cluster is recovered and reachable. kubectl get nodes returns the expected node count; kubectl get ns returns the expected namespaces. A cluster that cannot be reached cannot be validated
· Identify the cluster's identity in the GitOps controller. Argo CD: argocd cluster list. Flux: the cluster's KubeConfig in the flux-system namespace. The cluster identity determines which Applications should be reconciled
· Capture the cluster's current state. kubectl get all,cm,secret,pvc,sa,clusterrole,clusterrolebinding -A -o yaml > /tmp/recovered-cluster.yaml. The capture is the baseline; the diff against Git is the validation
· Identify the expected Applications/Kustomizations for this cluster. Argo CD: read the GitOps repo and list all Application CRs with destination.server matching this cluster. Flux: list all Kustomizations/HelmReleases for this cluster. The expected set is what must be reconciled
· Verify the GitOps controller is installed and reachable. Argo CD: kubectl get pods -n argocd. Flux: kubectl get pods -n flux-system. If the controller is not installed, install it first (see git-cicd-gitops-rb-20-recover-argocd-controller)
· Verify the controller's credential to the Git repo works. See git-cicd-gitops-rb-17-troubleshoot-argocd-auth. A controller that cannot read Git cannot reconcile anything
· Identify the cluster's expected node count, namespace list, and version. These are in the cluster's bootstrap manifests (e.g., Terraform output) or the team's cluster inventory. The expected state is the validation target

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1STEP 1 - Compare the recovered cluster to the expected state. kubectl get nodes -o wide and kubectl get ns. Diff against the cluster inventory: diff <(kubectl get nodes --no-headers | awk "{print \\$1}") <(cat /tmp/expected-nodes.txt). The diff is the recovery gap
2STEP 2 - Verify the GitOps controller is healthy. Argo CD: kubectl get pods -n argocd -o wide shows all pods Running. Flux: kubectl get pods -n flux-system -o wide shows all pods Running. CrashLoopBackOff or ImagePullBackOff is a controller problem, not a cluster problem
3STEP 3 - Verify the controller's Git connection. Argo CD: argocd repo list shows all expected repos with status Successful. Flux: flux check returns no errors. A failed connection means the controller cannot pull manifests
4STEP 4 - For Argo CD: trigger a sync of every Application for this cluster. argocd app list -o wide enumerates the Applications; for app in $(argocd app list -o name); do argocd app sync "$app" --force; done. The --force flag overrides any sync-protection. Watch the progress: argocd app list -o wide
5STEP 5 - For Flux: trigger a reconciliation of every Kustomization and HelmRelease. flux reconcile takes one object name at a time, so iterate: flux get kustomizations --no-header | awk "{print \\$1}" | xargs -r -n1 flux reconcile kustomization --with-source, and for HelmReleases include the namespace: flux get helmreleases -A --no-header | awk "{print \\$1, \\$2}" | while read -r ns name; do flux reconcile helmrelease "$name" -n "$ns" --with-source; done. The --with-source flag forces a Git refresh
6STEP 6 - Detect drift between the cluster and Git. Argo CD: argocd app diff <app> for each Application. Flux: flux diff kustomization <name>. The diff names the resources that the controller wants to create, update, or delete. Unexpected diffs indicate either an out-of-date Git state or a recovered cluster that does not match Git
7STEP 7 - For unexpected resources (resources in the cluster that are not in Git): they were either created manually during the recovery or survived the recovery. Argo CD will mark them as "Extra" in the diff. Decide: delete them via the controller (add to .gitignore or remove from the cluster manually with kubectl delete), or commit them to Git if they are intentional
8STEP 8 - For missing resources (resources in Git but not in the cluster): they did not survive the recovery. Trigger a sync with prune to re-create them. Argo CD: argocd app sync <app> --replace --prune. Flux: pruning is controlled by the Kustomization's spec.prune field, not a CLI flag — verify it is enabled (kubectl get kustomization <name> -n flux-system -o jsonpath="{.spec.prune}" returns true), then flux reconcile kustomization <name> --with-source
9STEP 9 - Verify every Application is Synced + Healthy. argocd app list -o wide | grep -E "OutOfSync|Degraded" returns nothing. flux get kustomization -A | grep -v "True\\s*True" returns nothing. Every Application is reconciled and healthy
10STEP 10 - Run a smoke test on representative workloads. For each tier of the application (frontend, backend, database): curl -fsS https://<service>/healthz | jq .status and curl -fsS https://<service>/readyz | jq .checks. The smoke test verifies the application stack, not just the Kubernetes objects
11STEP 11 - Verify observability is working. The metrics endpoint for each workload returns 200: curl -fsS http://<pod-ip>:8080/metrics | head. The Prometheus federation is reachable: kubectl port-forward svc/prometheus 9090 and check the targets. The logs are being collected: kubectl logs -n <ns> -l app=<service> --tail=10
12STEP 12 - Compare the cluster's resource utilization to the pre-recovery baseline. kubectl top nodes and kubectl top pods -A. A recovered cluster should show similar CPU/memory utilization to the pre-recovery state; large deviations indicate a missing workload (the controller did not reconcile it) or a runaway workload (a bug introduced during recovery)
13STEP 13 - Verify RBAC and ServiceAccounts. kubectl auth can-i list pods --as=system:serviceaccount:argocd:argocd-application-controller returns yes. kubectl auth can-i get secrets -n kube-system --as=system:serviceaccount:default:default returns no (default SA should not have cluster-wide access). The recovered cluster should have the same RBAC posture as the pre-recovery cluster
14STEP 14 - Document the validation result. Open a ticket with: the cluster identity, the Applications reconciled, the drift found (resources added/removed/updated), the smoke test results, the operator, the time, the differences from the expected state. The record is the audit trail and the basis for any follow-up

4 · Verification

Confirm the procedure actually fixed the problem.

✓Every expected Application is Synced + Healthy (Argo CD) or Ready=True (Flux). No OutOfSync, no Degraded
✓The cluster's node count, namespace list, and version match the expected state
✓A smoke test on representative workloads returns 200: curl -fsS https://<service>/healthz | jq .status
✓Observability is working: metrics endpoint returns 200, logs are being collected, Prometheus targets are up
✓Resource utilization is within 20% of the pre-recovery baseline (CPU, memory, pod count)
✓RBAC posture matches the pre-recovery baseline: kubectl auth can-i tests return the same answers
✓The cluster's identity in the GitOps controller matches the cluster inventory (no orphaned Applications pointing to a different cluster)
✓A test manifest change in Git results in the cluster being updated within the sync interval (end-to-end validation)

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶If the validation fails for a specific Application and the controller cannot reconcile it: see git-cicd-gitops-rb-18-troubleshoot-argocd-reconcile. The controller-side fix is independent of the cluster recovery
↶If the validation reveals that the recovered cluster is missing a critical resource (a database, a service mesh sidecar, a cert-manager): the recovery is incomplete. Re-run the cluster recovery process; the missing resource indicates a step that was skipped
↶If the smoke test fails but the Kubernetes objects are correct (pods Running, services have endpoints): the application is broken, not the cluster. Investigate the application. The cluster is recovered; the application has a separate problem
↶If the validation reveals that the GitOps manifests do not match the recovered cluster (the manifests are outdated because the recovery used a different version): sync the cluster to the manifests and accept the differences, or update the manifests to match the recovered state. The decision must be made by the application owner; do not unilateral apply either
↶If observability is not working (metrics endpoint returns 503, Prometheus targets are down): the recovery did not re-deploy the observability stack. See the observability runbook for recovery. The cluster may be functional but invisible
↶If RBAC is too permissive in the recovered cluster (the recovery used a broader ClusterRole than the pre-recovery state): the recovery is insecure. Engage the security team. The cluster may need to be re-bootstrapped with the correct RBAC
↶If the validation takes longer than the recovery time objective (RTO) and the cluster is still partially functional: prioritize by business impact. Critical workloads first (production user-facing), internal tools last. The validation can be staged

6 · Escalation

When the runbook isn't enough, contact:

· The recovered cluster is missing the GitOps controller entirely: the cluster is unmanaged. Reinstall the controller (helm install argocd argo-cd/argo-cd) and re-import every Application. This is a multi-hour recovery; engage the platform team
· The recovered cluster has drifted significantly from Git (hundreds of resource differences): the recovery used a different version or a different backup. The decision to accept the drift or re-reconcile must be made by the engineering leadership. Engage the application owners and the platform team
· The validation reveals that critical secrets were not restored (database passwords, API tokens, TLS certs): the cluster is non-functional until secrets are restored. Engage the security team and the secrets-management team. Secrets must be restored from a backup or rotated
· The recovered cluster is healthy but the upstream dependencies it relies on are not (a database service in another region, a SaaS API that was down during recovery): the cluster is recovered; the dependencies are not. Engage the application owner. The cluster may be functional but useless until dependencies are available
· The validation surfaces a security issue (a workload running with too many permissions, a Secret in plaintext, an unauthenticated service): the recovery was insecure. Engage the security team immediately. The cluster must be re-bootstrapped with the correct security posture
· The validation finds resources that were not in Git and not in the pre-recovery cluster inventory (unknown workloads, unknown ConfigMaps): investigate. These may be leftover from a previous tenant, a forgotten experiment, or unauthorized activity. Engage the security team
· The cluster was recovered but the GitOps repo is also lost: this is the worst case. The cluster has no source of truth. Engage the platform team and the engineering leadership; the GitOps coverage must be re-established from any remaining infrastructure-as-code

A recovered cluster is functional but not yet aligned to the team”s desired state. The GitOps controller is the bridge — it reads Git, compares to the cluster, and reconciles. The validation drives every Application to Synced + Healthy and proves the cluster is back under GitOps control.

1. Verify the cluster is recovered and the controller is healthy

Read-only / Safe

$ echo "--- cluster nodes ---"
kubectl get nodes -o wide
echo "--- namespaces ---"
kubectl get ns
echo "--- controller pods ---"
kubectl get pods -n argocd
kubectl get pods -n flux-system
echo "--- controller logs ---"
kubectl logs -n argocd -l app.kubernetes.io/name=argocd-application-controller --tail=100 | tail -30

The cluster recovery is complete when nodes and namespaces match the inventory and the controller pods are Running. The controller logs should show no startup errors.

2. Verify the Git connection

Read-only / Safe

$ echo "--- Argo CD repos ---"
argocd repo list
echo "--- test ls-remote against each ---"
REPO=$(argocd repo list -o json | jq -r '.[0].repo')
kubectl exec -n argocd deploy/argocd-repo-server -- git ls-remote "$REPO" 2>&1 | head
echo "--- Flux ---"
flux check 2>&1 | head -20

The Git connection is the controller”s lifeline. A failed connection means the controller cannot reconcile. The ls-remote test confirms the controller can pull.

3. Trigger sync of every Application (Argo CD)

Read-only / Safe

$ echo "--- enumerate Applications ---"
argocd app list -o wide > /tmp/apps-before-sync.txt
wc -l /tmp/apps-before-sync.txt
echo "--- sync each ---"
for app in $(argocd app list -o name); do
echo "syncing: $app"
argocd app sync "$app" --force 2>&1 | tail -3
done
echo "--- wait for all to settle ---"
for app in $(argocd app list -o name); do
argocd app wait "$app" --health --timeout 300 2>&1 | tail -1
done
echo "--- post-sync status ---"
argocd app list -o wide | grep -E "OutOfSync|Degraded|Unknown" || echo "ALL HEALTHY"

The --force flag overrides sync-protection. The --timeout 300 gives each Application 5 minutes to reconcile. The post-sync status should show every Application as Synced + Healthy.

4. Reconcile every Kustomization and HelmRelease (Flux)

Read-only / Safe

$ echo "--- enumerate ---"
flux get kustomization -A > /tmp/kust-before.txt
flux get helmrelease -A >> /tmp/kust-before.txt
wc -l /tmp/kust-before.txt
echo "--- reconcile each ---"
flux get kustomizations --no-header | awk '{print $1}' | xargs -r -n1 flux reconcile kustomization --with-source
flux get helmreleases -A --no-header | awk '{print $1, $2}' | while read -r ns name; do
flux reconcile helmrelease "$name" -n "$ns" --with-source
done
echo "--- wait ---"
sleep 60
echo "--- post-reconcile status ---"
flux get kustomization -A | grep -vE "True\s+True\s+Applied" || echo "ALL READY"
flux get helmrelease -A | grep -vE "True\s+True\s+Released" || echo "ALL RELEASED"

The --with-source flag forces a Git refresh. flux reconcile takes one object name at a time, so the loops cover every Kustomization and HelmRelease (passing the namespace for each HelmRelease). The post-reconcile status should show every kustomization as Ready=True and Applied.

5. Detect drift

Read-only / Safe

$ echo "--- Argo CD apps with drift ---"
argocd app list -o json | jq -r '.[] | select(.status.sync.status != "Synced") | .metadata.name'
echo "--- diff for each drifted app ---"
for app in $(argocd app list -o json | jq -r '.[] | select(.status.sync.status != "Synced") | .metadata.name'); do
echo "=== $app ==="
argocd app diff "$app" | head -30
done

The drift is the gap between Git and the cluster. The diff names the resources that need to be created, updated, or deleted.

6. Reconcile unexpected or missing resources

Read-only / Safe

$ APP="checkout-api"
echo "--- unexpected resources (in cluster, not in Git) ---"
argocd app manifests "$APP" > /tmp/git-manifests.yaml
kubectl get all,cm,secret,pvc -n prod -o yaml > /tmp/cluster-resources.yaml
echo "--- missing resources (in Git, not in cluster) ---"
argocd app diff "$APP" --server-side-generate 2>&1 | grep -A3 "will be created" | head
echo "--- sync with prune to clean up ---"
argocd app sync "$APP" --replace --prune
argocd app wait "$APP" --health --timeout 300

The --prune flag tells the controller to delete resources that are in the cluster but not in Git. Use with caution — it will delete anything not in Git, including resources that may have been intentionally added during the recovery.

7. Smoke test representative workloads

Read-only / Safe

$ echo "--- frontend ---"
curl -fsS https://frontend.prod.internal/healthz | jq .status
echo "--- backend ---"
curl -fsS https://api.prod.internal/healthz | jq .status
echo "--- database ---"
kubectl exec -n db deploy/postgres -- pg_isready -U postgres
echo "--- service mesh ---"
kubectl get pods -n istio-system
echo "--- ingress ---"
kubectl get ingress -A | head

The smoke test exercises the application stack, not just the Kubernetes objects. A passing smoke test proves the cluster is functional end-to-end.

8. Verify observability and RBAC

Read-only / Safe

$ echo "--- metrics endpoints ---"
POD=$(kubectl get pod -n prod -l app=checkout-api -o jsonpath='{.items[0].metadata.name}')
kubectl exec -n prod "$POD" -- curl -fsS http://localhost:8080/metrics | head -5
echo "--- Prometheus targets ---"
kubectl port-forward -n monitoring svc/prometheus 9090 &
sleep 5
curl -fsS http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | select(.health != "up") | .labels.job' | head
echo "--- RBAC posture ---"
kubectl auth can-i list pods --as=system:serviceaccount:argocd:argocd-application-controller
kubectl auth can-i get secrets -n kube-system --as=system:serviceaccount:default:default
echo "--- resource utilization ---"
kubectl top nodes
kubectl top pods -A | head

The metrics endpoint confirms observability is wired. The Prometheus target check confirms the federation is up. The RBAC tests confirm the cluster”s security posture.

9. Document the validation result

Read-only / Safe

$ CLUSTER="prod-use1"
gh issue create --repo myorg/myorg --title "GitOps validation: cluster $CLUSTER recovered" \
--body "Cluster: $CLUSTER. Operator: $USER. Time: $(date -u +%FT%TZ). Apps reconciled: $(argocd app list -o name | wc -l). Drift found: REPLACE_WITH_LIST. Smoke test: PASS. Observability: PASS. RBAC: PASS. Follow-up: REPLACE_WITH_LIST." \
--label cluster-recovery --label gitops --label validation

The validation record closes the loop. The cluster is recovered and under GitOps control; the audit trail is captured.

Verification

Every expected Application is Synced + Healthy (Argo CD) or Ready=True (Flux). The cluster”s node count, namespace list, and version match the expected state. A smoke test on representative workloads returns 200. Observability is working (metrics, logs, Prometheus). Resource utilization is within 20% of the pre-recovery baseline. RBAC posture matches the pre-recovery baseline. A test manifest change in Git results in the cluster being updated within the sync interval.

Rollback

If a specific Application cannot reconcile, see the controller reconcile runbook. If the cluster is missing a critical resource, re-run the cluster recovery. If the smoke test fails but Kubernetes objects are correct, the application has a separate problem. If manifests do not match the recovered cluster, sync or update the manifests — the decision belongs to the application owner. If observability is broken, see the observability runbook. If RBAC is too permissive, engage security. If validation takes longer than RTO, prioritize by business impact.

1 · Prerequisites

2 · Pre-checks

3 · Procedure

4 · Verification

5 · Rollback

6 · Escalation

1. Verify the cluster is recovered and the controller is healthy

2. Verify the Git connection

3. Trigger sync of every Application (Argo CD)

4. Reconcile every Kustomization and HelmRelease (Flux)

5. Detect drift

6. Reconcile unexpected or missing resources

7. Smoke test representative workloads

8. Verify observability and RBAC

9. Document the validation result

Verification

Rollback

References