Runbook: Reconcile an Emergency Manual Production Change

1 · Prerequisites

Confirm every item is in place before any state change.

• git-cicd-gitops-rb-18-troubleshoot-argocd-reconcile
• kubectl configured against the affected cluster context
• Access to the GitOps repository and permission to push to the protected branch (or via PR)
• Knowledge of which Application/namespace the changed resource belongs to
• For Helm/Flux: knowledge of the HelmRelease/Kustomization that owns the resource

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

• · Identify the changed resource: kubectl get <resource>/<name> -n <ns> and compare to the last-known Git manifest. The diff is the manual change
• · Identify the GitOps owner of the resource. Argo CD: argocd app get <app> -o jsonpath='{.status.resources[?(@.name=="<resource>")].group}' and look for the namespace, app, and source path. Flux: flux tree --namespace <ns> to find the Kustomization/HelmRelease that owns the resource
• · Capture the full live state of the changed resource: kubectl get <resource>/<name> -n <ns> -o yaml > /tmp/live-resource.yaml. This is the artifact that must be committed to Git
• · Capture the Git history of the resource. git log --oneline -- <path-to-manifest> shows the most recent commits that touched the manifest. The manual change should be committed as a new commit referencing the incident ID, not amending history
• · Confirm the manual change is still needed. If the manual change was a hotfix and a proper PR is in progress, the manual change can be reverted to align with the PR. If the manual change is the only fix, it must be preserved in Git
• · Identify the change approver. GitOps branches are typically protected. The PR will need approval from the platform team or the application owner. Identify the approver BEFORE pushing the PR, not after

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1. 1STEP 1 - Extract the effective manifest. Argo CD can produce the live manifest: argocd app manifests <app> shows what the controller thinks the manifest is; argocd app resources <app> shows the live cluster state. For pure cluster extraction: kubectl get <resource>/<name> -n <ns> -o yaml --export > /tmp/effective-manifest.yaml (the --export flag strips cluster-specific fields, present in older Kubernetes versions; on newer versions, use kubectl get ... -o yaml | yq to strip status, managedFields, etc.)
2. 2STEP 2 - Strip server-side noise from the manifest. Server-applied fields (status, metadata.managedFields, metadata.resourceVersion, metadata.uid, metadata.creationTimestamp) must be removed before committing. kubectl get <resource>/<name> -n <ns> -o yaml | yq 'del(.status) | del(.metadata.managedFields) | del(.metadata.resourceVersion) | del(.metadata.uid) | del(.metadata.creationTimestamp)' > /tmp/clean-manifest.yaml. Diff against the Git version: diff -u <path-to-manifest-in-git> /tmp/clean-manifest.yaml
3. 3STEP 3 - Update the manifest in Git. Either replace the existing file (the manual change is now the desired state) or amend the file with just the diff. Replacing is safer: it ensures the Git version exactly matches the live cluster. Commit with the incident ID in the message: git add <path> && git commit -m "hotfix(reconcile): manual change for INC-1234 - <description>"
4. 4STEP 4 - Push to the GitOps repo. If the branch is protected, push the branch and open a PR. The PR description must reference the incident, the manual change, the operator who made the manual change, and the reason Git did not have this change first. The PR must be reviewed by the platform team or the application owner before merge — this is the audit checkpoint
5. 5STEP 5 - After merge, trigger reconciliation. Argo CD: argocd app sync <app> --revision HEAD to pull the new commit. Flux: flux reconcile source git <name> then flux reconcile kustomization <name>. The controller must reconcile the new Git state with the live cluster. The desired outcome: the live cluster is unchanged (it was already in the new state) and the controller reports Synced
6. 6STEP 6 - Verify the application is Healthy. argocd app get <app> reports Synced + Healthy. flux get kustomization <name> reports Ready=True. The manual change is now in Git; the controller agrees the cluster matches Git
7. 7STEP 7 - For Helm values overrides done via helm upgrade --set or kubectl patch: the Git manifest is a HelmRelease. The override must be committed to the values file in Git. flux get helmrelease <name> -n <ns> -o yaml | yq '.spec.values' > /tmp/live-values.yaml and commit that as the new values.yaml. Re-sync and verify
8. 8STEP 8 - For Kustomize patches done via kubectl patch: the patch must be encoded in Git. Identify the patch: kubectl get <resource>/<name> -n <ns> --show-managed-fields -o yaml | yq '.metadata.managedFields[].fieldsType'. The patch is in the GitOps repo as a kustomization.yaml patch. Add the patch and commit
9. 9STEP 9 - For Argo CD Application overrides (argocd app set --revision): the override is in Argo CD, not Git. Reset to the Git default: argocd app unset <app> --revision. The controller will read the branch HEAD and reconcile to that
10. 10STEP 10 - Record the reconciliation. Open or update the incident ticket: link the Git commit, link the PR, capture the manual change that was made, capture the operator who made it, and capture the reason Git was bypassed. The record is the audit trail and the basis for any follow-up process improvement

4 · Verification

Confirm the procedure actually fixed the problem.

• ✓git diff <commit-before>..<commit-after> -- <path-to-manifest> shows the manual change is now in Git
• ✓kubectl get <resource>/<name> -n <ns> -o yaml matches the Git version (excluding server-side noise: status, managedFields, etc.)
• ✓argocd app get <app> reports Synced + Healthy
• ✓flux get kustomization <name> reports Ready=True
• ✓A subsequent refresh does not generate a new diff: argocd app diff <app> returns nothing
• ✓The incident ticket contains the Git commit SHA, the PR URL, the manual change description, the operator, and the bypass reason

5 · Rollback

If verification fails, undo the procedure in reverse order.

• ↶If the Git commit was wrong and the manual change was lost: revert the commit (git revert <commit>), push, and let the controller reconcile. The live cluster is restored to the pre-commit state
• ↶If the PR was rejected (the reviewer disagreed with the manual change): revert the manual change first (back out the production fix), then merge the rejection. The order matters — the live cluster must match the final Git state at all times
• ↶If the controller reports OutOfSync after the reconciliation: the manifest in Git does not exactly match the live cluster. Re-extract the live manifest (kubectl get ... -o yaml), diff against Git, and commit the diff. Repeat until the diff is empty
• ↶If the HelmRelease values file is overwritten by a future Helm release: the reconciliation may need to be re-applied. The Argo CD/Flux sync hooks prevent this; ensure remediationPolicy: remediate is set on the Kustomization or syncOptions: [ServerSideApply=true] on the Application
• ↶If the manual change is later determined to be incorrect: revert the Git commit, push, and let the controller reconcile the live cluster back to the previous Git state. The manual change is undone in the same way a Git-driven change is undone — through Git
• ↶If the operator who made the manual change is no longer available: the Git commit and the incident ticket are the record. Do not attempt to "reproduce" the manual change in Git without understanding why it was made — read the incident ticket first

6 · Escalation

When the runbook isn't enough, contact:

• · The manual change was a SECURITY fix (CVE patch, RBAC tightening) that was made before the change-control process could run: the reconciliation to Git must happen IMMEDIATELY. The change is in production; Git must catch up. Engage security to confirm the fix is correct and to add any additional hardening
• · The manual change was made by an unauthorized operator (no change ticket, no approval): the change is in production and must be reconciled, but the incident is also a process violation. Engage the platform team and the operator's manager; do not use the reconciliation as an opportunity to "launder" the unauthorized change
• · The manual change cannot be reproduced in Git (e.g., a manual kubectl scale that the manifest does not support): the manifest in Git must be updated to support the manual change as a code change. The controller will not allow an out-of-band scale; the only way to keep the manual change is to make the manifest reflect it
• · The controller is ignoring the Git commit because the manual change was made in a namespace the controller does not manage: see kubernetes runbooks for cluster recovery; the GitOps coverage gap is a separate problem
• · Multiple manual changes were made during the incident, possibly by different operators, with no consistent record: this is a coordination failure. The reconciliation must capture the END state of the live cluster, not every intermediate change. The intermediate changes are lost unless the operators recorded them; engage the incident commander
• · The manual change was made via the cloud provider's console (not kubectl) on a managed service (RDS, GKE, EKS node group): the GitOps manifest cannot reflect a console change. See kubernetes or the cloud-provider runbook for managed-service reconciliation

References

1. Argo CD — Sync Best Practices
2. Argo CD — Application Specification
3. Flux — Kustomize Reconciler
4. Flux — Helm Reconciler
5. kubectl — jsonpath Reference
6. Kubernetes — Server-Side Apply