Post-incident Git reconciliation — the manual change must enter Git

The reconciliation commit is the Git-side closure of the break-glass procedure from LXXXVII-02 and LXXXVII-03. The emergency edit was applied to the cluster; the corresponding manifest in Git still says the old value. The next time self-heal is resumed, the controller will revert the edit; the next time an unrelated change is committed, the controller will produce a confusing diff. The reconciliation commit makes Git say what the cluster says, so the controller’s view of desired state matches the operator’s view of observed state.

The discipline is to commit the change before resuming self-heal. A commit-and-resume order produces a clean convergence; a resume-before-commit order produces a reversion loop that the responder must interrupt again.

flowchart LR
    A["Emergency edit applied to cluster"] --> B["Diff cluster vs Git"]
    B --> C["Update manifest in Git"]
    C --> D["Commit with incident ID in message"]
    D --> E["Resume self-heal"]
    E --> F["Validate convergence"]
    F --> G["Cluster == Git"]

What the reconciliation commit contains

The reconciliation commit is a regular commit on the regular branch. It is not a special branch; it is not a hotfix tag; it is the same Git history the team would produce for any other change. The contents are the manifest file(s) whose cluster state changed during the incident, with the values that the cluster now has.

The commit message is the one element that distinguishes the reconciliation commit from a regular commit:

fix(manifests): align api deployment with incident INC-1234

The api Deployment was running with image tag v3.2.7 during the
INC-1234 incident from 02:14 to 02:45 UTC. The manifest in Git
referenced v3.2.6, which produced a reversion loop while
self-heal was on. Update the manifest to v3.2.7 to align with
the cluster state and close the break-glass procedure.

Refs: INC-1234

The Refs: trailer links the commit to the incident record. The link is what the next on-call engineer searches; the link is what the auditor reads; the link is what the post-incident review cites. A reconciliation commit without an incident reference is indistinguishable from a regular commit and is the signal that the responder did not follow the procedure.

Validating convergence

After the commit lands and self-heal is resumed, the responder validates convergence. The validation has three steps:

• Diff is empty. argocd app diff $APP_NAME (Argo CD) or flux diff kustomization $KS_NAME (Flux) shows no changes between the cluster and the rendered manifests.
• Sync status is Synced. argocd app get $APP_NAME reports Sync Status: Synced and Health Status: Healthy.
• Self-heal is on. argocd app get $APP_NAME reports the sync policy and self-heal flag have been restored to their pre-incident values.

A validation that finds lingering drift is the signal that the reconciliation is incomplete. The responder either commits the remaining diff (if it represents an undeclared change) or reverts the cluster (if the commit was wrong).

flowchart LR
    A["Commit landed"] --> B["Resume self-heal"]
    B --> C["Wait one reconciliation tick"]
    C --> D{"Diff empty?"}
    D -- yes --> E["Synced + Healthy"]
    D -- no --> F["Reconcile remaining diff"]
    F --> C

The cost of leaving the edit uncommitted

The cost of skipping the reconciliation commit is paid in three ways:

• Silent drift. The cluster has a value that Git does not. The Application is OutOfSync. The drift alert fires. The next on-call engineer pages the team for a non-incident drift that the team has already paid to create.
• Audit gap. The incident record has a Kubernetes audit log entry for the edit; the GitOps audit trail has no commit hash. Six months later, the security team cannot answer “which commit introduced this value” - the answer is “no commit, a manual edit during an incident that was never reconciled”.
• Replay on next sync. A future operator runs argocd app sync $APP_NAME or the controller auto-syncs on a new commit. The next sync reverts the cluster to Git’s view, the emergency edit is lost, and the next incident has the same trigger.

The three costs compound. A team that skips the reconciliation commit once pays the silent-drift cost; pays it again on the next sync; pays it a third time when the audit gap is discovered. The discipline is to commit before resume.

Production discipline

1. Commit before resume. The order is the difference between a clean repair and a reversion loop.
2. Reference the incident ID in the commit message. The Refs: trailer is the audit trail.
3. Validate convergence with three reads. Diff, sync status, self-heal flag.
4. Treat lingering drift as an incomplete reconciliation. The repair is not done until diff is empty.

Cross-course references

• This course, Part LXXV-04 (Emergency drift) - the drift category the reconciliation closes.
• This course, Part LXXXV-06 (Post-rollback investigation)
  • the audit-trail discipline that includes the reconciliation commit hash.
• Conventional Commits - the message format that makes reconciliation commits greppable in the history.

Quiz