Skip to main content
RunBook Academy

← All runbooks in Git, CI/CD & GitOps

high riskservice affecting~45 min

Runbook: Resolve an Infrastructure Merge Conflict

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

  • · Capture which branches are conflicting: git status shows both modified: for the affected files; git rev-parse --abbrev-ref HEAD is the branch you are merging into
  • · Identify the merge tool: git config --get merge.tool (default vimdiff); set git config --global merge.tool nvimdiff3 or similar before continuing if unset
  • · For Terraform: confirm state lock is healthy before the conflict blocks other agents: terraform force-unlock <LOCK_ID> is wrong here — instead, terraform plan -lock=false -refresh-only to confirm state has not drifted underneath you
  • · For Ansible: confirm the inventory is current: ansible-inventory -i inventories/prod/hosts.yml --graph
  • · For Kubernetes manifests: confirm the cluster context matches the branch (prod overlay vs staging overlay): kubectl config current-context
  • · Identify the file format: YAML/JSON for K8s and Ansible, HCL for Terraform — conflict markers look different, the resolution strategy differs accordingly

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

  1. 1List the conflicting files: git diff --name-only --diff-filter=U and git status --porcelain | grep "^UU\\|^AA\\|^DD"
  2. 2For each conflicting file, identify which side is "ours" (current branch) and "theirs" (the branch being merged): git log --oneline HEAD -- <file> -n 5 and git log --oneline MERGE_HEAD -- <file> -n 5
  3. 3Open the conflict in a three-way merge tool: git mergetool -- <file> (configured tool) or use a CLI alternative for HCL/YAML
  4. 4Resolve by re-running the parser, not by reading conflict markers: for YAML, yq --version && yq eval-all ". as $item ireduce ({}; . *+ $item)" <file> > /tmp/resolved.yaml produces a merged object only if neither side changed the same key
  5. 5For Terraform HCL: do not hand-edit HCL if the conflict is structural (resource "aws_security_group" "x" { ... } added on both sides with different arguments). Re-author the block once, then run terraform fmt
  6. 6For Ansible YAML: resolve the conflict, then run ansible-playbook --syntax-check -i inventories/prod/hosts.yml <playbook> to confirm the YAML is still valid; the parser will reject what your eyes accept
  7. 7For Kubernetes manifests: validate with kubeconform -strict -summary <file> (uses the upstream CRDs) and kubectl apply --dry-run=server -f <file> against the same cluster context the branch deploys to
  8. 8Stage the resolution: git add <file> for each resolved file. Do not use git add -A — you want to review what is staged before committing
  9. 9Run the semantic check for the affected IaC tool (terraform plan, ansible-playbook --check, kubectl diff -f -R) to confirm the resolved file produces the expected plan: zero changes for non-functional merges (whitespace, comment), an additive plan for one-sided changes
  10. 10For Terraform: confirm state has not drifted by running terraform plan -lock=false and comparing the resource addresses to the expected merge — a changed address means the resolution is wrong
  11. 11For Ansible: confirm the resulting variable scope is what you expect by running ansible-inventory -i inventories/prod/hosts.yml --list and diffing against the pre-conflict version
  12. 12For Kubernetes: confirm the resolved manifest applies cleanly against a staging cluster with kubectl apply --dry-run=server -f <file> before committing to the conflict-resolving branch
  13. 13Commit the resolution with a message that names both sides and the chosen strategy: git commit --no-edit -m "merge $BRANCH into $TARGET: resolve <files> (kept <side>, applied <change>)"
  14. 14Push only after the CI pipeline validates the resolved branch: git push origin "$TARGET_BRANCH"

4 · Verification

Confirm the procedure actually fixed the problem.

  • ✓No conflict markers remain in the working tree: git grep -nE "^(<{7}|={7}|>{7})( |$)" -- &lt;file&gt; returns empty for every previously-conflicting file
  • ✓For Terraform: terraform validate returns success and terraform plan produces no unexpected changes outside the merged-in additions
  • ✓For Ansible: ansible-playbook --syntax-check and ansible-lint &lt;playbook&gt; both pass against the resolved file
  • ✓For Kubernetes: kubeconform -strict -summary &lt;file&gt; reports 0 failures and kubectl apply --dry-run=server -f &lt;file&gt; reports no errors
  • ✓git diff HEAD~1 --stat shows only the expected additions and no unexpected modifications
  • ✓The merge commit has both parents: git log -1 --format="%P" shows two SHAs (the merge of both sides)

5 · Rollback

If verification fails, undo the procedure in reverse order.

  • ↶If the resolution produced a syntactically invalid file, abort the merge: git merge --abort (only works if no resolution has been committed yet)
  • ↶If the resolution was committed and pushed and CI rejects it, push the revert: git revert -m 1 &lt;merge-commit-sha&gt; records the inverse of the merge as a new commit on the target branch
  • ↶If Terraform plan shows drift or unexpected changes after the resolution, the conflict resolution lost context — re-merge with --strategy-option=theirs for the lost file is wrong; instead, reset to the pre-merge state and resolve again with a three-way tool that shows both sides
  • ↶If Ansible syntax-check fails, the YAML is invalid; undo the resolution with git checkout --merge &lt;file&gt; and resolve with a YAML-aware tool (yq, ruamel.yaml) instead of a text editor
  • ↶If Kubernetes manifests apply cleanly but produce unexpected drift, the conflict resolution chose the wrong overlay; revert and re-merge with the right base branch

6 · Escalation

When the runbook isn't enough, contact:

  • · The conflict is in a CRD that no validator knows about (custom resource, no schema published): escalate to the CRD owner before merging, the schema check cannot catch the mistake
  • · Terraform state lock cannot be acquired because another agent holds it: wait for the lock to expire (default 30 minutes) or coordinate with the holder, do not force-unlock
  • · The merge is between two long-lived branches with conflicts in many files: do not try to resolve in one pass, merge one commit at a time with git merge -s recursive -X theirs for the lower-priority side, then resolve the high-priority side intentionally
  • · A dependency (provider, helm chart, kustomize base) was updated on both sides with incompatible versions: escalate to platform ownership, this is a manifest-library bug not a merge bug

A merge conflict in an infrastructure repository is not a text problem. It is a state problem. The two branches disagreed about what the infrastructure looks like; the resolution is the new contract. Reading the conflict markers and picking “ours” or “theirs” without re-running the tool’s parser is how broken clusters ship — the YAML parses, the Terraform plan passes syntax check, and kubectl apply accepts the manifest, but the resulting state is not what either branch intended.

The runbook is: read the markers to find the disagreement, resolve with a parser-aware tool, then validate with the tool that interprets the file (Terraform plan, Ansible syntax check, kubeconform + kubectl dry-run).

1. Identify the conflicting files and which side is which

Read-only / Safe
$ git diff --name-only --diff-filter=U
echo '--- our side (HEAD) ---'
git log --oneline HEAD -- "$(git diff --name-only --diff-filter=U | head -1)" -n 5
echo '--- their side (MERGE_HEAD) ---'
git log --oneline MERGE_HEAD -- "$(git diff --name-only --diff-filter=U | head -1)" -n 5
echo '--- conflict marker locations ---'
git grep -nE '^(<{7}|={7}|>{7})( |$)' -- '*.tf' '*.yml' '*.yaml' '*.json' || echo no-markers-in-fmt

For a single conflict the answer is usually obvious: one side adds a resource block, the other side edits an unrelated resource. For a structural conflict (the same resource edited differently), the resolution requires re-authoring the block, not picking a side.

2. Resolve with a parser-aware tool

Read-only / Safe
$ # YAML — use yq for additive merges
yq eval-all '. as $item ireduce ({}; . *+ $item)' main.tfvars prod.tfvars > /tmp/merged.tfvars
yq eval-all '. as $item ireduce ({}; . *+ $item)' values-staging.yaml values-prod.yaml > /tmp/merged-values.yaml

# JSON — use jq for additive merges
jq -s 'reduce .[] as $item ({}; . * $item)' side-a.json side-b.json > /tmp/merged.json

# HCL — re-author once, then terraform fmt
terraform fmt main.tf

yq eval-all ... ireduce merges documents additively when neither side touches the same key. If both sides edited the same key (e.g. replicas in a Deployment), the tool picks the second argument’s value silently — that is the case you must resolve by hand.

3. Validate with the tool that owns the file

Read-only / Safe
$ # Terraform
terraform init -backend=false
terraform validate
terraform plan -lock=false -refresh-only | tee /tmp/refresh-plan.txt
terraform plan -lock=false -out=/tmp/plan.tfplan
terraform show -no-color /tmp/plan.tfplan | head -80

# Ansible
ansible-playbook --syntax-check -i inventories/prod/hosts.yml site.yml
ansible-lint site.yml
ansible-inventory -i inventories/prod/hosts.yml --list > /tmp/post-merge-inv.json

# Kubernetes
kubeconform -strict -summary overlays/prod/kustomization.yaml
kubectl kustomize overlays/prod/ > /tmp/manifests.yaml
kubeconform -strict -summary /tmp/manifests.yaml
kubectl apply --dry-run=server -k overlays/prod/

If terraform plan shows resource drift that is not in the merge diff, the resolution lost context — the HCL parses, but the resource address changed. Stop and re-merge the block, do not apply the plan. For Ansible, syntax-check catches structural mistakes; ansible-lint catches deprecated modules and unsafe loops. For K8s, kubeconform validates against upstream CRDs and kubectl apply --dry-run=server catches admission-policy rejections that static validation misses.

4. Stage, commit, and verify

Read-only / Safe
$ git status --porcelain
git diff --staged --stat
git add REPLACE_WITH_FILE  # for each resolved file, individually
echo '--- no markers remain ---'
git grep -nE '^(<{7}|={7}|>{7})( |$)' || echo clean
git commit --no-edit -m "merge $BRANCH into $TARGET: resolve REPLACE_WITH_FILES (kept REPLACE_WITH_SIDE, applied REPLACE_WITH_CHANGE)"
git log -1 --format='%H %P %s'
echo '--- the commit has two parents ---'
git log -1 --format='%P' | wc -w

The commit message must name both sides and the resolution strategy. “merge origin/main” is not enough; the next agent who reads the merge needs to know why neither --strategy-option=ours nor --theirs was used.

Verification

git grep -nE "^(<{7}|={7}|>{7})( |$)" returns empty across the working tree. terraform validate (or ansible-playbook --syntax-check, or kubeconform -strict) reports success. terraform plan (or ansible-playbook --check, or kubectl diff -k overlays/prod/) shows only the expected changes from the merge — no unintended drift, no unintended resource replacements. The merge commit has two parents. If CI is configured, the resolved branch passes the plan/apply pipeline before the merge is merged into the protected branch.

Rollback

If the resolution was syntactically valid but semantically wrong (plan shows drift the merge did not intend), the merge has not been pushed yet: git merge --abort returns to the pre-merge state and the resolution can be retried. If the merge was pushed, push the inverse with git revert -m 1 &lt;merge-sha&gt; — that records the inverse as a new commit on the target branch and leaves the original merge in the audit trail. Never git reset --hard and git push --force on a merge that already reached the protected branch; revert is the correct tool.

References

  1. git-mergetool(1)
  2. git-merge(1) — including merge strategies and conflict style
  3. gitattributes(5) — merge drivers for JSON/YAML
  4. Terraform — State Locking
  5. kubeconform — Kubernetes manifest validation
  6. ansible-lint — playbook linting