Skip to main content
RunBook Academy

Git, CI/CD & GitOpsXVIII · Git RecoveryRecovery

The recovery decision tree — a flowchart for any "I lost X" scenario

Advanced⏱ ~26 mingit

What you'll learn

  • Apply the recovery decision tree to any Git accident (deleted branch, dropped stash, amended commit, hard reset, bad rebase, force-pushed ref)
  • Identify the first step of the recovery path based on the accident type and the recovery vector (reflog, fsck, remote ref, backup)
  • Recognise the four cross-cutting recovery paths (other clones, fsck, remote refs, backups) and the order of preference
  • Distinguish recoverable accidents (reflog-based) from unrecoverable accidents (prune-driven) and the boundary between them

Prerequisites

Verified against Git 2.55.x teaching target; 2.40+ minimum · GitHub Actions continuous service; Aug 2026 documentation baseline · Argo CD v3.5.x teaching target; v3.0+ minimum · Flux v2.9.x · Sigstore Cosign v3.1.x · SLSA v1.2 · OCI Distribution Specification v1.1 · Git LFS v3.7.1 · Kubernetes (cross-course target) 1.36.x

Not yet marked complete on this device.

Every Git accident is a member of one of four types and recovers via one of four paths. The decision tree is the matrix of accident type and recovery path. The first command is always git reflog; the second command is the recipe for the accident type. The cross-cutting recovery paths are the ones that work regardless of accident type: other clones, fsck against the object store, remote refs, and backups.

The decision tree is the last lesson in this part because it unifies the previous five. The mental model that the tree imposes is: what was lost and where it can be found. The answer to the first question is the accident type; the answer to the second question is the recovery path.

The top-level decision tree

flowchart TD
    A["Accident discovered"] --> B{"Step 1: STOP"}
    B --> C["git reflog"]
    C --> D{"Reflog has the entry?"}
    D -->|"yes"| E["Recover via reflog recipe"]
    D -->|"no"| F{"Other clone has the entry?"}
    F -->|"yes"| G["Fetch from that clone"]
    F -->|"no"| H{"fsck --unreachable finds the OID?"}
    H -->|"yes"| I["git branch name OID"]
    H -->|"no"| J{"Remote ref has the OID?"}
    J -->|"yes"| K["git fetch refs/pull/<n>/head"]
    J -->|"no"| L{"Backup has the bytes?"}
    L -->|"yes"| M["Restore from backup"]
    L -->|"no"| N["Reconstruct from history and memory"]

The decision tree has eight nodes. The first answer is the accident type (which determines the reflog recipe). The second answer is the recovery path (reflog, other clone, fsck, remote ref, backup). The terminal state is the recovery or the partial recovery.

The accident type → reflog recipe

The four accident types map to four reflog recipes:

flowchart LR
    A["Accident type"] --> B["reflog recipe"]
    A1["Deleted branch"] --> B1["git branch name OID"]
    A2["Dropped stash"] --> B2["git stash apply or branch from OID"]
    A3["Amended commit"] --> B3["git cherry-pick or branch from OID"]
    A4["Hard reset or bad rebase"] --> B4["git reset --hard OID"]
    A1 --> B1
    A2 --> B2
    A3 --> B3
    A4 --> B4
Accident typeReflog entryRecovery command
Deleted branchHEAD@{1} or the entry before the checkoutgit branch <name> <oid>
Dropped stashstash@{0} in git reflog show stashgit stash apply <oid> or git stash branch <name> <oid>
Amended commitHEAD@{1} (the entry before the amend)git cherry-pick <oid> or git branch <name> <oid>
Hard reset or bad rebaseHEAD@{1} (the entry before the reset/rebase)git reset --hard <oid>

The first three columns are the inputs; the fourth column is the recipe. The first command is git reflog (or git reflog show stash for the dropped stash); the second command is the recipe.

The recovery path → command

The five recovery paths map to five commands:

flowchart LR
    A["Recovery path"] --> B["command"]
    A1["Local reflog"] --> B1["git reflog; git branch / stash apply / cherry-pick / reset --hard"]
    A2["Other clone"] --> B2["git fetch from another clone; git reflog on that clone"]
    A3["fsck"] --> B3["git fsck --unreachable --no-reflogs; git branch name OID"]
    A4["Remote ref"] --> B4["git fetch origin refs/pull/<n>/head:pr-n"]
    A5["Backup"] --> B5["Restore from CI artifact / config backup"]
Recovery pathOrder of preferenceCommand
Local reflog1git reflog → recipe
Other clone2git fetch <clone> → reflog on that clone
fsck3git fsck --unreachable --no-reflogs
Remote ref4git fetch origin refs/pull/<n>/head
Backup5Restore from CI artifact or config backup

The order of preference is constant: the local reflog is fastest and most local; the backup is slowest and most expensive. The intermediate paths (other clone, fsck, remote ref) are progressively more remote and more work.

The decision tree on a single accident

A worked example: an engineer has just run git reset --hard HEAD~5 on feature/iam-rotation. The branch is shared. The decision tree is:

  1. STOP. Hands off the keyboard.
  2. INSPECT. git reflog -20. The output shows the pre-reset OID at HEAD@{1}.
  3. LOCATE. Copy the OID to a PR comment.
  4. RECOVER. git reset --hard HEAD@{1}git push --force-with-lease origin feature/iam-rotation.

The recipe is the four-step discipline plus the accident-specific recovery command. The decision tree is the mapping from accident to recipe; the discipline is the sequence of steps.

The case where the reflog has expired

Past the 90-day reflog window, the recovery decision tree continues:

flowchart TD
    A["Reflog expired"] --> B["Other clone has the entry?"]
    B -->|"yes"| C["git fetch ssh://ops@git-mirror/srv/git/repo.git"]
    B -->|"no"| D["fsck --unreachable on local clone"]
    D -->|"finds the OID"| E["git branch name OID"]
    D -->|"no hit"| F["Remote ref has the OID?"]
    F -->|"yes"| G["git fetch origin refs/pull/<n>/head"]
    F -->|"no"| H["Backup has the bytes?"]
    H -->|"yes"| I["Restore from CI artifact"]
    H -->|"no"| J["Partial recovery: reconstruct from history"]

The expanded tree is the second half of the recovery. The first half is the reflog-based recovery; the second half is the post-reflog recovery. The boundary is the 90-day default or the next git gc --prune=now.

The cross-cutting recovery paths

The four cross-cutting recovery paths are the ones that work regardless of accident type. The decision tree narrows them to the right path for the situation:

  1. Other clone. Any clone that fetched the affected ref before the accident has the OID in its reflog. The recovery is to fetch the OID from that clone’s reflog. This is the most reliable post-reflog path because clones have independent reflogs and the union of the reflogs is the union of the recovery windows.
  2. fsck. git fsck --unreachable --no-reflogs on a clone that has the objects but no reflog entry lists the orphan commits. Any commit reported is recoverable. The path is the forensic scan.
  3. Remote ref. Most forges keep refs/pull/<n>/head for the lifetime of the PR. The recovery is to fetch the PR ref and create a branch from it. The path is durable for the lifetime of the PR.
  4. Backup. The bytes may be in a CI artifact store, a configuration backup, a container image, or a developer’s local backup. The recovery is to restore the bytes from the backup. The path is the slowest and the most expensive.

The boundary between recoverable and unrecoverable

The decision tree has a hard boundary: the recovery is mechanical when the OID is reachable via the reflog or another clone’s reflog; the recovery is forensic when the OID is reachable via fsck or remote ref; the recovery is manual when the OID is reachable via a backup; the recovery is impossible when nothing has the OID.

The boundary is determined by:

  1. Reflog expiry. The 90-day reachable default and the 30-day unreachable default. Past the unreachable default, the OID is a prune candidate.
  2. git gc --prune=now. The prune is the act that removes the orphan objects. The first prune past the reflog window is the point of no return.
  3. Object store scan. git fsck --unreachable finds the OID if the objects are still in the store. The path closes when the objects are pruned.
  4. Backup retention. The CI artifact store’s retention window, the configuration backup’s retention window, the container image’s retention. The path closes when the backup is rotated.

The boundary is not a single moment; it is a sequence of moments. The recovery window closes progressively as the backup retention expires, the reflog expires, the objects are pruned, and the remote refs are purged. The earlier in the sequence the recovery is performed, the easier it is.

Production discipline

  1. The first command after any accident is git reflog. Never before, and regardless of the accident type. The reflog is the first path; the decision tree is the mapping from accident type to recovery recipe.
  2. Always copy the OID to a durable location before the recovery. The OID alone is the recovery target. The OID outlives the reflog; the OID is the same in every clone, every backup, every CI artifact.
  3. Document the recovery in the runbook. Every accident that required a recovery is a candidate for a new runbook entry. The decision tree is the team’s institutional knowledge; the tree grows with every accident.
  4. Rehearse the four accident types on a regular cadence. A rehearsal is a simulated accident on a throwaway repository. The rehearsal reveals gaps in the runbook and trains the team to follow the decision tree without thinking.

Cross-course references

  • Git, CI/CD & GitOps — Part XVII (Reflog) — the prerequisite for this lesson; the reflog’s location and scope.
  • Git, CI/CD & GitOps — Part XVIII (GitRecovery) — the previous five lessons in this part; the recipes for the four accident types.
  • GitOps with Argo CD — Part VI (MergeStrategies) — the GitOps controller’s recovery from a bad sync uses the same decision tree against the controller’s clone, which is configured with gc.reflogExpire never so the recovery window is effectively infinite.

Quiz

Knowledge check · 4 questions

  1. Q1. An engineer has just discovered that a teammate's rebase has force-pushed away a commit on a shared branch. The local reflog has expired. What is the next recovery path in the decision tree?

  2. Q2. The recovery decision tree has a hard boundary: the recovery is mechanical when the OID is reachable via the reflog, forensic when reachable via fsck or remote ref, manual when reachable via a backup, and impossible when nothing has the OID.

  3. Q3. Name the four accident types and the four recovery paths, and describe the order of preference for the recovery paths.

  4. Q4. Walk the decision tree for an engineer who has just discovered that a critical commit on a shared branch has been lost during a teammate's repository cleanup.

    An engineer on `feature/iam-rotation` discovers that a commit made three months ago is no longer reachable from the branch. The teammate's cleanup included a `git gc --prune=now` on the local clone and a rebase of the branch; the cleanup was performed eight weeks ago. The local clone's reflog has expired. The engineer needs to recover the commit to update the IAM rotation module.

Passing score: 75%. Answers are checked in this browser.