The recovery mindset — stop, inspect, locate, recover

Every Git accident is the same accident at the moment it happens: an engineer believes the history has changed and the work is gone. The belief is wrong. The commits are almost always still in the object store, still reachable through the reflog, and still recoverable. What determines whether the recovery succeeds is not the severity of the accident but the first command typed after the accident. A git reflog restores the situation. A panic-driven git reset --hard followed by git push --force followed by git gc --prune=now destroys it permanently.

The recovery mindset is the discipline that prevents the second path from being taken.

The four-step discipline

The recovery path is the same path regardless of the accident — deleted branch, dropped stash, amended commit, bad rebase, or mistaken force-push. The shape is constant; only the target changes.

flowchart LR
    A["Accident discovered"] --> B["Step 1: STOP"]
    B --> C["Step 2: INSPECT"]
    C --> D["Step 3: LOCATE"]
    D --> E["Step 4: RECOVER"]
    E --> F["Verify"]
    F --> G["Document the runbook entry"]

Step 1 — Stop. Hands off the keyboard. Do not type the next command until you have read the next paragraph. The commands that follow accidents are the most expensive commands in the discipline, because they are the ones a panicking engineer types without reading. The recovery path is always through the reflog, and the reflog is always the next place to look. There is no recovery path that does not begin with git reflog.

Step 2 — Inspect. Run git reflog and read the output. The output is the clone’s history of HEAD updates, in reverse chronological order, with the OID, the action, and the message that produced each update. The most recent entry (HEAD@{0}) is the action that caused the accident; the entry immediately before it (HEAD@{1}) is almost always the recovery target.

Step 3 — Locate. Identify the OID, the branch, the working tree state, or the stash entry that needs to be recovered. Copy the OID to a durable location — a ticket, a chat message, a PR comment — before step 4. The OID alone is the recovery target; once the reflog has been read, the OID is the only piece of information that matters.

Step 4 — Recover. Apply the recipe for the specific accident: git branch <name> <oid> for a deleted branch, git stash apply <stash@{n}> for a dropped stash, git cherry-pick <oid> for an amended commit, git reset --hard <oid> for a hard reset. The recovery command is always non-destructive on the orphan objects; it makes them reachable again from a named ref.

Why panic wastes time

The panic response is the deterministic response of an engineer who believes the work is gone. The visible actions are aggressive: another reset, a force-push to “fix” the remote, a git gc --prune=now to “clean up”. Each of those actions is locally reasonable and globally catastrophic. The sequence below is the panic sequence; it has been observed on production infrastructure repositories and has, in at least three documented cases, cost the team a day’s work.

# The panic sequence — DO NOT TYPE
git reset --hard HEAD@{1}     # guess at the recovery
git push --force origin main  # propagate the guess
git gc --prune=now            # "clean up"
# at this point the recovery is gone on every clone that fetched

# The four-step sequence — type this
git reflog                    # read the clone's history
# identify the OID at HEAD@{1}
PREVIOUS_OID=$(git reflog -1 HEAD@{1} | awk '{print $1}')
git branch recovery $PREVIOUS_OID  # make the orphan reachable
git log --oneline recovery     # verify before any push
git push origin recovery      # propagate as a new branch

The four-step sequence takes longer. The first command is the same — git reflog — but the second, third, and fourth commands are typed after the engineer has read the reflog, identified the OID, and chosen a target. The panic sequence is typed by reflex; the four-step sequence is typed by reading.

The blast radius of the next command

The recovery discipline is graded by the blast radius of the next command typed. The first command typed after the accident is the highest-leverage command in the entire recovery; the second command is the second-highest; and so on.

flowchart LR
    A["accident"] --> B["next command"]
    B --> C{"blast radius"}
    C -->|"local"| D["reflog: recoverable"]
    C -->|"shared"| E["remote: recoverable per clone"]
    C -->|"global"| F["gc --prune: not recoverable"]

• Local blast radius: git reflog, git fsck --unreachable, git show <oid>, git branch <name> <oid>. These commands read state; they do not modify it. The recovery is invariant after they have been run.
• Shared blast radius: git push --force (without --force-with-lease), git reset --hard on a branch a collaborator has also fetched. The recovery is harder because the remote state has changed and the collaborator’s clones have diverged.
• Global blast radius: git gc --prune=now on a clone that is the only remaining holder of an orphan; a git push --force to a branch that other clones have already pruned from. Anything written here is unrecoverable across the team.

The discipline is to keep the next command in the local category until the OID has been located and the recovery target has been chosen. The cost of the discipline is the cost of reading the reflog before typing the next command. The cost of not following the discipline is the cost of irreversible loss.

The four-step discipline as a runbook

The discipline is most useful when it is also a runbook entry the entire team can paste into a terminal. The condensed form:

# Step 1: STOP. Do not type the next command.

# Step 2: INSPECT.
git reflog -20
# read the output; identify HEAD@{0} (the accident) and HEAD@{1}
# (the recovery target)

# Step 3: LOCATE.
RECOVERY_OID=$(git reflog -1 HEAD@{1} | awk '{print $1}')
echo "Recovery OID: $RECOVERY_OID"
git show --stat $RECOVERY_OID
# verify the OID is the right one before any further action

# Step 4: RECOVER.
# pick the right recipe for the accident:
#   git branch <name> $RECOVERY_OID      # deleted branch
#   git stash apply <stash@{n}>         # dropped stash
#   git cherry-pick $RECOVERY_OID        # amended commit
#   git reset --hard $RECOVERY_OID       # hard reset
#   git push --force-with-lease ...      # propagate

The runbook is the discipline. The discipline is the difference between a recoverable accident and a two-day incident.

Production discipline

1. The first command after an accident is git reflog. Never before. The exceptions are emergencies in which the working tree is being destroyed by something other than Git (filesystem, disk, container), and even then the reflog should be read before any history-rewriting command.
2. Never run git gc --prune=now immediately after an accident. The prune may remove the orphan commits before the reflog has been read. The recovery window is closed permanently by an early prune.
3. Always copy the OID to a durable location before recovery. The OID alone is the recovery target. The OID 8a3f9d2 is the same commit in every clone, in every backup, and in every CI artifact that has ever referenced it. The reflog is one of many places the OID lives; the OID itself outlives the reflog.
4. Document the recovery in the team’s runbook. Every accident that required a recovery is a candidate for a new runbook entry. The discipline is institutional, not personal.

Cross-course references

• Git, CI/CD & GitOps — Part XVII (Reflog) — the location and the scope of the reflog; the prerequisite for this lesson.
• Git, CI/CD & GitOps — Part XV (Reset) — the most common accident this discipline recovers from. The four-step recipe is the recovery path for git reset --hard, and the discipline is the same.
• GitOps with Argo CD — Part VI (MergeStrategies) — the GitOps controller’s clone is configured with gc.reflogExpire never for the same reason: the recovery window in a GitOps controller must be effectively infinite.

Quiz