Troubleshooting mindset — the discipline that prevents incidents from becoming outages

Every firewall operator will debug a production incident. Most will debug one at 03:00 with a manager on the line and a customer waiting. The incident is the same shape every time — something that was working is no longer working, and the clock is running. What separates the operator who fixes the incident from the operator who makes it worse is not technical depth. It is the discipline they bring to the investigation.

This lesson is about that discipline. The technical tools (pfctl, tcpdump, the routing table, packet capture, the log search) come in the next five lessons. None of them help if the operator runs them in the wrong order, jumps to a fix before establishing the symptom, or applies a change they cannot reverse.

The first rule: no change without evidence

A change made on a hypothesis is a guess. A guess in production is a future incident. The discipline that prevents that is “no change without evidence”:

• Do not edit a rule until you have evidence that the rule is the cause.
• Do not add a route until you have evidence that the routing table is wrong.
• Do not reboot until you have evidence that the running system is in a state that requires a reboot.
• Do not restore a configuration until you have evidence that the saved configuration is correct and the running one is not.

The strongest temptation during an incident is to apply the fix the operator remembers from the last similar incident. The trap is that the last incident was similar, not the same. A change made from memory on a different incident is exactly the kind of guess that produces a follow-up incident.

Slow is fast

The instinct during an incident is to act immediately. The instinct is wrong. The fastest fix is almost always the slowest investigation:

1. Confirm the symptom (one minute).
2. Form a hypothesis (one minute).
3. Collect the evidence for that hypothesis (two to five minutes).
4. If the evidence supports the hypothesis, apply a reversible fix (one minute).
5. Verify the fix (one minute).
6. Document the incident (five minutes after the recovery).

Total: ten to fifteen minutes. The operator who tries to skip steps to “save time” usually spends thirty minutes undoing the wrong fix before they get back to step one.

The discipline is to make each step small and reversible. “Confirm the symptom” is one command. “Form a hypothesis” is a sentence. “Collect the evidence” is two or three commands. “Apply a reversible fix” is one change that can be reverted in the same step. The cost of discipline is a few extra minutes. The cost of skipping it is a second incident.

The symptom is not the cause

The user reports “the Internet is down”. The firewall operator’s first job is to distinguish:

• The user-observed symptom. What the user sees. Often imprecise.
• The firewall-observed symptom. What pfctl -s state, the log, and the routing table actually show.
• The actual cause. What produces the firewall-observed symptom.

Three different layers, three different things to investigate. “The Internet is down” might mean “DNS is failing” or “HTTPS is timing out” or “the gateway IP is unreachable”. Each maps to a different subsystem, a different evidence-gathering command, and a different fix.

The trap is to accept the user-observed symptom as the cause. “The Internet is down” becomes “the WAN interface must be down” and the operator spends twenty minutes on the WAN interface. The actual cause might be that a single rule blocked HTTPS, and the user uses HTTPS for everything, so “the Internet” looks “down” to them.

Cognitive traps under time pressure

Three traps derail firewall incidents:

1. Anchoring. The first hypothesis sticks. The operator reads evidence through the lens of the first hypothesis and stops seeing the alternatives. The cure is to write the first hypothesis down and explicitly enumerate two alternatives before collecting evidence.
2. Confirmation bias. The operator finds evidence that supports the hypothesis and stops collecting evidence that would refute it. The cure is to ask “what would prove me wrong?” before reading each piece of evidence.
3. Action bias. The operator feels pressure to act and applies a change they cannot justify with evidence. The cure is to ask “what is the cost of waiting five more minutes?” If the cost is low, the discipline is to wait.

These traps are not character flaws. They are how human cognition works under pressure. The discipline is to recognise them and to put small frictions in place that interrupt them — a written hypothesis, a checklist, a peer review of the proposed change.

The diagnostic loop

The diagnostic loop has four steps and runs many times during an incident:

1. Observe. A command produces output. The output is a piece of evidence.
2. Orient. The evidence matches one hypothesis and contradicts another. The operator updates the running list of hypotheses.
3. Decide. The next command is the one that best distinguishes the remaining hypotheses.
4. Act. Run the command.

Then back to step one. The loop runs as many times as needed. Each iteration should produce a small piece of evidence that narrows the search. An iteration that does not narrow the search is wasted time; the operator should pick a different command.

A note on escalation

The operator who escalates a properly-investigated incident is doing their job. The operator who escalates an uninvestigated incident is delegating work they have not done. The escalation handoff should include:

• The user-observed symptom (in the user’s words).
• The firewall-observed symptom (commands and outputs).
• The hypotheses considered and the evidence that refuted each one.
• The remaining hypotheses and the next command to run.
• What has been changed so far, and the rollback for each change.

A senior engineer receiving that handoff can continue the investigation without retracing the operator’s steps. The senior engineer receiving “Internet down, please fix” has to start from zero.

Summary

• No change without evidence: a change made on a hypothesis is a guess, and a guess in production is a future incident.
• Slow is fast: the ten-minute disciplined investigation is faster than the thirty-minute recovery from the wrong fix.
• The symptom is not the cause: distinguish what the user sees, what the firewall shows, and what produces it.
• Recognise the cognitive traps: anchoring, confirmation bias, action bias. Put frictions in place to interrupt them.
• Document the diagnostic loop so the next on-call operator can continue the trail.