Troubleshooting anti-patterns — reboot before evidence, blame the firewall, cargo-cult configs

The troubleshooting anti-patterns are the operator behaviours that consistently make incidents worse. The operator who reboots before collecting evidence destroys the state that would have revealed the cause. The operator who blames the firewall without verification changes a configuration that may not be the cause. The operator who copies a configuration from the internet without understanding introduces a fragile dependency on a workaround. The discipline of professional troubleshooting is to recognise and avoid these anti-patterns.

This lesson is the sixth and final lesson in Part LII: the common troubleshooting anti-patterns, the production failure modes they cause, and the discipline that prevents them.

Anti-pattern 1: Reboot before evidence

The operator reboots the router before collecting evidence. The reboot destroys the state that would have revealed the cause: the BGP session state, the routing table at the time of the failure, the firewall counters, the conntrack table, the logs. After the reboot, the operator cannot determine what caused the failure.

flowchart LR
  S["Symptom reported"] --> R["REBOOT<br/>(no evidence)"]
  R --> W["State destroyed"]
  W --> X["Cannot diagnose"]
  X --> Y["Issue recurs"]

The discipline: never reboot without evidence. If the symptom is severe enough to require a reboot, collect the evidence first (logs, counters, state) and save it to a file or external system. Then reboot.

Anti-pattern 2: Blame the firewall

The operator assumes the firewall is the cause without verifying. The operator adds an accept any any rule to “fix” the issue. The rule may open the firewall to traffic it should not allow; the actual cause (which may be BGP, MTU, or routing) is not addressed.

flowchart LR
  S["Symptom reported"] --> B["BLAME FIREWALL<br/>(no verification)"]
  B --> A["Add accept any any"]
  A --> C["Security regression"]
  A --> X["Real cause not addressed"]

The discipline: verify the firewall is the cause before changing it. The verification:

show firewall name <name>
# Check the rule counters; is the rule matching?
conntrack -L | grep <flow>
# Is the conntrack entry present?
nft list ruleset | grep <flow>
# Is the nftables ruleset allowing the flow?

If the firewall is dropping the flow, the rule counters show it. If the firewall is allowing the flow, the issue is elsewhere. The operator who adds an accept any any without verification may open the firewall and miss the real cause.

Anti-pattern 3: Cargo-cult configurations

The operator copies a configuration from the internet (a forum post, a StackOverflow answer, a vendor’s example) without understanding it. The configuration may work for the poster’s environment but not for the operator’s environment. The operator is now depending on a workaround they do not understand.

flowchart LR
  S["Symptom reported"] --> I["Search internet"]
  I --> C["Copy config"]
  C --> W["Workaround applied"]
  W --> D["Dependency on workaround<br/>(operator does not understand)"]
  D --> R["Workaround breaks in future"]

The discipline: understand every configuration change. If the operator cannot explain what the configuration does and why, the configuration is not ready for production. The operator who copies a configuration without understanding is gambling with the network.

The canonical cargo-cult configurations:

• set system sysctl parameter net.ipv4.ip_forward value 1 (already enabled by default on VyOS).
• set firewall ipv4 name ANY default-action accept (defeats the firewall).
• set protocols bgp <asn> neighbor <peer> soft-reconfiguration inbound (doubles memory usage; rarely needed).
• set system conntrack timeout tcp-established 7200 (extending the timeout may cause conntrack table exhaustion).

The operator who applies these without understanding may fix the immediate symptom but introduce a future failure.

Anti-pattern 4: Escalation without a clear symptom

The operator escalates the incident to a senior engineer without defining the symptom or scoping the impact. The senior engineer receives a vague report (“the network is broken”) and cannot investigate efficiently. Time is wasted on clarification that the operator should have done.

flowchart LR
  S["Symptom reported"] --> E["Escalate<br/>(no symptom defined)"]
  E --> Q["Senior engineer asks for symptom"]
  Q --> C["Operator clarifies"]
  C --> T["Time wasted"]

The discipline: define the symptom and scope the impact before escalating. The escalation message should include:

• What is broken? (symptom in observable terms)
• For whom is it broken? (scope, who is affected)
• What has been tried? (evidence collection, hypotheses tested)
• What is needed? (specific help, not “please look”)

The senior engineer who receives a clear escalation can begin investigating immediately.

Anti-pattern 5: Change-and-pray

The operator makes a configuration change and assumes it fixed the issue without validating. The change may have fixed the symptom but not the cause; the cause persists; the issue recurs. Or the change may have masked the symptom; the underlying cause is still active; the operator has not addressed it.

flowchart LR
  S["Symptom reported"] --> C["Make change"]
  C --> A["Assume it worked"]
  A --> X["No validation"]
  X --> R["Issue recurs"]

The discipline: every change is validated. The operator runs the same diagnostic before and after the change; the comparison proves the change had the expected effect. If the change did not have the expected effect, the operator reverts and tries the next hypothesis.

Anti-pattern 6: Documenting without acting

The opposite of action-without-evidence: the operator documents everything but does not act. The incident remains unresolved while the operator writes extensive documentation. The discipline: evidence first, then action, then validation, then documentation. The operator who documents without acting has not finished the troubleshooting.

flowchart LR
  S["Symptom reported"] --> E["Collect evidence"]
  E --> D["Document evidence"]
  D --> X["No action"]
  X --> R["Issue unresolved"]

The discipline: action follows evidence. The operator who has evidence but has not acted is not troubleshooting; is delaying. The operator who documents while acting (or after acting) is professional.

Anti-pattern 7: Trusting memory over output

The operator “knows” what the configuration is because they configured it. The operator’s memory may be wrong; the configuration may have been changed by another operator; the operator may be confusing this router with another. The operator who relies on memory may form a hypothesis based on a configuration that does not exist.

flowchart LR
  S["Symptom reported"] --> M["Recall configuration from memory"]
  M --> H["Form hypothesis based on memory"]
  H --> W["Memory is wrong"]
  W --> R["Wasted investigation"]

The discipline: always run show configuration commands. Trust the output, not the memory. The output is the canonical state of the router.

Anti-pattern 8: Following the runbook blindly

The operator follows the runbook without thinking. The runbook is for the canonical case; the operator’s incident may not be canonical. The operator who follows the runbook blindly may apply the wrong fix because the runbook assumed a different cause.

flowchart LR
  S["Symptom reported"] --> R["Follow runbook"]
  R --> B["Runbook assumes canonical cause"]
  B --> W["Actual cause is different"]
  W --> F["Wrong fix applied"]

The discipline: use the runbook as a guide, not as a script. The operator who understands the runbook’s assumptions can adapt when the incident differs from the canonical case. The operator who follows the runbook without thinking will apply the wrong fix when the incident is non-canonical.

Production discipline

Cross-course references

• Part LII-01 (LII-VyOS-Troubleshoot / define-and-scope) covers the incident-definition that prevents escalation-without-symptom.
• Part LII-02 (LII-VyOS-Troubleshoot / evidence-first) covers the evidence collection that prevents reboot-before-evidence.
• Part LII-03 (LII-VyOS-Troubleshoot / hypothesis-driven) covers the hypothesis formation that prevents change-and-pray.
• Part LII-04 (LII-VyOS-Troubleshoot / subsystem-by-subsystem) covers the subsystem isolation that prevents blame-the-firewall.
• Part LII-05 (LII-VyOS-Troubleshoot / return-path) covers the bidirectional verification that prevents one-way-troubleshooting.
• The Observability course covers the monitoring side (post-mortem documentation, anti-pattern tracking).

Quiz