Hypothesis-driven — generate hypotheses, test each, bisection, post-mortem

The hypothesis-driven discipline is the operator’s commitment to forming and testing hypotheses rather than guessing. The operator who guesses at the cause and applies a “fix” without testing is gambling. The operator who forms a hypothesis, predicts the expected evidence, applies one change, and validates with new evidence is reasoning. Hypothesis-driven troubleshooting is the difference between gambling and engineering.

This lesson is the third in Part LII: the discipline of generating and testing hypotheses, the use of bisection to narrow scope, the post-mortem documentation, and the production workflow that converts evidence into a defensible root-cause analysis.

The hypothesis-driven workflow

sequenceDiagram
  participant Operator
  participant System as VyOS router

  Note over Operator: Step 1: Generate hypotheses<br/>(from evidence)
  Operator->>Operator: H1, H2, H3, ...
  Note over Operator: Step 2: Rank by likelihood
  Operator->>Operator: Most likely first
  Note over Operator: Step 3: For each hypothesis:<br/>predict expected evidence
  Operator->>Operator: If H1 is true, evidence E should change
  Note over Operator: Step 4: Apply one test
  Operator->>System: One change, isolated variable
  Note over Operator: Step 5: Re-collect evidence
  Operator->>System: Same show commands as before
  Note over Operator: Step 6: Compare before/after
  Operator->>Operator: If predicted → hypothesis confirmed<br/>Else → hypothesis rejected, try next

The workflow is iterative. Each iteration has six steps. The operator iterates until a hypothesis is confirmed or all hypotheses are rejected (in which case, generate new hypotheses).

Generating hypotheses

The operator generates hypotheses from the evidence. For a “BGP session down” symptom, the hypotheses:

flowchart TD
  S["BGP session down"]
  S --> H1["H1: Wrong remote AS"]
  S --> H2["H2: Wrong update-source"]
  S --> H3["H3: MD5 password mismatch"]
  S --> H4["H4: Hold time too short<br/>for the path"]
  S --> H5["H5: Path MTU problem"]
  S --> H6["H6: Firewall blocking TCP 179"]
  S --> H7["H7: ISP-side issue"]

Each hypothesis maps to a specific subsystem and a specific evidence collection:

Each hypothesis names a subsystem, and — this is the part that makes it a hypothesis rather than a hunch — names the evidence that would refute it:

• H1: Wrong remote AS. The peer rejects the OPEN message. show ip bgp neighbors 198.51.100.1 shows a state short of Established and a last-reset line naming an OPEN message error; monitor log carries the same notification. Refuted if the configured remote-as matches what the peer’s operator states.
• H2: Wrong update-source. The router sources the session from an address the peer has no neighbour statement for, so the peer never answers. show ip bgp neighbors 198.51.100.1 prints the local host address in use; compare it with the address the peer expects. Refuted if they match.
• H3: MD5 password mismatch. There is no BGP-level error to read for this one, because the kernel’s TCP-MD5 support discards segments whose signature does not verify before BGP ever sees them. tcpdump -ni eth0 tcp port 179 shows repeated SYNs with no SYN-ACK. Refuted if the three-way handshake completes.
• H4: Hold time too short for the path. The session reaches Established and then resets on a timer. show ip bgp neighbors 198.51.100.1 reports the reset reason and the negotiated hold time; correlate the reset timestamps against latency and loss to the peer over the same window. Refuted if the resets are not periodic at the hold-time interval.
• H5: Path MTU problem. The session establishes and then stalls part-way through the initial update — show ip bgp summary shows Established with a prefix count that stops moving. Test the path with ping 198.51.100.1 do-not-fragment size 1472. Refuted if the full-size probe returns.
• H6: Firewall blocking TCP 179. BGP to the router itself is filtered by set firewall ipv4 input filter, not by the forward chain — a rule set that only filters transit traffic is not the one to read. show firewall statistics shows whether a rule is counting the drop. Refuted if no input rule counts a match on port 179.
• H7: Something on the peer’s side. The router’s SYNs leave (tcpdump), nothing comes back, and no local rule counts a drop. This is the hypothesis of last resort because it is the only one the operator cannot refute alone; it needs the peer’s operator to look at their side.

The discipline: each hypothesis is falsifiable. The operator can state, in advance, the observation that would prove it wrong — and H7 is at the bottom of the list precisely because that observation is not available locally.

Bisection

For complex issues, the operator uses bisection to narrow the scope. Bisection is the technique of dividing the problem space in half and testing each half:

flowchart TD
  S["Symptom: all external traffic fails"]
  S --> Q1{Is the local router's egress working?}
  Q1 -->|yes| Q2{Is the BGP session up?}
  Q1 -->|no| Q3["Issue is local<br/>(interface, routing, firewall)"]
  Q2 -->|yes| Q4{Is the path to ISP working?}
  Q2 -->|no| Q5["Issue is BGP<br/>(configuration, peer, hold-time)"]
  Q4 -->|yes| Q6{Is the remote side working?}
  Q4 -->|no| Q7["Issue is transit<br/>(link, ISP network)"]
  Q6 -->|yes| Q8["Issue is application/destination"]
  Q6 -->|no| Q9["Issue is ISP-side<br/>(contact ISP)"]

Halving beats walking: a path of eight subsystems takes about three tests to bisect rather than up to eight to walk. That is the whole arithmetic case, and it holds only under a precondition worth stating out loud.

Bisection needs a space that is ordered and a property that is monotone along it — everything below the cut works, everything above it fails. A forwarding path has that shape: hops are ordered, and a break at hop 4 makes hops 5 and beyond unreachable. A list of hypotheses does not have that shape at all; hypotheses are unordered alternatives, and testing “the middle one” is meaningless. Rank hypotheses by likelihood and test them in order; bisect the path, the configuration, or the commit history.

An intermittent fault breaks the monotone property, and this is where bisection quietly lies. A test that passes because the fault was not firing at that moment is read as “clean below this point”, the search moves the wrong way, and the operator ends up with a confident answer that is wrong. If the symptom is intermittent, repeat each bisection probe enough times to see the fault before you believe a passing result.

The operator uses bisection to:

1. Isolate the subsystem. The operator determines which subsystem (routing, firewall, interface, application) is the issue.
2. Isolate the configuration. The operator determines which configuration block is the issue.
3. Isolate the recent change. The operator determines which recent change is the cause.

Testing a hypothesis

For each hypothesis, the operator:

1. Predicts the expected evidence. “If H1 (wrong remote AS) is true, the peer will reject our OPEN message, so the session will never leave Active/Connect and the last-reset line will name an OPEN message error rather than a hold-timer expiry.”
2. Applies a single test. This may be a configuration change, a show command, or a connectivity test.
3. Re-collects the evidence. The same show commands as before.
4. Compares before/after. If the evidence matches the prediction, the hypothesis is confirmed. If not, the hypothesis is rejected.

Example for H1 (wrong remote AS). The evidence first:

vyos@vyos:~$ show ip bgp neighbors 198.51.100.1

BGP neighbor is 198.51.100.1, remote AS 64511, local AS 64512, external link
BGP version 4, remote router ID 0.0.0.0, local router ID 192.0.2.1
BGP state = Active
Last read 00:00:07, Last write never
Hold time is 180, keepalive interval is 60 seconds
Last reset 00:00:07,  Notification received (OPEN Message Error/Bad Peer AS)

Read the reset reason, not the state. A state of Active says only “not established”; every one of the seven hypotheses can produce it. The Notification received (OPEN Message Error/Bad Peer AS) line is the one that distinguishes H1 from the rest, and it is the peer telling us what it thinks of our OPEN message.

Then the change — one variable, with a safety net under it:

# What is configured, in the router's own words
show configuration commands | match remote-as

# ISP-A confirms their AS is 64511. One variable changes.
configure
set protocols bgp neighbor 198.51.100.1 remote-as 64511
commit-confirm 5

# The prediction: state reaches Established and prefixes arrive
run show ip bgp summary

# Prediction held. Keep the change and persist it.
confirm
save

Note the shape of the set command on VyOS 1.4 and later: the local AS lives once at set protocols bgp system-as 64512, and neighbours hang off set protocols bgp neighbor ... without repeating it. The 1.3 form that embedded the local AS in every neighbour line — set protocols bgp 64512 neighbor ... — is rejected by commit on 1.5.

The hypothesis is confirmed: the session is Established, the prefix count is non-zero, and the reset reason that pointed at H1 is gone. Two of those three matter — a session that establishes and carries no prefixes has moved the problem, not solved it.

The discipline: each hypothesis test has a prediction, a change, and a validation. The operator can defend each step in the post-mortem.

The post-mortem

After the issue is resolved, the operator documents the post-mortem:

# Post-mortem — BGP session down to ISP-A
# Author: operator@example.com
# Date: 2026-08-15 14:30 UTC

## Summary
The BGP session to ISP-A (198.51.100.1) was down from 2026-08-14 09:15 UTC to
2026-08-15 14:00 UTC (approximately 29 hours). The session is now Established.

## Impact
External traffic was intermittently unavailable during the incident. The impact
was limited because ISP-B was still providing connectivity; some external
destinations were slower than usual.

## Root cause
The configured remote AS for ISP-A was 64510. ISP-A's actual AS is 64511
(their network was renumbered on 2026-08-14). The mismatch caused the BGP
session to fail to establish.

## Hypothesis and validation
- Hypothesis 1: Wrong remote AS.
  Prediction: If the remote AS is wrong, `show ip bgp neighbors 198.51.100.1`
  should show the session short of Established with a reset reason naming an
  OPEN message error, not a hold-timer expiry.
  Evidence before: BGP state = Active; "Notification received
  (OPEN Message Error/Bad Peer AS)".
  Action: Verify the configured remote AS against ISP-A's actual AS.
  Evidence after: ISP-A confirms AS is 64511, not 64510.
  Validation: Change `remote-as 64511`, BGP session becomes Established.

## Resolution
Set `remote-as 64511` for ISP-A. The BGP session is now Established.

## Lessons learned
1. The remote AS should be verified against the ISP's documentation during
   onboarding, not assumed.
2. Configuration changes at the ISP side should be communicated to the
   customer in advance.
3. The BGP session should be monitored; a flap should generate an alert.

## Action items
- [ ] Update the runbook to include ISP-AS verification during onboarding.
- [ ] Configure BGP session monitoring (Part XLIX) with alerts.
- [ ] Schedule a post-mortem review with ISP-A.

The post-mortem documents:

• Summary. What happened, when, and the resolution.
• Impact. Who was affected, for how long, and how severely.
• Root cause. The underlying cause, not just the symptom.
• Hypothesis and validation. The reasoning chain from evidence to root cause.
• Resolution. The fix applied.
• Lessons learned. What the operator should do differently.
• Action items. Concrete tasks to prevent recurrence.

The discipline: the post-mortem is the operator’s defence against the same incident recurring. The operator who skips the post-mortem will see the same incident recur because the root cause was not addressed.

Production failure modes

The hypothesis-driven failure modes the operator encounters:

• Premature hypothesis. The operator forms a hypothesis before collecting evidence. The hypothesis may be wrong. Fix: collect evidence first; form hypothesis after.
• Single-hypothesis fixation. The operator tests only one hypothesis and assumes it is correct. The actual cause may be different. Fix: generate at least three hypotheses; test each.
• Hypothesis without prediction. The operator makes a change without predicting the expected evidence. The operator cannot tell whether the change worked. Fix: predict the expected evidence before making the change.
• Multiple variables in one test. The operator changes three things at once and assumes the change worked. The operator cannot tell which change had the effect. Fix: one variable at a time.
• Hypothesis confirmed but wrong. The operator’s change appears to fix the issue but the issue recurs. The hypothesis was wrong (the change was coincidental). Fix: validate the fix over time; if it recurs, revisit the hypothesis.
• No post-mortem. The operator resolves the incident but does not document the post-mortem. The lessons learned are lost. Fix: post-mortem within 24 hours; track action items.

Rollback

Hypothesis-driven changes are about analysis, not rollback. The discipline:

• Apply one change at a time, validate, then apply the next.
• Use commit-confirm 5 for any configuration change so the auto-rollback fires if the change has unintended consequences — including the case where the change severs the session you are working over, which is the one the operator cannot recover from manually.
• Confirm and then save. confirm cancels the pending revert; it does not write the configuration to disk. A change that is confirmed but never saved disappears at the next reboot, which turns a fixed incident into a recurring one.
• Document each change in the post-mortem.

Production discipline

Cross-course references

• Part LII-01 (LII-VyOS-Troubleshoot / define-and-scope) covers the incident-definition that precedes hypothesis formation.
• Part LII-02 (LII-VyOS-Troubleshoot / evidence-first) covers the evidence collection that precedes hypothesis formation.
• Part LII-04 (LII-VyOS-Troubleshoot / subsystem-by-subsystem) covers the subsystem isolation that uses hypothesis testing.
• Part LII-06 (LII-VyOS-Troubleshoot / troubleshooting anti-patterns) covers the anti-patterns of hypothesis-driven troubleshooting.
• The Observability course covers the monitoring and alerting side (hypothesis generation from alerts).

Quiz