Evidence first — collect before changing, write down symptoms, no action without data

The evidence-first discipline is the operator’s commitment to collect data before acting. The operator who changes configuration without evidence is gambling: the change may fix the issue, may make it worse, or may mask it without fixing. The operator who collects evidence first has a baseline to compare against, can prove the change had the expected effect, and can revert if it did not.

This lesson is the second in Part LII: the discipline of collecting evidence before acting, the canonical evidence checklist for routing-protocol issues, and the production discipline that prevents evidence-less changes.

The evidence-first principle

flowchart LR
  S["Symptom reported"] --> E["Collect evidence<br/>(logs, counters, state)"]
  E --> W["Write down the evidence<br/>(with timestamps)"]
  W --> H["Form hypothesis"]
  H --> T["Test hypothesis<br/>(one variable at a time)"]
  T --> V["Validate<br/>(compare evidence before/after)"]
  V --> F["Fix or rollback"]

The principle: every action is preceded by evidence collection; every hypothesis is tested with evidence; every change is validated by comparing evidence before and after.

The discipline: an operator who cannot answer “what evidence did you collect before this change?” has not followed the evidence-first principle. The change may have been correct, but it cannot be defended in the postmortem.

The evidence checklist

For a routing-protocol issue, the canonical evidence checklist:

flowchart TD
  E["Evidence checklist"]
  E --> E1["1. Current configuration<br/>(show configuration commands)"]
  E --> E2["2. Recent changes<br/>(show system commit, compare N M)"]
  E --> E3["3. Live state<br/>(show ip route, show bgp ipv4)"]
  E --> E4["4. Protocol state<br/>(show bgp ipv4 summary, show ip ospf neighbor)"]
  E --> E5["5. Interface state<br/>(show interfaces, ip link)"]
  E --> E6["6. Logs<br/>(journalctl, monitor log)"]
  E --> E7["7. Counters<br/>(show ip route summary, ip -s link)"]
  E --> E8["8. Connectivity<br/>(ping, traceroute)"]

Each item answers a specific question:

1. Current configuration. What is the router configured to do? show configuration commands prints the running configuration as the set of set commands that produced it — the form you can diff, paste into a ticket, and compare against a peer router.
2. Recent changes. What was changed recently, and by whom? show system commit lists the stored revisions with their timestamp, user and commit comment. Revision 0 is the running configuration and the numbers count backwards in time, so revision 1 is the state before the most recent commit. That listing tells you when and who; it does not tell you what. For the content of a change, compare 1 0 diffs the previous revision against the running one, and compare <N> <M> diffs any two. If the configuration is also kept as code, git log gives the same answer with review history attached.
3. Live state. What is the router actually doing? show ip route, show bgp ipv4, show ip ospf database.
4. Protocol state. What is the protocol-specific state? show bgp ipv4 summary, show ip ospf neighbor, show bgp ipv4 <prefix>.
5. Interface state. What is the interface state? show interfaces for the CLI view; ip -s link show eth0 for the kernel’s error, drop and carrier-transition counters, which the CLI summary does not break out.
6. Logs. What does the log say? journalctl -u frr, journalctl -u strongswan, and monitor log for the live tail.
7. Counters. What are the counters, and are they moving? A single reading of show ip route summary or ip -s link show eth0 is a number; two readings a minute apart are evidence. Capture both.
8. Connectivity. Is the path actually working? ping, traceroute, tracepath.

The operator captures each item before testing any hypothesis. The output is saved to a file or ticket.

Writing down the evidence

The discipline of writing down the evidence with timestamps:

# Evidence — R1 BGP session down to ISP-A
# Captured: 2026-08-15 13:42:00 UTC by operator@example.com
# This is the operator's notes file. Each heading names the command
# that produced the material under it, so a reviewer can re-run it.

Configuration (show configuration commands | match bgp):
  set protocols bgp system-as '64512'
  set protocols bgp neighbor 198.51.100.1 remote-as '64510'
  set protocols bgp neighbor 198.51.100.1 update-source 'eth0'
  set protocols bgp address-family ipv4-unicast network '203.0.113.0/24'

Recent changes (show system commit):
  0   2026-08-14 09:15:12 by operator@example.com via cli
  1   2026-07-30 11:02:44 by operator@example.com via cli
  -> only one commit since the last known-good state; diff it with
     `compare 1 0` and paste the diff here rather than guessing.

Live state (show bgp ipv4 summary):
  peer 198.51.100.1, remote AS 64510, state Active,
  Up/Down 00:00:00, prefixes received 0
  -> the session has never reached Established in this window.

Protocol state (show bgp ipv4 neighbors 198.51.100.1 routes):
  no routes — consistent with a session that is not Established.

Interface state (show interfaces ethernet eth0):
  eth0 admin up, link up, MTU 1500, address 198.51.100.2/24
  ip -s link show eth0: rx errors 0, tx errors 0, carrier changes 0
  -> the physical path is not flapping.

Logs (journalctl -u frr --since "1 hour ago"):
  Aug 15 13:30:00 r1 bgpd[1234]: %NOTIFICATION: sent to neighbor 198.51.100.1 (64510) code 4 (Hold Timer Expired)
  Aug 15 13:30:01 r1 bgpd[1234]: %NOTIFICATION: received from neighbor 198.51.100.1 (64510) code 4 (Hold Timer Expired)

Connectivity (ping 198.51.100.1):
  4 packets transmitted, 4 received, 0% packet loss, rtt avg 0.5 ms
  -> IP reachability to the peer address is fine.

Two things make this a usable artefact rather than a wall of text. Every section names the command that produced it, so the next person can re-run it and get a comparable reading. And every section ends with the one-line conclusion the operator drew from it — because in an hour’s time, “state Active” will still be legible but “and therefore the link is not the problem” will not be obvious.

Note the shape of the configuration capture: show configuration commands emits set lines, not the brace-and-indent tree that show configuration prints. Capture the set form. It is the form you can diff against another router, paste into a ticket, and — if it comes to that — replay.

The evidence is timestamped and saved. The operator can refer back to it during hypothesis testing.

The evidence-first workflow

sequenceDiagram
  participant Operator
  participant System as VyOS router

  Operator->>System: Step 1: Capture configuration
  System-->>Operator: Configuration output
  Operator->>System: Step 2: Capture live state
  System-->>Operator: Live state output
  Operator->>System: Step 3: Capture logs
  System-->>Operator: Logs output
  Operator->>System: Step 4: Capture counters
  System-->>Operator: Counters output
  Operator->>System: Step 5: Capture connectivity
  System-->>Operator: Connectivity output
  Operator->>Operator: Step 6: Form hypothesis
  Operator->>Operator: Step 7: Predict expected evidence<br/>(what should change if hypothesis is correct)
  Operator->>System: Step 8: Apply one change
  Operator->>System: Step 9: Re-capture evidence
  Operator->>Operator: Step 10: Compare evidence before/after
  Operator->>Operator: Step 11: Hypothesis confirmed or rejected

Steps 1-5 are evidence collection. Step 6 is hypothesis formation (Part LII-03). Step 7 is prediction of expected evidence. Step 8 is the action (single change, isolated variable). Steps 9-11 are validation.

The discipline: each step has evidence; each change has predicted and observed evidence; the comparison is the validation.

The evidence without action anti-pattern

The most common failure mode of evidence-first is “evidence without action”: the operator collects evidence but does not act. The evidence is captured but the fix is not applied. This is sometimes called “analysis paralysis” or “documenting without fixing”.

The opposite anti-pattern is “action without evidence”: the operator acts without collecting evidence. The fix may be applied, but the operator cannot defend it in the postmortem; if the fix causes a regression, the operator cannot revert because they do not know what state the router was in before the fix.

The discipline: evidence first, then action, then validation. The operator who collects evidence and acts on it is the operator who resolves incidents quickly and defensibly.

Production failure modes

The evidence-first failure modes the operator encounters:

• Action without evidence. The operator changes the BGP timer without collecting the current timer, the recent changes, or the logs. The change may fix or break the issue; the operator cannot tell which because there is no baseline. Fix: collect evidence first.
• Evidence without action. The operator collects all the evidence but does not act. The incident remains unresolved. Fix: act on the evidence; do not let evidence collection become the goal.
• Evidence without timestamps. The operator collects evidence but does not timestamp it. The evidence cannot be correlated with the incident timeline. Fix: timestamp every piece of evidence.
• Evidence without prediction. The operator makes a change but does not predict what evidence should change. The operator cannot tell whether the change worked. Fix: predict the expected evidence before making the change.
• Multiple variables in one change. The operator changes three things at once. The operator cannot tell which change had the effect. Fix: one variable at a time.
• Evidence in the operator’s head. The operator “knows” what the configuration is and does not run show configuration commands. The operator’s memory may be wrong. Fix: always run the show command; trust the output, not the memory.

Rollback

Evidence collection changes are about audit, not rollback. The discipline:

• Save evidence outputs to a file or ticket.
• Reference the evidence in the change-management ticket.
• Compare evidence before/after the change.

Production discipline

Cross-course references

• Part LII-01 (LII-VyOS-Troubleshoot / define-and-scope) covers the incident-definition that precedes evidence collection.
• Part LII-03 (LII-VyOS-Troubleshoot / hypothesis-driven) covers the hypothesis formation that follows evidence collection.
• Part LII-04 (LII-VyOS-Troubleshoot / subsystem-by-subsystem) covers the subsystem isolation that uses evidence.
• Part LII-06 (LII-VyOS-Troubleshoot / troubleshooting anti-patterns) covers the anti-patterns of evidence-first (action without evidence, evidence without action).
• The Observability course covers the monitoring side (logs, metrics, traces — the sources of evidence).
• The Ansible course’s XLII-Ansible-BeyondLinux covers the automation hand-off (evidence collection automated via playbooks).

Quiz