The troubleshooting loop - ten steps and their stop-rules

Under pressure, everybody falls back to habit. If the habit is “restart it and see”, that is what happens at 03:00 with the whole company watching. The loop below is the habit this course trains instead. Every runbook, break/fix scenario, and incident lesson in this course refers back to it by name.

It is a loop, not a checklist, because step 7 sends you back to step 6 far more often than it sends you forward.

The ten steps

1. Define the symptom - turn a complaint into a measurement
2. Determine the impact - who is affected, and how badly
3. Check recent changes - what moved in the window before the symptom
4. Collect evidence - capture state before you disturb it
5. Identify the subsystem - narrow to one layer
6. Form a hypothesis - one falsifiable statement
7. Test safely - the smallest reversible test
8. Restore service - mitigate, which is not always the fix
9. Find the root cause - after the pressure is off
10. Prevent recurrence - close the detection gap and the failure gap

Each step below has a stop-rule: the condition that says the step is finished. Without stop-rules people either move on with nothing, or collect data forever. The stop-rule is the part most methodologies leave out.

Step 1 - Define the symptom

Turn the complaint into a statement with an observed value, an expected value, and a start time. “The application is slow” is not a symptom. “The /health endpoint returns in 5.2 s instead of 100 ms, first seen at 14:10 UTC” is.

$ date -u; curl -sS -o /dev/null -w 'http=%{http_code} dns=%{time_namelookup}s connect=%{time_connect}s total=%{time_total}s
' https://app.example.com/health

Mon Aug 10 14:22:07 UTC 2026
http=200 dns=0.004s connect=0.009s total=5.213s

Stop-rule: you can write the symptom as observed value versus expected value, with a first-seen time. If you cannot reproduce or observe it yourself, you are not finished - go and get a measurement, or record explicitly that it is currently not reproducible.

Step 2 - Determine the impact

Impact decides urgency, severity, and whether you keep troubleshooting alone or declare an incident and hand command to someone else.

$ systemctl list-units --state=failed --no-pager; ss -s; journalctl -p err -b --since '1 hour ago' --no-pager | wc -l

  UNIT              LOAD   ACTIVE SUB    DESCRIPTION
● myapp.service    loaded failed failed Application server
1 loaded units listed.
Total: 412 (kernel 0)
TCP:   118 (estab 41, closed 52, orphaned 0, timewait 51)
2317

Stop-rule: you can name what is affected, roughly how many users or services, and whether it is stable, degrading, or recovering - and you have assigned a severity. If impact is total or growing, declare the incident now and continue the loop under an incident commander.

Step 3 - Check recent changes

This is the highest-yield question in incident response and the one people skip because it feels like admin rather than engineering. Most production incidents follow a change. Ask what moved before you ask what broke.

$ journalctl --since '4 hours ago' -p warning --no-pager | head -40

Aug 10 13:58:11 web01 systemd[1]: Reloading nginx.service...
Aug 10 14:09:44 web01 kernel: TCP: request_sock_TCP: Possible SYN flooding on port 443
Aug 10 14:10:02 web01 myapp[2214]: WARN pool exhausted, waiting for connection

$ grep ' install \| upgrade \| remove ' /var/log/dpkg.log | tail -20

2026-08-10 13:55:02 upgrade nginx:amd64 1.24.0-2 1.24.0-3
2026-08-10 13:55:09 status installed nginx:amd64 1.24.0-3

$ git -C /etc log --since '24 hours ago' --stat --no-pager | head -30; ls -lt /etc | head -10

commit 4f2a9c1 2026-08-10 13:54 +0000  root
committing changes in /etc after apt run
nginx/conf.d/upstream.conf | 4 +--
1 file changed, 2 insertions(+), 2 deletions(-)

Also check the sources that live outside the host: the change management ticket queue, the configuration management run log (journalctl -u ansible-pull, Puppet or Salt reports), the deployment pipeline, and any infrastructure change such as a firewall rule, DNS record, or certificate rotation.

Stop-rule: you have a time-ordered list of every change in the window spanning the symptom start, from packages, configuration, deployments, and infrastructure. If a change lines up with the start time, that is your first hypothesis - carry it forward to step 6 rather than starting from scratch.

Step 4 - Collect evidence

Capture state before you disturb it. A restart is the single most common way engineers destroy the evidence that would have explained the incident.

$ INC=/var/tmp/inc-$(date -u +%Y%m%dT%H%M%SZ); mkdir -p $INC; journalctl -b --no-pager > $INC/journal.txt; dmesg -T > $INC/dmesg.txt; systemctl status myapp --no-pager -l > $INC/unit.txt; df -h > $INC/df.txt; df -i > $INC/dfi.txt; free -m > $INC/free.txt; ss -tanp > $INC/sockets.txt; ps auxf > $INC/ps.txt

(no output; $INC now holds the capture)

Stop-rule: new commands stop changing your picture of the problem. If the last three things you ran told you nothing you did not already know, stop collecting and move on.

Step 5 - Identify the subsystem

Narrow the problem to one layer: CPU, memory, disk, network, the application itself, or a dependency. Sweep the layers rather than diving into the first one you suspect.

$ vmstat 1 5; iostat -xz 1 5; sar -n DEV 1 5; ss -s

procs -----------memory---------- ---system-- ------cpu-----
r  b   swpd   free   buff  cache   in   cs us sy id wa st
1  9      0 412332  91224 2044108 4102 8811  6  4 12 78  0
Device  r/s   w/s  rkB/s  wkB/s  r_await  w_await  aqu-sz  %util
nvme0n1 11.0 940.0  176.0 118400   0.41    41.20    38.7    99.8

Stop-rule: exactly one subsystem shows saturation or errors while the others look normal. If two look bad, work out which is downstream: a caller always looks unhealthy when the callee is unhealthy, so blame the callee first.

Step 6 - Form a hypothesis

A hypothesis is one sentence in the form “if X is the cause, then Y will be observable”. Y is the part that matters, because Y is what you can go and check.

Good: “if the connection pool is exhausted because of the 13:55 upstream change, then ss -tan state established will show the pool at its configured maximum and the application log will show pool-wait warnings.”

Bad: “the database is probably struggling.”

Stop-rule: you can write down an observation that would prove the hypothesis wrong. If nothing could falsify it, it is not a hypothesis - rewrite it until it is.

Step 7 - Test safely

Test the prediction with the smallest, most reversible action available. Prefer read-only. Where a change is unavoidable, validate it offline first and change one thing at a time.

$ nginx -t; sshd -t; nft -c -f /etc/nftables.conf; systemd-analyze verify myapp.service; visudo -c

nginx: configuration file /etc/nginx/nginx.conf test is successful
/etc/sudoers: parsed OK

Stop-rule: the test either confirmed or refuted the prediction. If it refuted it, go back to step 6, not step 1 - your evidence is still good, only the explanation was wrong. If the test would change more than it measures, it is the wrong test.

Step 8 - Restore service

Mitigation is not the same as the fix. Failing over, rolling back the 13:55 change, restarting the unit, draining the node, or rate-limiting an abusive client can all restore service without anyone yet knowing the root cause. That is correct sequencing, not a shortcut.

Stop-rule: the measurement from step 1 is back inside its expected range and you have said so to the people tracking the incident. Then stop debugging on the live path - continue on a copy, a snapshot, or the evidence bundle from step 4.

Step 9 - Find the root cause

Now, with the evidence bundle and no clock pressure, reconstruct the timeline. Distinguish the trigger (the 13:55 config change) from the contributing factors (no pool-saturation alert, no canary deployment) from the root cause (the pool size was tuned for the old upstream and nothing tied the two together).

$ coredumpctl list; journalctl -b -u myapp --since '13:50' --until '14:30' -o short-precise --no-pager

TIME                        PID  SIG COREFILE  EXE
Mon 2026-08-10 14:11:03 UTC 2214 SIGABRT present  /usr/bin/myapp
14:10:02.114 web01 myapp[2214]: WARN pool exhausted, waiting for connection
14:11:03.882 web01 myapp[2214]: FATAL pool wait timeout after 60s

Stop-rule: you can explain every observation in the evidence, including the ones that did not fit your first story. An unexplained observation means the story is incomplete, and an incomplete story produces a prevention action that prevents the wrong thing.

Step 10 - Prevent recurrence

Close two gaps, not one. The failure gap is why it broke. The detection gap is why nobody knew until a user complained. Most teams only ever address the first.

Stop-rule: at least one action exists with a named owner and a due date, the detection gap has an action of its own, and the runbook or checklist that would have helped has been updated. Until then the incident is not closed.

Looping back

The loop has three legs, and knowing which one you are on saves a lot of thrashing.

• A hypothesis was refuted at step 7: return to step 6.
• The evidence turns out to be about a different problem than the one reported: return to step 1.
• A new symptom appears while you work: start a second pass at step 1 and track it separately. Do not fold it into the current one - two symptoms in one investigation is how investigations stall.

Knowledge check