Narrowing to one subsystem - bisect the path, then test one thing

Steps 5, 6 and 7 are the middle of the loop and where the time goes. You have a measured symptom and a change list; you do not yet know which part of the system is at fault. The work now is to halve the search space repeatedly, then to state one explanation that could be wrong, then to find out cheaply whether it is.

Two ways to narrow, and they are not interchangeable

There are two independent ways to cut the search space, and choosing the wrong one is why investigations stall.

• Bisect the path. A request crosses a sequence of hops: client, resolver, load balancer, host, process, dependency. Test at a midpoint and you eliminate half the hops with one command. Use this when the symptom is a failure or a specific slowness - something is broken somewhere on a known route.
• Sweep the resources. Check utilisation, saturation and errors for CPU, memory, disk and network on one host. Use this when the symptom is generalised degradation on a host where everything is slow at once. That method is the USE methodology, and it answers a different question: not “which hop”, but “which resource on this host is exhausted”.

If everything on a host is slow, sweep. If one thing is broken, bisect. Sweeping when you should bisect produces a page of healthy-looking metrics; bisecting when you should sweep leads you to the first hop that looks busy and stops there.

Bisecting the path

Write down the hops before you test any of them. The list is the search space, and it is usually shorter than it feels.

1. Client resolves the name - resolver, cache, search domain
2. Client connects to the address - routing, firewall, NAT
3. Load balancer terminates TLS and picks a backend - certificate, pool membership, health checks
4. Backend host accepts the connection - listener, socket backlog, local firewall
5. Process handles the request - threads, pool, locks, garbage collection
6. Dependency answers - database, cache, object store, external API

Then test the midpoint. Each test below either passes or fails, and either answer eliminates half the list.

$ getent hosts app.example.com; dig +short @192.0.2.53 app.example.com; ip route get 192.0.2.10

192.0.2.10      app.example.com
192.0.2.10
192.0.2.10 via 192.0.2.1 dev eth0 src 192.0.2.55 uid 1000

$ curl -sS -o /dev/null --connect-timeout 3 -w 'lb %{http_code} %{time_total}\n' https://app.example.com/health; curl -sS -o /dev/null --connect-timeout 3 --resolve app.example.com:443:192.0.2.21 -w 'be21 %{http_code} %{time_total}\n' https://app.example.com/health; curl -sS -o /dev/null --connect-timeout 3 --resolve app.example.com:443:192.0.2.22 -w 'be22 %{http_code} %{time_total}\n' https://app.example.com/health

lb 200 5.190
be21 200 0.094
be22 200 10.021

$ ss -ltn; ss -tn state established '( dport = :5432 )' | wc -l; ss -tni state established '( dport = :5432 )' | head -6

State  Recv-Q Send-Q  Local Address:Port
LISTEN 0      511           0.0.0.0:443
LISTEN 128    511         127.0.0.1:8080
41
ESTAB  0      0      192.0.2.22:47110  192.0.2.30:5432
cubic wscale:7,7 rto:204 rtt:0.28/0.05 cwnd:10 bytes_sent:81234

Every one of those commands is read-only, and between them they cut a six-hop path down to one hop in under a minute.

The downstream rule

At the end of a bisection two subsystems often both look unhealthy. Decide the direction before you act, because a caller always looks sick when its callee is sick.

• The application shows connection timeouts and the database shows high CPU. The database is the callee. Look there first.
• The load balancer reports backends flapping and the backends report high load. The backends are the callee.
• Disk latency is high and the application is queuing. Storage is downstream of the application. Storage first.

The exception is worth knowing because it inverts the conclusion: a caller that retries aggressively can cause the callee to look saturated. If the callee is saturated with work that is mostly retries of requests that already timed out, the caller is the fault. Check whether the callee is doing more work than usual or the same work more slowly - a request rate that jumped 5x at the moment of the incident is retry storm, not organic load.

Writing a hypothesis that can be wrong

Step 6 asks for one sentence in the form “if X is the cause, then Y will be observable”. Y is the whole point: it is what you can go and check, and it is what makes the hypothesis capable of being wrong.

A useful discipline: also write down what you expect to see if the hypothesis is false. If both outcomes look the same on the evidence you plan to collect, the test cannot distinguish them and you need a different test.

Designing a safe test

The test that settles a hypothesis should measure much more than it changes. Rank your options and take the highest one on this list that can distinguish the outcomes.

1. Read live state: ss, ip, systemctl show, /proc, journalctl - changes nothing
2. Validate configuration offline: nginx -t, sshd -t, visudo -c, findmnt --verify, systemd-analyze verify
3. Reproduce on a copy: a staging host, a snapshot, a second member of the same pool that is already out of service
4. Change one thing on one host, with the rollback command already written down
5. Change one thing on all hosts - which is not a test, it is a fix, and it belongs at step 8

$ nginx -t; sshd -t; visudo -c; findmnt --verify --verbose | tail -3; systemd-analyze verify /etc/systemd/system/myapp.service

nginx: configuration file /etc/nginx/nginx.conf test is successful
/etc/sudoers: parsed OK
Success, no errors found in 6 entries

Three properties make a test safe:

• One variable. Two changes at once and a recovery tells you nothing about which one mattered - and you now have to keep both, including the one that was pointless.
• A known rollback. Write the undo command before you run the do command. If you cannot state the rollback, you are not running a test, you are making a change.
• A bounded blast radius. One host out of a pool, one request path, one tenant. A test that can only be run on everything at once is a test you should schedule, not one you run at 03:00.

When the test refutes the hypothesis

This is the normal case, not a failure. A refuted hypothesis costs one command and eliminates one explanation.

Return to step 6, not step 1. The evidence is still good; only the story about it was wrong. Concretely:

• Keep the evidence bundle and the change list.
• Cross the hypothesis off in the incident log, with the observation that refuted it.
• Ask what the refuting observation itself suggests - it is new evidence, and it frequently points straight at the next hypothesis.

Going back to step 1 after every refutation is what makes an investigation feel like it is going in circles. It usually is.

Knowledge check