Defining the symptom - turning a complaint into a measurement

Step 1 of the loop looks like paperwork and is actually the step that decides how long the whole investigation takes. An investigation that starts from “the application is slow” has an unbounded search space. One that starts from “the /health endpoint on web02 returns in 5.2 s instead of 100 ms, first seen at 14:10 UTC, and web01 is unaffected” has already excluded most of the estate.

This lesson is the practical detail behind that one step. The loop itself, and what the other nine steps do, is in the troubleshooting loop.

The symptom statement

A usable symptom has five parts. Write them in this order, because each one narrows the next.

1. The thing being measured: a named endpoint, unit, mount point, query or interface - not "the app"
2. The observed value: a number you took yourself, with its units
3. The expected value: what it is normally, and where that expectation comes from
4. The first-seen time, in UTC: when it started, not when someone reported it
5. The scope: which hosts, users, regions or requests show it - and which demonstrably do not

The last part is the one people leave out, and it is the one that halves the work. “Every host” and “one host” are different investigations, and so are “every request” and “one in fifty”.

From complaint to measurement

Users report consequences, not symptoms. The translation is yours to do, and it is mostly a matter of choosing the smallest thing you can measure repeatedly.

The curl write-out format is the fastest way to split a latency complaint into its stages, because it separates resolution, connection, TLS handshake and the application in a single request.

$ date -u; curl -sS -o /dev/null -w 'code=%{http_code} dns=%{time_namelookup} tcp=%{time_connect} tls=%{time_appconnect} ttfb=%{time_starttransfer} total=%{time_total}
' https://app.example.com/health

Tue Aug 11 09:14:02 UTC 2026
code=200 dns=0.004 tcp=0.009 tls=0.041 ttfb=5.180 total=5.213

Take the measurement at least three times before you believe it. A single sample cannot distinguish a persistent 5-second response from one slow request in a healthy series, and those are different problems.

$ for i in 1 2 3 4 5; do curl -sS -o /dev/null -w '%{http_code} %{time_total}\n' https://app.example.com/health; sleep 2; done

200 5.204
200 0.098
200 5.181
200 0.101
200 5.219

That output changes the symptom statement. It is no longer “the endpoint is slow”; it is “half the responses are slow”, which points at a load-balanced pool rather than at the application as a whole.

The four scoping questions

Ask these before you run anything expensive. Each answer removes a layer from consideration.

• Who is affected? One user, one team, everyone. A single user usually means a client, a session, a permission or a cached record - not the server.
• From where? The same host, the same segment, the same region, the internet. A failure that reproduces from the server itself is not a network problem.
• Since when? Compare the first-seen time against the change window. This is what feeds step 3.
• What else shares the path? If one service on a host is affected and four others are not, the host is probably fine. If all five are affected, the host or its network is the suspect.

$ systemctl is-active nginx myapp postgresql; ss -ltn '( sport = :443 or sport = :5432 )'; uptime

active
active
active
State  Recv-Q Send-Q Local Address:Port  Peer Address:Port
LISTEN 0      511         0.0.0.0:443        0.0.0.0:*
LISTEN 0      244       127.0.0.1:5432       0.0.0.0:*
09:15:41 up 41 days,  2:11,  2 users,  load average: 0.94, 1.02, 0.88

Reproduce it from more than one place

A symptom you can reproduce on demand is a symptom you can test a fix against. Reproduce it deliberately, and reproduce it from progressively narrower vantage points, because the point where it stops reproducing is itself the answer to step 5.

1. From the reporter, exactly as they did it - including their client, their credentials and their region
2. From outside the estate against the public name
3. From another host inside the estate against the load balancer address
4. From the load balancer against one backend directly, bypassing the pool
5. From the backend host itself against 127.0.0.1, bypassing the network entirely

$ curl -sS -o /dev/null -w 'via-lb %{http_code} %{time_total}\n' https://app.example.com/health; curl -sS -o /dev/null --resolve app.example.com:443:192.0.2.21 -w 'backend-21 %{http_code} %{time_total}\n' https://app.example.com/health

via-lb 200 5.198
backend-21 200 0.097

When it will not reproduce

Intermittent symptoms are the ones that get closed as “could not reproduce” and then page someone else next week. They are workable, but they need different tactics.

1. Record the symptom statement anyway, with the reporter as the source of the observed value and a note that you could not observe it directly
2. Widen the sampling: run the measurement on a loop and keep the results, rather than watching once and moving on
3. Look for the pattern in what you already collect - time of day, one host in a pool, one client, one code path, one shard
4. Add the instrumentation you wished you had, and say so explicitly as a follow-up action
5. Close it as "not currently reproducible with monitoring added", never as "fixed"

$ while :; do printf '%s ' "$(date -u +%H:%M:%S)"; curl -sS -o /dev/null -w '%{http_code} %{time_total}\n' https://app.example.com/health; sleep 10; done | tee -a /var/tmp/probe-$(date -u +%Y%m%d).log

09:20:11 200 0.096
09:20:21 200 0.099
09:20:31 200 5.187
09:20:41 200 0.101

The symptom is the closing criterion

Step 8 says restore service, and the stop-rule for step 8 is that the measurement from step 1 is back inside its expected range. That only works if the measurement was written down and is still being taken.

Keep the probe running through the fix. Two things go wrong when it is not:

• The change is declared successful because the error stopped appearing in a log that the change also stopped writing.
• The symptom recurs twenty minutes later and nobody notices, because everyone stopped looking at the same moment.

Knowledge check