Evidence-based diagnosis - capture before you disturb · Linux

Step 4 of the troubleshooting loop is where most investigations are won or lost. The evidence that explains an incident is usually volatile: it lives in process memory, in open file descriptors, in socket state, and in a journal that is about to rotate. A restart removes all of it in under a second.

This lesson is the practical detail behind that one step.

Volatile first, durable second

Order your capture by how fast the evidence disappears.

Volatile and lost on restart: process list, memory maps, open files, socket state, thread stacks
Volatile and lost on reboot: the current boot journal, dmesg ring buffer, tmpfs contents, cgroup counters
Volatile and lost on rotation: older journals, application logs, audit logs
Durable: configuration files, package history, monitoring time series, the change ticket

If you only have thirty seconds before someone restarts the service, spend them on the first line.

The capture

Write everything into one timestamped directory. A capture scattered across a scrollback buffer is not evidence - it cannot be handed over, attached to a ticket, or re-read a week later during the review.

Read-only / Safeset up the bundle— One directory per incident, named in UTC. Using systemctl show to resolve the PID avoids the classic mistake of grepping ps and capturing the grep itself.

$ INC=/var/tmp/inc-$(date -u +%Y%m%dT%H%M%SZ); mkdir -p $INC; PID=$(systemctl show -p MainPID --value myapp.service)

(no output; $INC is the bundle and $PID is the process under investigation)

Illustrative output

Read-only / Safevolatile state— File-descriptor count against the limit in /proc/PID/limits diagnoses descriptor exhaustion in one comparison. /proc/PID/status shows the memory footprint and thread count. None of this exists after the process is restarted.

$ ps auxf > $INC/ps.txt; ss -tanp > $INC/sockets.txt; ls -l /proc/$PID/fd | wc -l > $INC/fdcount.txt; cat /proc/$PID/status > $INC/status.txt; cat /proc/$PID/limits > $INC/limits.txt; cat /proc/$PID/stack > $INC/kstack.txt 2>/dev/null; lsof -p $PID > $INC/lsof.txt

(no output; all of these are read-only inspections of live state)

Illustrative output

Read-only / Safeboot-scoped state— Capture df -i as well as df -h: inode exhaustion produces no-space-left errors on a filesystem that looks half empty. Capture the previous boot journal too if the host rebooted, because that is where the reason for the reboot lives.

$ journalctl -b --no-pager > $INC/journal-boot.txt; journalctl -b -1 --no-pager > $INC/journal-prevboot.txt 2>/dev/null; dmesg -T > $INC/dmesg.txt; systemctl status myapp -l --no-pager > $INC/unit.txt; df -h > $INC/df.txt; df -i > $INC/df-inode.txt; free -m > $INC/free.txt; uptime > $INC/uptime.txt

(no output; the bundle now holds the boot-scoped evidence)

Illustrative output

Reading the journal with intent

Unfiltered journalctl output is not evidence, it is volume. Every query should answer a stated question.

Read-only / Safeone unit, one window— short-precise gives microsecond timestamps. Ordinary output rounds to the second, which is not enough to order events during a fast cascade.

$ journalctl -u myapp --since '2026-08-10 13:50' --until '2026-08-10 14:30' -o short-precise --no-pager

Aug 10 14:10:02.114338 web01 myapp[2214]: WARN pool exhausted, waiting for connection
Aug 10 14:11:03.882901 web01 myapp[2214]: FATAL pool wait timeout after 60s

Illustrative output

Read-only / Safekernel-level evidence— -k restricts to kernel messages, -p err to error priority and above. An OOM kill, an I/O error, or a read-only remount reframes the whole investigation, so check for them early.

$ journalctl -k -b -p err --no-pager; journalctl -b -p warning --no-pager | grep -iE 'oom|i/o error|read-only|segfault|timeout'

Aug 10 14:11:01 web01 kernel: Out of memory: Killed process 2214 (myapp) total-vm:8214512kB
Aug 10 14:11:01 web01 kernel: oom-kill: constraint=CONSTRAINT_MEMCG

Illustrative output

Useful filters worth memorising: -u for a unit, -b and -b -1 for this and the previous boot, -p for priority, --since and --until for a window, -k for the kernel, _PID= for a specific process, -f to follow, and -o json when you need to hand the result to a script.

Observation, inference, action

Keep the three apart in your notes and in the incident channel. Most bad calls under pressure come from an inference being relayed as an observation.

Observation: “df -i /var reports 100% inode use.”
Inference: “the mail spool is probably creating millions of small files.”
Action: “counting files under /var/spool to test that.”

Write observations with the command that produced them and the UTC time you ran it. When someone joins the call an hour later, that log is the handover.

Read-only / Safevalidate the timeline— Run this on every host that contributes to a cross-host timeline. A host that is thirty seconds out will make its logs appear to precede events that actually caused them.

$ timedatectl; chronyc tracking

               Local time: Mon 2026-08-10 14:35:12 UTC
System clock synchronized: yes
NTP service: active
Reference ID    : 0A0A0A0A (ntp1.example.com)
System time     : 0.000241 seconds fast of NTP time

Illustrative output

Negative evidence counts

“The backup job did not run” and “no configuration change was recorded in the window” are findings, and they should go in the bundle with the command that established them. Absence of a recorded change is the fact that rules out an entire class of hypothesis, and it is exactly the sort of thing that gets re-checked three times during a long incident because nobody wrote it down the first time.

Volatile first, durable second

The capture

Reading the journal with intent

Observation, inference, action

Negative evidence counts

Knowledge check