Lab: Process investigation end-to-end

This lab trains the muscle memory of “the service is slow” or “the service is stuck” investigations. Three scenarios, one discipline: gather evidence, identify the subsystem, act.

Objective

By the end of this lab, you can:

Find a service’s PID from its name.
Trace the service’s process tree, threads, and resource use.
Read /proc/<pid> for forensic data.
Choose the right signal and order to shut it down.

Architecture

flowchart LR
  SVC["Service"]
  PID["Find PID<br/>pgrep -f"]
  TREE["pstree<br/>process ancestry"]
  PROC["/proc/<pid>/status<br/>memory, state, threads"]
  IOCG["/proc/<pid>/io<br/>cgroup memory.current<br/>CPU time"]
  ACT["Signal<br/>TERM → wait → KILL"]
  SVC --> PID --> TREE --> PROC --> IOCG --> ACT

Requirements

A Linux host with at least one long-running service (sshd, cron, nginx if installed).
Root or sudo access.
pstree installed (apt install psmisc).

Scenario A — service is slow

A colleague reports that myapp is responding slowly. You have five minutes.

Task 1: Find the PID

pgrep -f myapp
pgrep -fl myapp   # show command lines

pgrep -f matches the full command line — appropriate when the process name is generic. Note the PIDs.

Task 2: Trace the tree

pstree -p $(pgrep -f myapp | head -1)

This shows the process tree under the myapp master. The tree helps you see workers, children, and unrelated processes that happen to match.

Task 3: Read the state

top -bn1 -p $(pgrep -f myapp | head -1)
ps -o pid,ppid,stat,pcpu,pmem,rss,etime,comm -p $(pgrep -f myapp | head -1)

Identify the state (R, S, D, Z). A D state is a stuck-kernel- syscall problem — usually storage. An R state with high CPU is a hot-loop problem. A S state with high memory is a leak.

Task 4: Read /proc/<pid>

PID=$(pgrep -f myapp | head -1)
cat /proc/$PID/status
cat /proc/$PID/io
ls -l /proc/$PID/fd/ | head

status shows VmRSS, VmSize, threads, capabilities. io shows read/write bytes and syscalls. fd/ shows open file descriptors — a stuck process often has a hung network socket or a held lock.

Task 5: Read the cgroup

CGROUP=$(cat /proc/$PID/cgroup)
echo "$CGROUP"
cat /sys/fs/cgroup/system.slice/myapp.service/memory.current
cat /sys/fs/cgroup/system.slice/myapp.service/memory.max

The cgroup view shows memory use against any systemd-applied limit. If memory.current is approaching memory.max, the service is memory-throttled.

Scenario B — service is stuck

A different colleague reports that myapp is not responding at all.

Task 6: Diagnose stuck state

Repeat Tasks 1-5. Note:

What state is the process in?
What is it waiting for? cat /proc/$PID/wchan shows the kernel function the process is sleeping in. D state with a non-empty wchan is a kernel-side problem; D state with no wchan is a short syscall in flight.
Is the process receiving CPU? cat /proc/$PID/stat | awk '{print "utime="$14, "stime="$15}'.

Task 7: Try the disciplined shutdown

PID=$(pgrep -f myapp | head -1)
echo "sending SIGTERM to $PID"
kill -TERM $PID
sleep 10
ps -o pid,stat,comm -p $PID   # confirm exit or zombie

Wait 10 seconds. If the process is gone, you finished. If it is a zombie, the parent has not reaped it; find the parent and investigate the parent.

If the process is still running after 10 seconds:

echo "escalating to SIGKILL"
kill -KILL $PID
sleep 1
ps -o pid,stat,comm -p $PID

SIGKILL worked if the PID no longer exists. The process cannot catch SIGKILL — if the SIGKILL did not work, the process is in D state and the kernel is stuck on I/O.

Scenario C — collect evidence before escalation

Before any escalation (TERM, KILL, restart), collect evidence.

Task 8: Snapshot the state

PID=$(pgrep -f myapp | head -1)
OUTDIR=/tmp/myapp-investigation-$(date +%Y%m%d-%H%M%S)
mkdir -p $OUTDIR
ps -o pid,ppid,stat,pcpu,pmem,rss,etime,cmd -p $PID > $OUTDIR/ps.txt
top -bn1 -p $PID > $OUTDIR/top.txt
cat /proc/$PID/status > $OUTDIR/status.txt
cat /proc/$PID/io > $OUTDIR/io.txt
ls -l /proc/$PID/fd/ > $OUTDIR/fd.txt
cat /proc/$PID/cgroup > $OUTDIR/cgroup.txt
journalctl _COMM=myapp --since "1 hour ago" --no-pager > $OUTDIR/journal.txt
tar -czf $OUTDIR.tgz $OUTDIR
echo "evidence saved to $OUTDIR.tgz"

The evidence is the artefact the post-incident review uses to answer “what happened?”. Without it, the incident becomes “someone restarted myapp and we don’t know why”.

Validation

The lab is complete when:

You can find any service’s PID from its name in under 30 seconds.
You can read the service’s state, memory, threads, and cgroup without resorting to kill -9 first.
You have demonstrated TERM → wait → KILL escalation.
You have produced a saved evidence directory that an incident responder could open tomorrow.

Expected outcome

A repeatable process-investigation workflow that gathers evidence before acting and escalates signals in the production order.

Troubleshooting

pgrep -f returns nothing — the service may be running under a different name. Try pgrep -a to see the full command line, or systemctl status myapp to see the systemd unit.
The process is in D state and won’t die — that is the expected behaviour for D state; document it, escalate to the application owner, plan a controlled reboot; do not keep sending SIGKILL.
The cgroup view shows memory limit exceeded — the service is OOM-killed or throttled. dmesg | grep -i oom confirms OOM kills; check the systemd unit for MemoryMax and adjust.

Cleanup

The lab is non-destructive if you followed the TERM → wait → KILL discipline. Killed processes do not leave artifacts. The evidence directory is in /tmp and will be cleaned by reboots.

rm -rf /tmp/myapp-investigation-*

What you learned

You can now triage a stuck or slow service by reading process state without restarting it first. The discipline is to gather evidence before acting, identify the subsystem, escalate signals in order, and save the investigation for post-incident review.

Objectives

Prerequisites