Break/Fix: the server is slow and every resource looks busy

A server slow drill: identify the bottleneck resource using the USE methodology, drill in with specific tools, and fix the issue.

Tasks

Task 1: Apply the load

Check headroom first, then generate the load inside a scope with explicit limits:

# Precondition: confirm free space on the scratch filesystem.
df -h /var/tmp
mkdir -p /var/tmp/slowlab

sudo systemd-run --unit=slowlab --scope \
  -p MemoryMax=2G -p CPUQuota=200% \
  stress-ng --timeout 60s \
            --cpu 4 \
            --vm 1 --vm-bytes 1G \
            --hdd 2 --hdd-bytes 256M --temp-path /var/tmp/slowlab

Three details make this safe, and each one is a habit worth carrying into real work:

• --temp-path pins where the --hdd workers write. Without it they write into the current working directory. Started from /root or / on a small root filesystem, that is a self-inflicted full-filesystem outage.
• --hdd-bytes bounds how much each worker writes. Without a bound the workers write until the filesystem is full.
• systemd-run --scope with MemoryMax and CPUQuota keeps the load inside a cgroup, so the kernel throttles the drill rather than OOM-killing something you cared about. It also gives you one handle to stop everything.

Note that --io workers are omitted. They call sync(2) in a tight loop, which flushes the whole page cache repeatedly and distorts every other measurement you are about to take.

The server is now slow. The metrics show the bottleneck.

Task 2: USE methodology

For each resource, check:

• CPU: utilisation, saturation, errors.

  top
  mpstat 1 5

• Memory: utilisation, saturation, errors.

  free -h
  vmstat 1 5

• Disk: utilisation, saturation, errors.

  iostat -x 1 5

• Network: utilisation, saturation, errors.

  sar -n DEV 1 5        # per-interface throughput (sysstat)
  sar -n EDEV 1 5       # errors and drops - the saturation signal
  ss -s                 # socket totals, including retransmit-heavy states

  nicstat gives a tidier %ifutil column but is not in the Debian, Ubuntu or RHEL base repositories, so it is not available on a host you have just been handed. sar ships with sysstat, which this course installs, and /sys/class/net/<iface>/speed gives the link rate to compute utilisation against.

Task 3: Identify the bottleneck

Based on the USE metrics, identify the resource with the bottleneck. Judge on saturation signals, not on a single utilisation figure:

CPU      %us high AND /proc/pressure/cpu "some avg60" > 10   -> CPU-bound
         (100% CPU with no pressure is a healthy busy job)

Memory   MemAvailable < ~10% of MemTotal AND
         /proc/pressure/memory "some avg60" > 10 AND
         vmstat si/so non-zero                               -> memory-bound
         (high "used" or low "free" on its own means nothing)

Disk     await rising while IOPS is flat, plus tasks in D state
         (thresholds are media-dependent: >20ms is bad for a
          spinning disk, >2ms is bad for NVMe)               -> disk-bound

Network  drop RATE > 0, i.e. the delta of two samples of
         ip -s link or netstat -s - not a lifetime counter   -> network-bound

Collect the signals:

cat /proc/pressure/cpu /proc/pressure/memory /proc/pressure/io
grep -E 'MemTotal|MemAvailable' /proc/meminfo
vmstat 1 5                     # si/so columns
iostat -x 1 5                  # await against r/s + w/s
ip -s link show eth0; sleep 10; ip -s link show eth0

Task 4: Drill in

Use specific tools for the bottleneck:

• CPU: pidstat, perf top, strace.
• Memory: pmap, valgrind, slabtop.
• Disk: iotop, perf record, biosnoop.
• Network: tcpdump, ss, wireshark.

Task 5: Fix

Apply the fix for the bottleneck:

• CPU: profile, optimise code, scale out.
• Memory: identify leak, optimise, add RAM.
• Disk: faster disk, optimise I/O, add cache.
• Network: optimise, add bandwidth, fix congestion.

Task 6: Document

Fill in every field yourself. Each diagnosis line must name the signal you read and its value, not just the resource.

TROUBLESHOOTING REPORT: Server Slow
Date:
Symptom:
Impact:
Bottleneck:            (resource + the saturation signal that proves it)
Evidence:              (metric name, value, and where you read it)
Resources ruled out:   (and the signal that ruled each one out)
Root cause:
Fix:
Prevention:

A worked example of the evidence line, from a run of this drill:

Bottleneck:          CPU
Evidence:            /proc/pressure/cpu some avg60=41.7; mpstat %usr 98.2
Resources ruled out: memory - MemAvailable 26 GB of 32 GB, pressure 0.00,
                     vmstat si/so both 0 (page cache was high, which is normal)
                     disk - iostat await 0.4ms, no D-state tasks
                     network - ip -s link drop delta 0 over 10s

Task 7: Clean up

The drill is not finished until the host is back to normal.

sudo systemctl stop slowlab.scope 2>/dev/null
rm -rf /var/tmp/slowlab

# Confirm: no stress-ng left, space returned, pressure back to zero
pgrep -a stress-ng
df -h /var/tmp
cat /proc/pressure/cpu

Knowledge check