Lab: Troubleshooting drill - apply the methodology

This lab applies the ten-step troubleshooting loop to a sample scenario. The discipline is to follow every step, in order, from symptom to prevention.

The ten steps, from the loop lesson, are: define the symptom, determine the impact, check recent changes, collect evidence, identify the subsystem, form a hypothesis, test safely, restore service, find the root cause, prevent recurrence. The tasks below map onto them one for one.

Tasks

Task 1: Define the scenario

Scenario: A user reports “the application is slow”. The application is a Node.js service behind nginx. The database is PostgreSQL.

Task 2: Symptom and impact

• Symptom: “the application is slow”.
• Impact: “all users are affected; response time has increased from 100ms to 5s over the last 24 hours”.

Restate the symptom as a measurement before continuing. “Slow” is a complaint; “the /health endpoint returns in 5.2 s instead of 100 ms, first seen at 14:10 UTC” is a symptom.

date -u
curl -sS -o /dev/null \
  -w 'http=%{http_code} dns=%{time_namelookup}s connect=%{time_connect}s total=%{time_total}s\n' \
  http://localhost/health

Task 2b: Check recent changes

This is step 3 of the loop and the highest-yield question in incident response. Ask what moved before you ask what broke. Open the window comfortably before the first-seen time.

# Service and system events in the window
journalctl --since "24 hours ago" -p warning --no-pager | head -50
systemctl list-units --state=failed --no-pager

# Package changes - Debian family
grep ' install \| upgrade \| remove ' /var/log/dpkg.log | tail -20

# Package changes - RHEL family
dnf history list --reverse | tail -20
rpm -qa --last | head -20

# Configuration changes (etckeeper puts /etc under Git)
git -C /etc log --since "24 hours ago" --stat --no-pager | head -40
ls -lt /etc | head -10

# Configuration management runs
journalctl -u ansible-pull --since "24 hours ago" --no-pager | tail -20

Outside the host, check the change-management ticket queue, the deployment pipeline, and any infrastructure change in the window: firewall rules, DNS records, certificate rotations, storage maintenance.

Write the result as a time-ordered list, including the negative finding if there is one. “No package, configuration, or deployment change recorded between 13:00 and 14:10 UTC” is itself evidence, and it rules out a whole class of hypothesis.

Stop-rule: you have a time-ordered list of every change in the window spanning the symptom start. If a change lines up with the start time, carry it forward as your first hypothesis in Task 4.

Task 3: Evidence

# System metrics
top
iostat 1 5
vmstat 1 5
free -h

# Application logs
journalctl -u myapp --since "1 hour ago"

# Database logs
journalctl -u postgresql --since "1 hour ago"

Capture the data into one timestamped directory so it survives a restart and can be handed over:

INC=/var/tmp/inc-$(date -u +%Y%m%dT%H%M%SZ); mkdir -p "$INC"
journalctl -b --no-pager        > "$INC/journal.txt"
dmesg -T                        > "$INC/dmesg.txt"
systemctl status myapp -l --no-pager > "$INC/unit.txt"
ps auxf                         > "$INC/ps.txt"
ss -tanp                        > "$INC/sockets.txt"
df -h                           > "$INC/df.txt"
df -i                           > "$INC/df-inode.txt"
free -m                         > "$INC/free.txt"

Every command above is read-only. Run the capture before any restart: process state, open descriptors, and socket state are destroyed by a restart and are usually where the answer is.

Stop-rule: new commands stop changing your picture of the problem.

Task 3b: Identify the subsystem

This is step 5 of the loop. Narrow to one layer before you theorise. Sweep the layers rather than diving into the first one you suspect.

# CPU and run queue
vmstat 1 5
top -b -n1 | head -20

# Disk
iostat -xz 1 5

# Memory
free -m
journalctl -k -b -p err --no-pager | grep -i 'out of memory'

# Network
sar -n DEV 1 5
ss -s
ss -tan state established | wc -l

# Dependencies, tested directly rather than through the application
dig +short db.internal
time psql -h db.internal -U app -c 'SELECT 1'

Record which layer shows saturation or errors and which layers are clean.

Stop-rule: exactly one subsystem shows saturation or errors while the others look normal. If two look bad, work out which is downstream - a caller always looks unhealthy when the callee is unhealthy, so investigate the callee first.

Task 4: Hypothesis

Based on the evidence, form one falsifiable hypothesis in the form “if X is the cause, then Y will be observable”. Example:

• “If the database is slow because of lock contention on a table, then pg_stat_activity will show several sessions with a non-null wait_event_type of Lock.”

Stop-rule: you can write down an observation that would prove the hypothesis wrong. “The database is probably struggling” cannot be falsified, so it is not a hypothesis.

Task 5: Test

# Check for lock waits
SELECT datname, usename, state, wait_event, wait_event_type
FROM pg_stat_activity
WHERE state != 'idle';

If many queries are waiting on locks, the hypothesis is confirmed. If not, return to Task 4, not to Task 1: a refuted hypothesis invalidates the explanation, not the evidence.

Task 6: Restore service

Based on the test, apply the smallest change that restores service. Mitigation is not always the same as the fix - rolling back the change from Task 2b, failing over, or restarting a unit can all restore service before anyone knows the root cause. That is correct sequencing, not a shortcut. Example actions:

• “Add an index on the column that is being locked.”
• “Tune the application to use shorter transactions.”
• “Add connection pooling to reduce connection contention.”

Apply one fix at a time. Test after each.

Task 7: Verify

After the fix:

# Check that the performance is restored
time curl http://app/

# Monitor the database
iostat 1 5

# Compare to baseline

The fix should restore the baseline performance.

Task 8: Document

TROUBLESHOOTING REPORT
=======================
Date: 2026-08-09
Reporter: ops
Issue: Application slow; users affected

Symptom: Response time from 100ms to 5s over 24 hours
Impact: All users affected; SLA violated

Evidence:
- Database lock waits: many queries waiting on table X
- I/O: normal
- Memory: normal
- CPU: high on database

Hypothesis: Long-running transaction on table X
Test: Confirmed via pg_stat_activity
Fix: Tuned application to use shorter transactions

Verification: Response time back to 100ms

Root cause: Application code held a transaction open during
           external API call
Prevention: Code review checklist for transaction scope
           Monitoring on long-running transactions

Task 9: Prevent recurrence

Close both gaps, not one.

• Failure gap: why it broke. Example: the application held a transaction open across an external API call, so any slowdown at the third party became a database lock storm.
• Detection gap: why nobody knew for 24 hours. Example: there was no alert on transaction age, so the first signal was a user complaint.

Write one action per gap, and check each against all four tests: a named individual owner, a due date, a concrete change, and a stated verification.

FOLLOW-UP ACTIONS
=================
1. Failure gap - Ada moves the external API call outside the
   transaction in myapp by 24 Aug.
   Verify: staging load test with a 30s stubbed API delay shows
   no lock waits.

2. Detection gap - Bo adds an alert on transactions older than
   60s, routed to the platform rota, by 20 Aug.
   Verify: hold a transaction open in staging and confirm the
   alert fires and pages.

3. Loop feedback - Bo adds pg_stat_activity capture to the
   standard evidence bundle by 20 Aug.

Reject anything of the form “be more careful”, “add documentation” on its own, or “add an alert” with no threshold and no rota.

Validation

• The symptom is written as an observed value, an expected value, and a first-seen time.
• Task 2b produced a time-ordered change list, or an explicit negative finding.
• The evidence bundle exists under /var/tmp and was captured before any restart.
• Task 3b named one failing subsystem and stated why the others were ruled out.
• The hypothesis in Task 4 is falsifiable.
• Task 9 produced at least one action per gap, each with an owner, a date, and a verification.

Cleanup

# Remove the evidence bundle once it is attached to the ticket
rm -rf /var/tmp/inc-*

# Revert any change made in Task 6 on the disposable host
systemctl daemon-reload
systemctl status myapp --no-pager

Keep the troubleshooting report and the follow-up actions. Those are the deliverables; the bundle is scratch.