This lab trains the muscle memory of using journalctl to investigate a production incident. The scenario is contrived but realistic — every diagnostic step uses real journalctl capabilities.
Objective
By the end of this lab, you can:
- Read the journal with the right combination of filters for each step of an investigation.
- Cross-reference journal entries with traditional log files.
- Save evidence for post-incident review.
- Identify the difference between application-level and kernel-level incidents.
Architecture
flowchart LR
S[Symptom]
J[journalctl]
K[Kernel dmesg]
L[Log files]
V[Verify hypothesis]
E[Save evidence]
S --> J --> K --> L --> V --> ERequirements
- A disposable Linux host with systemd-journald and rsyslog.
- Root or sudo access.
- Enough free space in
/var/logfor a persistent journal (a few hundred MB is ample for this lab).
Task 0 creates everything else the lab investigates. Do not
skip it: myapp is a fixture built here, not a service that
already exists on your host.
Scenario
A colleague reports: “myapp is failing every morning around 03:00 and restarting. The service is up during business hours.”
You have 30 minutes. Walk the investigation from first symptom to root cause using journalctl, and save the evidence.
Tasks
Task 0: Build the subject
Two things have to be true before any of the later tasks return data, and both are worth understanding rather than pasting past.
Make the journal persistent. By default on many
distributions the journal lives in /run/log/journal, which
is tmpfs and is emptied at every boot. Every task below asks
for --since "2 days ago", and on a volatile journal the
answer is always “nothing” — which looks exactly like “the
service never logged anything”. This is not a lab detail; it
is the single most common reason a real post-incident
investigation finds no evidence of the reboot that caused it.
sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald
journalctl --header | grep -i 'storage\|File path' # expect a /var/log/journal pathCreate the failing service. A small script that logs a few
normal lines and then exits non-zero, with Restart=on-failure
so systemd restarts it the way the scenario describes:
sudo tee /usr/local/bin/myapp-sim.sh >/dev/null <<'EOF'
#!/usr/bin/env bash
set -uo pipefail
echo "myapp starting, pid $$"
echo "connecting to database"
sleep 2
echo "ERROR: connection pool exhausted" >&2
exit 1
EOF
sudo chmod 0755 /usr/local/bin/myapp-sim.sh
sudo tee /etc/systemd/system/myapp.service >/dev/null <<'EOF'
[Unit]
Description=MyApp (lab fixture - not a real service)
[Service]
Type=simple
ExecStart=/usr/local/bin/myapp-sim.sh
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
EOF
sudo systemd-analyze verify /etc/systemd/system/myapp.service
sudo systemctl daemon-reload
sudo systemctl start myappLet it cycle for a minute or two so there is a repeating
failure pattern to find. Restart=on-failure with
RestartSec=5 will loop until systemd’s default start-limit
kicks in and leaves the unit failed — which is itself the
state Task 1 asks you to read.
Seed some free-standing entries. The journal timestamps entries when it receives them, so you cannot write into the past — the “03:00” of the scenario is whatever hour you are running this in, and Task 4 tells you to adjust the window accordingly.
for i in 1 2 3; do
echo "myapp: nightly batch started (marker $i)" | systemd-cat -t myapp -p info
echo "myapp: batch failed, connection pool exhausted" | systemd-cat -t myapp -p err
done
journalctl -t myapp -n 10 --no-pagerNote the filter: -t myapp, not -u myapp. systemd-cat
entries carry a syslog identifier but no _SYSTEMD_UNIT, so
journalctl -u myapp will not show them. That is worth
knowing before you meet it in an incident — a service that
logs through a wrapper, a cron job, or anything not started by
systemd is invisible to -u and you will conclude it produced
no logs at all. journalctl -t <tag> and
journalctl _COMM=<binary> are the fallbacks.
Task 1: Start the investigation
systemctl status myappRead the state. Note:
- Active state (active/failed).
- Sub-state (running/dead/exited).
- Main PID and whether it has changed.
- Memory and CPU since the service started.
- Last few log lines from the embedded journal excerpt.
Task 2: Pull the journal entries for the service
journalctl -u myapp --since "2 days ago" --no-pager | wc -l
journalctl -u myapp --since "2 days ago" --no-pager | headHow many entries? What is the overall pattern?
Task 3: Filter to error priority
journalctl -u myapp -p err --since "2 days ago" --no-pagerAre there errors? What patterns repeat? Note the timestamps.
Task 4: Look at the failure window
# Substitute the hour you actually ran Task 0 in - the fixture's
# entries are timestamped now, not at 03:00.
journalctl -u myapp --since "03:00" --until "04:00" --no-pager
# For the fixture, this is the equivalent window:
journalctl -u myapp --since "30 min ago" --no-pagerNarrowing to the failure window is the step that turns a wall of entries into a readable sequence. What preceded the failure? In a real 03:00 incident that question usually answers itself — a backup job, a certificate renewal, a logrotate postrotate hook that restarts the service.
Task 5: Check the kernel
journalctl -k --since "2 days ago" --no-pager | grep -i 'oom\|kill\|error'
dmesg | grep -i 'oom\|kill\|error'Were there OOM kills? Driver errors? Hardware issues? The kernel journal sometimes explains application-level symptoms.
Task 6: Cross-reference with syslog
tail -50 /var/log/syslog | grep myapp
tail -50 /var/log/auth.logThe journal has more structure but the text files are sometimes quicker to scan. Use both.
Task 7: Save the evidence
OUTDIR=/tmp/myapp-investigation-$(date +%Y%m%d-%H%M%S)
mkdir -p $OUTDIR
journalctl -u myapp --since "2 days ago" -o json --no-pager > $OUTDIR/journal.json
journalctl -u myapp --since "2 days ago" --no-pager > $OUTDIR/journal.txt
journalctl -k --since "2 days ago" --no-pager > $OUTDIR/kernel.txt
cp /var/log/syslog $OUTDIR/syslog.txt
cp /var/log/auth.log $OUTDIR/authlog.txt
tar -czf $OUTDIR.tgz $OUTDIR
echo "evidence: $OUTDIR.tgz"The evidence is what an incident responder uses tomorrow. Save the journal both as text (for reading) and as JSON (for jq queries).
Task 8: Build a hypothesis
Based on the evidence, write a one-paragraph hypothesis:
- What is failing and why?
- What evidence supports this hypothesis?
- What evidence would refute it?
- What is the next step to confirm or refute?
Share the hypothesis with a colleague or the runbook. The discipline of writing a hypothesis before acting is what separates a senior responder from one who restarts everything.
Validation
The lab is complete when:
- You can produce the journal entries that show the failure pattern.
- You have saved evidence in both text and JSON format.
- You have a written hypothesis backed by specific journal entries.
- You have identified whether the failure is application-level or kernel-level.
Expected outcome
A complete diagnostic trace: the journal entries that reveal the failure, the cross-references to syslog that confirm it, the kernel entries that explain it, and a written hypothesis ready for peer review.
Troubleshooting
- No entries for the service — the service may be writing to
stdout only and not via journal. Check the unit file for
StandardOutput=journalor similar. Some services write to files only; check/var/log/<app>/. - Time filters return nothing — confirm the timezone. The journal uses local time; cron and logrotate may use UTC.
- JSON export is too large to grep — pipe to jq:
journalctl -u myapp -o json --since "1 hour ago" | jq -r 'select(.PRIORITY == "3") | .MESSAGE'
Cleanup
The investigation artifacts are in /tmp and will be removed by a reboot:
rm -rf /tmp/myapp-investigation-*Remove the fixture service, which is still restart-looping:
sudo systemctl stop myapp
sudo systemctl disable myapp 2>/dev/null || true
sudo rm -f /etc/systemd/system/myapp.service /usr/local/bin/myapp-sim.sh
sudo systemctl daemon-reload
systemctl status myapp # expect: Unit myapp.service could not be foundLeave the persistent journal in place. It is not lab
scaffolding — a host whose journal does not survive a reboot
cannot be investigated after the reboot that mattered, so
/var/log/journal belongs in your base image.
What you learned
You can now use journalctl as the primary diagnostic tool for every systemd-managed host. The discipline is to combine filters, export in both text and JSON, cross-reference with traditional syslog when needed, and write a hypothesis before acting.