Runbook: Investigate a systemd service that will not start

A service that will not start is the most common ticket in any Linux estate. It is also the one most often closed without a cause, because systemctl restart sometimes works and everybody moves on. This runbook finds the cause first.

When to use this runbook

• A unit is failed, or activating and never reaching active.
• A unit is flapping: it starts, dies, and restarts.
• A unit reports active but the service does not answer.

Do not use this runbook for a unit managed by Pacemaker. Restarting a cluster resource by hand fights the cluster and can trigger a fence. Use the Pacemaker runbook.

Inputs

• The exact unit name.
• What changed: a deploy, a package upgrade, a config management run, a certificate renewal, a reboot.
• Whether the service ever worked on this host.

Step 1: Read the state, including the exit code

systemctl status nginx --no-pager -l
systemctl show nginx -p Result -p ExecMainStatus -p NRestarts -p ActiveState

Three fields decide where you go next.

• Result — exit-code means the process ran and returned non-zero. timeout means it never signalled readiness. oom-kill means the kernel killed it. protocol usually means the wrong Type=.
• ExecMainStatus — the exit status. Most daemons use their own codes; 203 is systemd’s own “could not execute” and almost always means a bad ExecStart path or a missing execute bit.
• NRestarts — if this is climbing, you have a crash loop, not a one-off failure.

Step 2: Read the actual error

systemctl status shows the last ten lines. The cause is usually above them.

# This boot, this unit, in full
journalctl -u nginx -b --no-pager

# Just the failure window, with priority filtering
journalctl -u nginx --since "20 min ago" -p warning --no-pager

# Follow the next start attempt live
journalctl -u nginx -f

If the unit is flapping, -f in one window and the restart in another is the fastest way to see the first error rather than the tenth.

Step 3: Classify before you fix

Match the journal against these five shapes. Each has a different fix and a different blast radius.

Fix one shape at a time. Stacking three speculative changes means you cannot say afterwards which one mattered, and the next occurrence starts from zero.

Step 4: Validate the application config

Every serious daemon has a checker. Use it before the restart, while the old process is still serving.

sudo nginx -t
sudo apachectl configtest
sudo sshd -t
sudo named-checkconf
sudo haproxy -c -f /etc/haproxy/haproxy.cfg
sudo chronyd -Q -f /etc/chrony.conf

This is the difference between a five-second fix and an outage. A restart with a broken config kills the running process and then fails to start a new one. The checker tells you first, at no cost.

Step 5: Validate the unit file

systemd-analyze verify /etc/systemd/system/myapp.service

# See the effective unit after every drop-in is merged
systemctl cat myapp.service

# Which drop-ins exist and where they came from
systemd-delta --type=extended | grep -i myapp

The three faults that account for most of these:

• 203/EXEC. The ExecStart binary does not exist, is not executable, or has a bad interpreter line. Check with ls -l and head -1 on the script.
• More than one ExecStart. Only legal for Type=oneshot, where all lines run in sequence. For any other type the unit refuses to load and the job silently never runs.
• Wrong Type=. A daemon that forks under Type=simple looks like an instant exit. A daemon that stays in the foreground under Type=forking times out.

sudo systemctl daemon-reload
sudo systemctl restart myapp.service

Step 6: Follow the dependency chain

result 'dependency' means this unit never ran. Something it required failed first, and that is the unit to investigate.

systemctl list-dependencies myapp.service --all
systemctl --failed --no-pager
systemctl list-units --state=failed --no-pager

# Ordering is not the same as requirement - check both
systemctl show myapp.service -p Requires -p After -p Wants -p BindsTo

After= only orders; it does not require. A unit with After=network-online.target but no Wants= will start without waiting, and a service that binds an address before the address exists fails in a way that looks random.

Step 7: Permissions and mandatory access control

# AppArmor
sudo journalctl -k --since "20 min ago" | grep -i 'apparmor="DENIED"'
sudo aa-status | head

# SELinux
sudo ausearch -m avc -ts recent
sudo sealert -a /var/log/audit/audit.log 2>/dev/null | head -40

# Plain filesystem permissions on what the unit actually touches
sudo -u www-data test -r /etc/myapp/myapp.conf && echo readable || echo DENIED
namei -l /etc/myapp/myapp.conf

namei -l walks every component of the path. A daemon denied at /etc/myapp because the directory is 0700 looks identical in the journal to one denied at the file.

Step 8: Resource exhaustion

df -h /
df -i /
free -h
journalctl -k --since today | grep -i -E 'out of memory|oom-kill|killed process'

# What the unit was actually granted
systemctl show myapp -p LimitNOFILESoft -p LimitNOFILE -p MemoryMax -p MemoryCurrent
cat /proc/$(systemctl show myapp -p MainPID --value)/limits 2>/dev/null

Two exhaustion faults look like a service bug and are not:

• Full filesystem or exhausted inodes. The daemon cannot write its PID file, socket, or log. See the filesystem-full runbook.
• Too many open files. The soft RLIMIT_NOFILE is too low for the connection count. Fix it with LimitNOFILE= on the unit, not with nofile unlimited in limits.conf.

Step 9: Restart, and watch

# Reload if the daemon supports it - no dropped connections
sudo systemctl reload nginx

# Otherwise restart, and watch the result rather than assuming it
sudo systemctl restart nginx
systemctl is-active nginx
journalctl -u nginx -n 30 --no-pager

If the unit hit systemd’s start-limit throttle (start request repeated too quickly), the counter must be cleared or systemd will refuse to try again:

sudo systemctl reset-failed nginx
sudo systemctl start nginx

reset-failed clears a counter. It fixes nothing. If you find yourself running it more than once, the crash loop is the incident.

Step 10: Verify against the service, not the process

# Is it listening where you think
sudo ss -lntp | grep -E 'nginx|:80|:443'

# Does it answer
curl -sS -o /dev/null -w '%{http_code}\n' http://localhost/healthz

# Has it stayed up
systemctl show nginx -p NRestarts -p ActiveEnterTimestamp

Common patterns

Knowledge check