Failure: a service will not start, a host will not boot

“It will not start” is the most common ticket in Linux operations and the one most often answered with a guess. The methodology says form a falsifiable hypothesis. This lesson gives you the evidence that lets you form one in under a minute.

First split: never started, or started and died?

These are different failures with different evidence, and systemd tells you which one you have without any interpretation:

$ systemctl show myapp.service -p Result -p ExecMainStatus -p ExecMainCode -p NRestarts -p ActiveEnterTimestamp

Result=exit-code
ExecMainStatus=1
ExecMainCode=1
NRestarts=5
ActiveEnterTimestamp=Mon 2026-08-10 03:14:02 UTC

ActiveEnterTimestamp is the discriminator. If it is empty, the unit has never reached active state in this boot: the failure is at start. If it holds a timestamp and NRestarts is climbing, the service starts fine and then dies - a completely different investigation, usually resources or a crash on first request.

Result= names the class of failure:

The four failures behind Result=exit-code

Almost every start failure that reaches a queue is one of four things, and each has a fingerprint:

1. Config syntax - the daemon exits within milliseconds and prints a line number. Validate with the daemon own checker (nginx -t, sshd -t, named-checkconf) before blaming systemd.
2. Address already in use - the log says EADDRINUSE or bind(). Find the holder with ss -tlnp; it is frequently the previous instance that never exited.
3. Permission denied - the path exists and the user cannot read it. Check the unit User=, the file mode, and then the MAC layer: dmesg for AVC denials, or journalctl -t audit.
4. Missing dependency - the daemon starts before the thing it needs. The unit needs After= and Requires=, not a sleep.

The difference between a permission problem and a mandatory access control denial matters, because the file permissions look correct in the second case:

$ journalctl -b -k | grep -iE 'avc:|apparmor=' | tail

audit: type=1400 audit(1754870042.881:112): avc:  denied  { read } for  pid=2211 comm="nginx" name="app.conf" dev="dm-0" ino=131842 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:admin_home_t:s0 tclass=file permissive=0

ls -l shows mode 0644 and the operator concludes the permissions are fine. They are. The label is wrong.

When the host will not boot

The same split applies: did it fail before or after the root filesystem was mounted? The screen tells you which.

• An (initramfs) prompt means the root device was never found. The cause is upstream of the OS: a changed disk name or UUID, a missing storage module in the initramfs, an unassembled RAID array, an unactivated LVM volume group. blkid and lsblk from that prompt are your evidence.
• An “emergency mode” prompt with a working journal means the root filesystem mounted and something later failed. That is almost always /etc/fstab: a non-existent device, a wrong UUID or a filesystem that will not fsck.

From an emergency shell the evidence is one command:

# journalctl -b -p err --no-pager; systemctl --failed --no-pager

systemd[1]: Failed to mount /data.
systemd[1]: Dependency failed for Local File Systems.
UNIT              LOAD   ACTIVE SUB    DESCRIPTION
* data.mount        loaded failed failed /data

The failing unit names the failing mount, and the mount names the fstab line. Test a repaired fstab before the next reboot:

findmnt --verify --verbose    # parse and sanity-check /etc/fstab
mount -a                      # apply it now, in a state you can undo

Two habits remove most boot incidents entirely. Mount non-essential filesystems nofail so a missing data volume degrades the host instead of stopping the boot, and use UUID= rather than /dev/sdb1, which is not a stable name.

Knowledge check