Break/Fix: the service fails on one boot in four and starts fine by hand · Break/Fix · Linux

Reported symptoms

api.service fails to start on roughly one boot in four across a fleet of 40 hosts. On the other three it is fine.
After a failed boot, sudo systemctl start api works immediately and the service runs for weeks.
The application team points out that the same binary, same config, runs on every host — including the ones that just failed, once restarted.
Six months ago someone added After=network-online.target to the unit. The failures continued.
Someone else added ExecStartPre=/bin/sleep 10. The failure rate fell to about one boot in twenty, which read as progress.
Last month a BIOS update cut POST time by 18 seconds and the failure rate went back up to one in four.

Evidence provided

$ systemctl status api.service
* api.service - Platform API
     Loaded: loaded (/etc/systemd/system/api.service; enabled)
     Active: failed (Result: exit-code) since Tue 2026-08-11 09:01:12 UTC
    Process: 918 ExecStart=/usr/local/bin/api --config /etc/api/api.yaml (code=exited, status=1/FAILURE)

$ sudo journalctl -u api -b --no-pager | tail -3
Aug 11 09:01:12 host07 api[918]: listening on 192.0.2.50:8443
Aug 11 09:01:12 host07 api[918]: fatal: bind 192.0.2.50:8443: cannot assign requested address
Aug 11 09:01:12 host07 systemd[1]: api.service: Failed with result 'exit-code'.

$ systemctl show -p After api.service
After=systemd-journald.socket sysinit.target network-online.target basic.target system.slice

$ systemctl show -p Wants -p Requires api.service
Requires=sysinit.target
Wants=

$ systemctl is-active network-online.target
inactive

$ systemctl is-enabled systemd-networkd-wait-online.service
disabled

$ systemd-analyze critical-chain api.service
The time when unit became active or started is printed after the "@" character.

api.service @9.412s
`-basic.target @9.401s
  `-sockets.target @9.399s
    `-dbus.socket @9.395s
      `-sysinit.target @9.388s

$ sudo journalctl -b -u systemd-networkd --no-pager | tail -2
Aug 11 09:01:12 host07 systemd-networkd[701]: eth0: DHCPv4 address 192.0.2.50/24 via 192.0.2.1
Aug 11 09:01:13 host07 systemd-networkd[701]: eth0: Gained carrier

Work the evidence before reading on

Three of those commands contradict the unit file, and one of them is the whole answer:

systemctl show -p After does list network-online.target. So the ordering line is present and systemd parsed it.
systemctl is-active network-online.target says inactive — on a host that is up, networked and serving traffic.
systemd-analyze critical-chain does not mention any network unit at all.

Before continuing, answer: what does After=X mean when X never starts? That single question is the difference between a fix that works and the fix that has been in this unit file for six months.

Root cause

1. `network.target` does not mean the network is up

This is the most consistently misread target in systemd. network.target is a synchronisation point meaning network management has been started. It is reached when systemd-networkd or NetworkManager has been launched — not when a link has carrier, not when DHCP has replied, and certainly not when an address is assigned.

network-online.target is the one that means “at least one interface is configured”, and it means that only because a wait-online service sits in front of it and blocks until the network manager reports an address.

2. `After=` without `Wants=` orders against nothing

This is the defect that survived six months of investigation.

After= is purely an ordering statement. It says: if both units are being started in the same transaction, start the other one first. It does not cause the other unit to be started.

network-online.target is not pulled in by anything by default. If no unit Wants= it, it is never part of the boot transaction, so:

systemd sees After=network-online.target
the target is not in the transaction
the ordering constraint is vacuously satisfied
api.service starts immediately

The unit file looks correct. systemctl show -p After confirms the line took effect. And it accomplishes nothing.

# What was written - a no-op
[Unit]
After=network-online.target

# What was needed - both halves
[Unit]
Wants=network-online.target
After=network-online.target

3. The wait-online service was disabled

Even with Wants=, the target only delays anything if something makes it wait. On this host systemd-networkd-wait-online.service is disabled, so network-online.target would activate instantly and the ordering would still be worthless. The pair has to be:

the correct wait-online service for the network manager actually in use, enabled, and
a unit that Wants= and is After= the target.

Mismatching those is common on hosts that migrated between NetworkManager and systemd-networkd — the old wait-online service stays enabled, reports success for a stack that is no longer managing the interface, and the target goes active while the address is still absent.

4. Why it is intermittent, and why the firmware update made it worse

The race is between two independent timelines:

systemd’s start-up path to api.service: sysinit, sockets, basic, then the unit. Affected by journal flush size, udev settle, disk speed, and every other unit in the transaction.
DHCP completing on eth0: affected by switch port forwarding delay, spanning-tree convergence, and DHCP server latency.

Neither is bounded. On most boots the second finishes first. On some it does not.

The BIOS update removed 18 seconds of POST, which does not change either timeline directly — but it moved the whole boot earlier relative to the switch port coming up, so the network side got relatively slower and the race tilted.

`sleep` in ExecStartPre is a guess with an expiry date

ExecStartPre=/bin/sleep 10 reduced the failure rate, which made it look like a partial fix. What it actually did was buy ten seconds of margin against a race with no upper bound — and then a firmware update spent that margin.

Every sleep in a unit file is a hard-coded assumption about how fast a machine is. Hardware refreshes, faster disks, and lighter images all erode it silently, and the failure returns months later looking like a new bug. If you cannot express a real readiness condition, at least make the wait conditional on the thing you are actually waiting for:

# Substitute your own values before running:
BIND_ADDR=192.0.2.50
for _ in $(seq 1 30); do
  ip -o addr show | grep -q "$BIND_ADDR" && exit 0
  sleep 1
done
exit 1

That is still a poll, but it fails loudly at 30 seconds instead of silently succeeding at the wrong moment.

Resolution

Establish which network manager actually owns the interface. networkctl status eth0 for systemd-networkd, nmcli device show eth0 for NetworkManager. Enabling the wrong wait-online service produces a target that goes active immediately and a fix that does not fix anything
Enable the matching wait-online service.
``sudo systemctl enable systemd-networkd-wait-online.service``
Add both halves of the dependency in a drop-in, so a package update does not overwrite it:
``sudo systemctl edit api.service``
``[Unit] Wants=network-online.target After=network-online.target``
Remove the sleep. Leaving it in hides whether the real fix works and adds ten seconds to every boot
Reload and confirm the rendered dependencies changed, not just the file on disk: sudo systemctl daemon-reload then systemctl show -p After -p Wants api.service
If the address is a floating VIP rather than a host address, wait-online cannot help — the VIP may legitimately live on another node. Enable non-local bind instead:
``echo 'net.ipv4.ip_nonlocal_bind = 1' | sudo tee /etc/sysctl.d/60-nonlocal-bind.conf sudo sysctl --system``
**Add Restart=on-failure with a RestartSec** as a backstop for whatever race remains, and accept that it is a backstop and not the fix

Verification

Both halves are present in the rendered unit. systemctl show -p After -p Wants api.service lists network-online.target under each. Checking the file instead of the rendered output is how this was missed the first time
The target actually activates. systemctl is-active network-online.target returns active on a booted host. If it returns inactive, nothing is waiting and the ordering is still against a unit that never runs
The wait-online service ran and blocked. systemd-analyze blame | grep wait-online shows a non-trivial duration on a host where DHCP takes time
The critical chain now includes the network. systemd-analyze critical-chain api.service lists network-online.target ahead of the service
Reboot at least five times. systemctl is-failed api.service must return no after every one. A one-in-four race has a 32% chance of surviving three reboots undetected, so a single clean boot is not evidence
Prove the failure mode is gone rather than hidden. Delay DHCP deliberately — shut the switch port for 20 seconds after power-on, or add a delay on the DHCP server — and confirm the service still comes up rather than failing faster

Ordering dependencies you can actually rely on

Ordering against a target is only as strong as whatever makes the target mean something. Stronger options, in rough order of preference:

Socket activation. api.socket with ListenStream=192.0.2.50:8443 and FreeBind=true makes systemd own the bind. FreeBind is the socket-unit equivalent of non-local bind and does not need a global sysctl.
RequiresMountsFor= when the dependency is a filesystem, which systemd resolves to the right .mount unit for you.
Wants= plus After= on a real service — for example a database or a message broker on the same host — rather than on a synthetic target.
ExecStartPre= that tests the condition and exits non-zero, so the failure is attributable instead of silent.

And one to avoid: Requires= on a unit that can legitimately be stopped. Requires= propagates stops, so stopping the dependency stops your service too, which is rarely what an operator expects at 03:00.

Prevention

Read After= and Wants= as two separate statements, because they are. Ordering says when if both run; the requirement dependency says whether it runs at all. After= alone against a target nothing pulls in is the single most common systemd ordering bug.
Verify with systemctl show -p After -p Wants, never by reading the unit file. The rendered view accounts for drop-ins, generators and defaults; the file does not.
Check systemctl is-active network-online.target on a booted host as a standing fleet assertion. inactive means every unit ordered after it is unordered.
Ban sleep from unit files in review. If a wait is unavoidable, poll the real condition with a bounded loop that fails loudly.
Any bug reported as intermittent needs a boot-loop or a restart-loop in acceptance. A race that fires 25% of the time will pass a single-run test three times out of four.

What if the service needs a mount rather than an address?

The same class of bug with a different target, and the fix is more specific.

Ordering a service After=local-fs.target is the mount equivalent of After=network.target: it is coarse, it covers every local filesystem including ones you do not care about, and it says nothing about a network filesystem. Instead name the path:

[Unit]
RequiresMountsFor=/srv/data

systemd resolves that to the specific .mount unit for /srv/data, adds both the ordering and the requirement, and follows the path through nested mounts. It also keeps working if the mount later moves from a local disk to NFS, because the dependency is expressed on the path rather than on the storage type.

Confirm it took effect the same way as before:

systemctl show -p RequiresMountsFor -p After api.service

The failure mode when this is missing is nastier than a clean bind error: the service starts, the mount point is an empty directory, and the application writes to the root filesystem instead. See the fstab emergency-mode scenario for the other half of that story.

Reported symptoms

Evidence

Root cause

Remediation

Verification

Prevention

Reported symptoms

Evidence provided

Work the evidence before reading on

Root cause

1. network.target does not mean the network is up

2. After= without Wants= orders against nothing

3. The wait-online service was disabled

4. Why it is intermittent, and why the firmware update made it worse

Resolution

Verification

Prevention

1. `network.target` does not mean the network is up

2. `After=` without `Wants=` orders against nothing