Network filesystems in fstab - _netdev, automount, and the boot you cannot watch · Linux

Every local entry in /etc/fstab is a statement about this host. A network entry is a statement about a different host, evaluated at boot, on a console you are not sitting at. The filer being down at 03:00 is not a hypothetical - it is the normal case that the entry has to survive.

The whole lesson is one question: when the remote side is not there, does this host boot without it, or does it stop and wait for a human?

How systemd decides an entry is remote

systemd-fstab-generator sorts every fstab line into one of two targets. The classification decides when the mount is attempted and what happens when it fails.

Classification	Ordered before	Attempted	On failure without `nofail`
Local	`local-fs.target`	Early, as soon as the device unit appears	`local-fs.target` fails, boot drops to emergency
Remote	`remote-fs.target`	After `network-online.target`	`remote-fs.target` fails, boot drops to emergency

The generator classifies by filesystem type. nfs, nfs4, cifs, smb3, ceph and the other well-known network types are treated as remote automatically. Anything else is local - and that is where the trap is.

Read-only / Safewhat systemd thinks is remote— The authoritative answer to 'did my entry get classified as a network filesystem'. An NFS or CIFS entry that does not appear here was classified as local, which means systemd will try to mount it before the network exists. Run the same query against local-fs.target to see the other side.

$ systemctl list-dependencies remote-fs.target

remote-fs.target
* |-remote-fs-pre.target
* |-srv-share.mount
* |-srv-media.automount

Illustrative output

An ext4 filesystem on an iSCSI LUN is classified as local

The generator looks at the filesystem type column, not at where the device lives. An entry like

UUID=8f2c1b7a-... /srv/lun0 xfs defaults 0 2

says xfs, so systemd files it under local-fs.target and starts waiting for the device unit immediately - long before iscsid has logged into the portal and long before there is a network to log in over. The device never appears, the device job times out, local-fs.target fails, and the host lands in emergency mode.

man 5 systemd.mount names this case explicitly: _netdev is the option that reclassifies a device-based mount, such as an iSCSI LUN, as remote. The filesystem is local; the path to it is not, and only you know that.

UUID=8f2c1b7a-... /srv/lun0 xfs _netdev,nofail,x-systemd.device-timeout=120 0 2

The same applies to a filesystem on an FC LUN behind a slow fabric login, on a DRBD device that has not yet promoted, and on an NBD export.

The three timeouts, which are not the same timeout

Almost every “the boot hung for 90 seconds” question is really a question about which of these three fired.

Setting	What it bounds	Where it can be set
`x-systemd.device-timeout=`	How long to wait for the device to show up	fstab only - ignored in a `.mount` unit’s `Options=`
`x-systemd.mount-timeout=`	How long the `mount` command itself may run	fstab only - use `TimeoutSec=` in a unit
`DefaultTimeoutStartSec=`	The fallback for any job with no explicit timeout	`/etc/systemd/system.conf`

For an NFS mount the second one is what you want. The device already exists (there is no device), and what can hang is the mount.nfs call talking to a filer that is powered on, answers ICMP, and is not exporting anything.

Read-only / Safethe inherited default— Read the value rather than assuming 90 seconds - distributions change it and so do images. Any mount that does not set its own timeout inherits this one, both at boot and at shutdown. The stop timeout matters too: a hard NFS mount to a dead server can make shutdown wait this long per mount.

$ systemctl show -p DefaultTimeoutStartUSec -p DefaultTimeoutStopUSec

DefaultTimeoutStartUSec=1min 30s
DefaultTimeoutStopUSec=1min 30s

A network entry that degrades

Read-only / Safetwo entries that will not stop a boot— Both carry _netdev so they are ordered after the network, and nofail so a failure is logged and tolerated rather than failing remote-fs.target. The NFS entry bounds the mount attempt at 30 seconds instead of inheriting 90. The CIFS entry reads its password from a root-owned file rather than from fstab. Both use pass 0 - fsck never runs on a network filesystem.

$ grep -E 'nfs|cifs' /etc/fstab

nfs01.example.com:/export/data /srv/data nfs4 _netdev,nofail,x-systemd.mount-timeout=30,hard,noatime 0 0
//fs01.example.com/share /srv/share cifs _netdev,nofail,credentials=/etc/cifs/fs01.cred,vers=3.1.1,uid=1000,gid=1000 0 0

Read the option list as four separate decisions:

_netdev - order me after the network. Free, and wrong only if the filesystem is genuinely local.
nofail - my absence is not a reason to stop the boot. Correct for archives, media, scratch and reporting shares. Wrong for anything a service will write to; see below.
x-systemd.mount-timeout=30 - give up on the attempt after 30 seconds rather than 90. Shorter boots, and a failure you can see in the journal at a predictable time.
hard - once mounted, retry indefinitely rather than returning EIO into an application that may not check it.

Never put a password in /etc/fstab

/etc/fstab is mode 0644 on every distribution, and it has to be - mount, findmnt, df and half of userspace read it as an unprivileged user. A password= option in a CIFS entry is therefore readable by every account on the host, and it also ends up in /proc/mounts, in systemctl cat output, in the generated unit under /run/systemd/generator/, and in the journal.

Use a credentials file instead, owned by root and mode 0600:

# /etc/cifs/fs01.cred - chown root:root, chmod 0600
username=svc-backup
password=REPLACE_ME
domain=EXAMPLE

Then reference it with credentials=/etc/cifs/fs01.cred. Check the permissions after every image build, not just once:

sudo find /etc/cifs -type f ! -perm 600 -printf '%M %u %p\n'

The `nofail` decision is about what writes there

nofail is not a safety option to sprinkle everywhere. It changes which failure you get, and one of the two failures is much worse than the other.

Without the mount, /srv/data is still a directory - an empty one, on the root filesystem. A service that starts anyway writes into it happily, at root-filesystem speed, until the root filesystem fills. Then the filer comes back, the mount succeeds, and the data that was written underneath is shadowed: still consuming space, no longer visible under any path.

Configuration changelook underneath a mount— A bind mount of / exposes the root filesystem without any of the filesystems mounted over it, which is the only way to see files hidden beneath an active mount point. 4.2G here is data an application wrote while the NFS mount was missing. Unmount the bind view with 'sudo umount /mnt/rootview' when you are done - it is a view, not a copy, so deleting through it deletes the real files.

$ sudo mount --bind / /mnt/rootview && sudo du -sh /mnt/rootview/srv/data

4.2G	/mnt/rootview/srv/data

Illustrative output

The structural fix is not nofail and not its absence. It is to let the boot succeed and let the service carry the dependency:

# /etc/systemd/system/reporting.service.d/10-requires-mount.conf
[Unit]
RequiresMountsFor=/srv/data

Now the host boots, the mount failure is visible in systemctl --failed, and reporting.service refuses to start rather than writing 4 GB into the root filesystem. That is a far easier incident than an emergency prompt, and a far cheaper one than a shadowed directory nobody finds for a month.

What x-systemd.automount actually generates

x-systemd.automount makes the generator emit two units for one fstab line: the .mount you expect, and an .automount unit that is what actually starts at boot. The automount unit installs a kernel autofs trigger on the mount point and returns immediately. The real mount happens on the first process that touches the path, and that process blocks until it completes.

Three consequences worth knowing before you use it:

The boot always succeeds. Nothing is mounted, so nothing can fail. That is the point, and it is also why an automount entry can hide a dead filer for days.
The failure moves into the application. The first open() under the path blocks for the mount timeout and then returns an error to whatever code called it. Which is better if that code handles errors, and worse if it does not.
auto and noauto stop meaning anything. man 5 systemd.mount states that when x-systemd.automount is used, neither option has any effect.

Add x-systemd.idle-timeout= to unmount after a period of inactivity - useful for a share touched twice a day, and actively harmful for one under constant load, where it produces a mount/unmount cycle.

systemctl list-units --type=automount
systemctl status srv-media.automount

Validating a network entry without rebooting

You cannot fully test boot-time behaviour without a boot. You can eliminate every failure that is not about timing:

Read-only / Safesyntax and reachability— findmnt --verify parses every entry, checks that the target directory exists, and checks the filesystem type is known. It mounts nothing, so it is safe on a live host. It cannot tell you whether the filer will answer at boot - that is the timing class of failure, and no static check reaches it.

$ sudo findmnt --verify --verbose

/srv/data
[ ] target exists
[ ] FS type is nfs4
[ ] options: _netdev,nofail,x-systemd.mount-timeout=30,hard,noatime

0 parse errors, 0 errors, 0 warnings

Illustrative output

Configuration changeread what the generator produced— daemon-reload re-runs systemd-fstab-generator against the edited file. This is the step that turns an edit into something systemd knows about; without it, systemctl still describes the previous fstab. Confirm three things in the output: After=network-online.target proves _netdev was honoured, TimeoutSec reflects your mount-timeout, and the Options line matches what you wrote.

$ sudo systemctl daemon-reload && systemctl cat srv-data.mount

# /run/systemd/generator/srv-data.mount
# Automatically generated by systemd-fstab-generator

[Unit]
Documentation=man:fstab(5) man:systemd-fstab-generator(8)
SourcePath=/etc/fstab
After=network-online.target
Wants=network-online.target

[Mount]
What=nfs01.example.com:/export/data
Where=/srv/data
Type=nfs4
Options=_netdev,nofail,x-systemd.mount-timeout=30,hard,noatime
TimeoutSec=30s

Illustrative output

If After=network-online.target is missing, _netdev did not take effect and the entry is still classified as local. Fix that before rebooting, because the reboot is where you find out.

How systemd decides an entry is remote

The three timeouts, which are not the same timeout

A network entry that degrades

The nofail decision is about what writes there

Validating a network entry without rebooting

Knowledge check

The `nofail` decision is about what writes there