Cluster-managed mounts - why shared storage never goes in fstab

Every safe design for shared storage rests on one guarantee: the right number of nodes have the filesystem mounted at any moment - one, for a single-writer volume, or all of them for a cluster filesystem, and never a number in between decided by accident. That guarantee is enforced by the cluster manager, and it only holds if the cluster manager is the only thing that mounts.

An /etc/fstab entry is a second thing that mounts.

What fstab actually does on a cluster node

fstab mounts at boot, from local-fs.target or remote-fs.target, driven by systemd. Consider the sequence after a node has been fenced and is powering back up:

1. The node boots. systemd reaches remote-fs.target.
2. The iSCSI session comes up, the LUN appears, the fstab entry mounts it.
3. Corosync starts. The node rejoins the membership.
4. Pacemaker starts and reads the CIB.
5. Pacemaker discovers the service is already running on the surviving node.

Between steps 2 and 5 the filesystem is mounted on two nodes. Nothing asked the cluster whether that was allowed, because at step 2 the cluster did not exist yet on this host. For a single-writer XFS or ext4 volume, the corruption has already happened by the time Pacemaker has an opinion.

If you want the mount point documented in fstab, the entry must be inert:

/dev/vg_app/data  /mnt/app  xfs  noauto,_netdev,nofail  0 0

noauto is the one that matters: it stops local-fs.target/remote-fs.target from mounting it. _netdev orders the unit after the network for the case where somebody mounts it by hand. nofail keeps a missing device from dropping the boot into emergency mode.

A noauto entry is still a loaded gun in a different way: it makes mount /mnt/app work as a bare command, which is exactly what a tired operator types at 03:00. Many teams simply omit the entry entirely and document the device in the runbook.

The Filesystem resource agent

Pacemaker mounts shared storage through ocf:heartbeat:Filesystem. The essential parameters are the same four you would pass to mount:

sudo pcs resource create app_fs ocf:heartbeat:Filesystem \
  device=/dev/vg_app/data \
  directory=/mnt/app \
  fstype=xfs \
  options=noatime \
  op monitor interval=20s timeout=40s on-fail=fence \
  op start timeout=60s \
  op stop timeout=100s on-fail=fence \
  --group app

Three things in that command are doing real work.

on-fail=fence on the monitor. If the monitor finds the filesystem is not mounted where the cluster believes it is mounted, the node’s state is unknown. Recovering by restarting the resource risks mounting it somewhere while the old mount still holds writes. Fencing the node is the safe answer.

on-fail=fence on the stop. This is the important one, and the next section is about why.

A stop timeout longer than the start timeout. Unmounting means flushing dirty pages, and a filesystem with gigabytes of dirty data and a slow array takes far longer to unmount than to mount. A 20-second stop timeout on a busy volume produces a “failed stop” that is really a “did not wait long enough”, and the node gets fenced for no reason.

Order it after the storage and before the application:

sudo pcs constraint order app_lvm then app_fs
sudo pcs constraint order app_fs then app_service
sudo pcs constraint colocation add app_fs with app_lvm INFINITY
sudo pcs constraint colocation add app_service with app_fs INFINITY

Inside a resource group, pcs supplies that ordering and colocation for you - the group starts members in order and stops them in reverse. The explicit constraints matter when the resources are not in one group, which is the usual case once a cloned cluster filesystem is involved.

Why a failed stop ends in a fence

Pacemaker’s contract is that it must know a resource is stopped before starting it elsewhere. For a filesystem, “stopped” means unmounted, which means the node is no longer able to write.

If the stop operation fails - the unmount returned busy, or timed out - Pacemaker does not know whether the node is still writing. It has exactly one way to find out that does not involve trusting the node: fence it. Power off the host, and the question is settled.

This is not a bug and it is not a rough edge. It is the only correct behaviour, and it is why on-fail=fence on the stop operation is the setting to leave alone.

Under Pacemaker, the force_unmount parameter of the Filesystem agent decides what the stop does about processes holding the mount. The default kills them, which is usually right for a volume the cluster owns. Setting it to false means the stop fails instead - and therefore that the node is fenced. Both are defensible; a silent lazy unmount is not either of them.

Things that mount behind the cluster’s back

fstab is the obvious one. These are the ones that survive the fstab cleanup:

systemd automount units. x-systemd.automount in fstab, or a .automount unit, installs a trigger: the first process to touch the path causes the mount. Nothing schedules that touch, so it can be a backup agent, a monitoring check, a shell completion, or df. The cluster is not consulted and often does not notice.

A .mount unit generated from a stale fstab. systemd generates units from fstab at boot and on daemon-reload. Deleting the fstab line without running sudo systemctl daemon-reload leaves the generated unit in place until the next boot.

Filesystem scanners. updatedb, antivirus scanners, backup agents and indexers walk mount points. On a cluster filesystem this is not a mounting problem but a locking one: a full-tree scan on GFS2 takes DLM locks across every node and can stall the application. Exclude cluster filesystems explicitly - for updatedb, by adding the filesystem type to PRUNEFS in /etc/updatedb.conf.

A helpful colleague. mount /mnt/app on the standby node, to “check whether the data is there”. This is the one no configuration prevents. It is prevented by the volume not being mountable by hand: LVM system_id, or LUN masking, or a shared VG that requires the lock manager.

Audit what the node would mount without the cluster:

$ systemctl list-units --type=mount --all --no-pager | grep -i mnt

  mnt-app.mount   loaded inactive dead   /mnt/app

Bringing an existing mount under cluster control

# 1. EVERY node: confirm which nodes currently have it mounted.
findmnt /mnt/app

# 2. The node that has it: stop the application, then unmount.
sudo systemctl stop app.service
sudo umount /mnt/app

# 3. EVERY node: make the entry inert, then tell systemd.
#    Edit /etc/fstab so the line reads noauto,_netdev,nofail
#    (or delete the line entirely).
sudo systemctl daemon-reload

# 4. EVERY node: confirm nothing will mount it automatically.
sudo systemctl list-units --type=mount --all --no-pager | grep mnt-app
findmnt /mnt/app                       # expect no output

# 5. ONE node: create the cluster resource and let it start.
sudo pcs resource create app_fs ocf:heartbeat:Filesystem \
device=/dev/vg_app/data directory=/mnt/app fstype=xfs \
op monitor interval=20s timeout=40s on-fail=fence \
op stop timeout=100s on-fail=fence --group app

# 6. Verify the cluster owns it and that it moves.
sudo pcs status
sudo pcs resource move app --wait=120
sudo pcs resource clear app

Step 6 is the acceptance test. A mount that the cluster can start but cannot move is not under cluster control yet.

Knowledge check