Two nodes mounting the same block device - the data corruption risk

Two nodes mounting the same block device without a clustered filesystem is a data corruption disaster waiting to happen. This lesson covers why and what to do instead.

The problem

A regular filesystem (ext4, xfs) is designed for one node to mount at a time. It uses local caches and locks:

• Page cache for reads and writes.
• Journal for atomicity.
• Local locks for concurrent processes.

When two nodes mount the same block device, each node has its own cache and lock state. They do not coordinate. The result:

• Writes from node A go to the device.
• Writes from node B go to the device.
• The device has inconsistent metadata.
• The filesystem is corrupted.
• Recovery requires fsck (or worse, restore from backup).

This is the classic split-brain problem for shared storage.

Why this happens

Developers and operators sometimes do this for “simplicity”:

• “Just mount the same iSCSI LUN on both nodes.”
• “It works for read; let’s see if it works for write.”
• “We’ll add a cluster FS later.”

It does not work. The first concurrent write from both nodes corrupts the filesystem. The “we’ll add a cluster FS later” is too late.

Safe patterns

For shared block storage across multiple nodes:

• Clustered filesystem (GFS2, OCFS2, CephFS): designed for shared access with proper locking.
• One writer, multiple readers: use a cluster manager (Pacemaker) to ensure only one node is the active writer. Others are read-only or inactive.
• Block-level replication (DRBD): one node is primary; others are replicas. Not concurrent.

The right pattern for the workload

When a shared mount is acceptable (and the exact options it needs)

Read-only, with recovery explicitly disabled

A plain mount -o ro is not safe, and this is the single most expensive misconception in this lesson.

Mounting read-only stops your writes. It does not stop the filesystem’s own writes. On mount:

• ext4 loads and replays the journal unless you pass noload (norecovery is the same option). The man page is explicit that the option means “don’t load the journal on mounting” - which tells you it is loaded by default.
• XFS runs log recovery unless you pass norecovery. XFS also refuses a second mount of the same UUID unless you pass nouuid.

Journal replay and log recovery are writes to the shared device. If the filesystem was not cleanly unmounted, two nodes mounting it read-only will both replay the log, concurrently, writing conflicting metadata to the same blocks. That is exactly the corruption this lesson exists to prevent - reached by the procedure people reach for when they are trying to be careful.

The safe form names the options:

# # ext4 - suppress journal replay
mount -o ro,noload /dev/mapper/shared /mnt/ro

# XFS - suppress log recovery; nouuid allows a second mount of the same UUID
mount -o ro,norecovery,nouuid /dev/mapper/shared /mnt/ro

Understand what you get in return. With recovery suppressed, an unclean filesystem shows missing, stale or inconsistent files - the XFS man page warns that some files or directories may not be accessible. You are reading a crash-consistent image, not a clean one. Do not draw conclusions about data loss from it.

The better move, when the goal is to inspect a failed primary’s data, is to take an LVM or array snapshot and mount the snapshot. The snapshot is yours alone, so recovery on it writes only to your copy.

Snapshots

Each node mounts its own read-only snapshot of the same base. The base is updated by one node at a time. This is the pattern the nouuid option exists for.

Single-writer enforced by the cluster manager

One Pacemaker Filesystem resource, fencing enabled, so only one node can ever hold the mount. Fencing is the part that makes it true; without it you have a convention, not a constraint.

A cluster-aware filesystem

GFS2 or OCFS2 for genuine concurrent read-write. For active-active concurrent access this is not an option, it is mandatory.

Recovery from corruption

If a filesystem is corrupted by this mistake:

1. Unmount from both nodes.
2. Run fsck (may or may not recover).
3. If fsck does not recover, restore from backup.
4. Add a clustered filesystem before remounting.

The cost of a corrupted filesystem is data loss. The cost of a clustered filesystem is setup time. Setup time is cheaper.

Knowledge check