SAN and cluster storage - the enterprise shared filesystem · Linux

SAN (Storage Area Network) is the enterprise shared storage: a dedicated network for block storage, accessed by multiple hosts. This lesson covers how to use it for cluster storage.

What SAN is

A SAN is a dedicated network for block storage. Hosts see the SAN as local disks. The SAN provides:

Block-level access (like local disks).
Shared access (multiple hosts see the same LUNs).
Replication and HA (the SAN replicates).
Performance (Fibre Channel, NVMe over Fabrics).

Protocols

Fibre Channel (FC): dedicated fibre, high throughput, low latency. The traditional SAN.
iSCSI: SCSI over TCP/IP. Cheaper than FC, uses existing network.
NVMe over Fabrics (NVMe-oF): NVMe over FC or TCP. Lowest latency, highest throughput.
FCoE: Fibre Channel over Ethernet. Less common.

For most clusters, iSCSI is sufficient. For high performance, FC or NVMe-oF.

Configure iSCSI

# Substitute your own values before running:
ISCSI_SERVER=192.0.2.20
TARGET_NAME=iqn.2026-01.com.example:storage.lun0

# Install
sudo apt install open-iscsi

# Discover targets
sudo iscsiadm -m discovery -t sendtargets -p "$ISCSI_SERVER"

# Login
sudo iscsiadm -m node -T "$TARGET_NAME" -p "$ISCSI_SERVER" -l

The iSCSI device appears as a local disk:

lsblk
# sda or sdb: the new iSCSI LUN

Configure multipath

For HA, configure multipath (multiple paths to the LUN):

# Install
sudo apt install multipath-tools

# Configure
sudo tee /etc/multipath.conf <<'EOF'
defaults {
    user_friendly_names  yes
    find_multipaths      yes
    path_grouping_policy group_by_prio   # correct for ALUA arrays
    path_checker         tur
    prio                 alua
    failback             immediate
    no_path_retry        queue
}

# Blacklist the LOCAL boot disk by its WWID. Never blacklist a
# devnode pattern that matches all SCSI disks - those are the
# disks multipath exists to aggregate.
blacklist {
    wwid "3600508b1001c<local-boot-disk-wwid>"
}
EOF

sudo systemctl enable --now multipathd

# Verify BEFORE using the device.
sudo multipath -ll            # must list your LUN with 2+ paths, all "active ready"
sudo multipathd show paths    # per-path state
sudo multipathd show blacklist
ls -l /dev/mapper/            # with user_friendly_names: mpatha, mpathb, ...

Multipath aggregates multiple paths to the same LUN. If one path fails, traffic continues on the others.

no_path_retry queue is a deliberate trade-off, not a default to copy blindly. When every path is gone it holds I/O in the queue instead of returning errors, so a brief fabric outage is survived rather than turned into a read-only filesystem. The cost is that processes block in D state for as long as the outage lasts, and in a Pacemaker cluster a blocked monitor operation times out and gets the node fenced. Use queue where a short fabric blip is the expected failure and the workload can wait; use a bounded value such as no_path_retry 12 where the cluster must be allowed to notice the failure and fail the service over.

The blacklist is inverted by default - do not copy the un-negated form

The built-in default blacklist is the negated expression "!^(sd[a-z]|dasd[a-z]|nvme[0-9])". The leading ! inverts the match, so it excludes everything that is not a SCSI, DASD or NVMe disk. Drop the ! and you get devnode "^sd[a-z]", which means the opposite: every SCSI disk is excluded, including the iSCSI LUNs this section just attached as sda and sdb.

That mistake is silent. multipathd starts, systemctl is-active multipathd says active, and any monitoring check that only asks “is multipathd running?” passes. But no map is built, /dev/mapper/mpatha never appears, and the LUN is consumed through a single raw /dev/sdX path. The first HBA, switch-port or portal failure produces immediate I/O errors - the exact outage multipath was deployed to prevent.

The pattern is also incomplete on its own terms: ^sd[a-z] does not match sdaa and beyond, so it behaves differently on a host with more than 26 SCSI devices. Exclude specific devices by WWID, which is stable, rather than by devnode.

Always prove the config before building on it: multipath -ll must list the LUN and its paths, and multipathd show blacklist must not show your SAN devices.

Stop LVM claiming the individual paths

Multipath is only half the job. LVM scans every block device it can see, so it will find the same PV signature on /dev/sda, /dev/sdb and /dev/mapper/mpatha. LVM then picks one - often a raw single path - and you lose the redundancy you just configured, while pvs prints duplicate PV warnings.

Restrict LVM to the multipath device in /etc/lvm/lvm.conf:

devices {
    # Accept the multipath maps and the local boot disk. Reject
    # everything else, which is what excludes the raw SAN paths.
    global_filter = [ "a|/dev/mapper/mpath.*|", "a|^/dev/sda[0-9]*$|", "r|.*|" ]
    multipath_component_detection = 1
}

Replace sda with whatever your local boot disk actually is - if you filter it out by accident, the root VG stops activating at boot. Then rebuild the metadata cache and confirm each PV is reported on its mpath device:

sudo vgscan --cache
sudo pvs -o pv_name,vg_name    # must show /dev/mapper/mpatha, not /dev/sdb

Use the SAN in a cluster

For cluster storage, the SAN presents the same LUN to every node. That is the point of a SAN, and it is also the danger. The LUN is a dumb block device. It has no idea how many nodes are writing to it, and it will not stop you.

Never mkfs.xfs or mkfs.ext4 on a LUN more than one node will mount

XFS and ext4 are single-node filesystems. Each node keeps its own page cache, its own in-memory inode and allocation group state, and its own journal. Nothing coordinates them. Mount one on two nodes read-write and the first concurrent metadata write corrupts it. xfs_repair and fsck cannot reconstruct the result. Recovery is restore from backup, for every workload on the LUN.

Plain LVM has the same problem one layer down. A vgcreate without --shared produces a volume group that no lock manager protects, so two nodes running lvextend at the same time corrupt the VG metadata itself.

On shared block storage there are exactly two safe designs:

Concurrent: a cluster-aware filesystem (GFS2, OCFS2) on a shared VG, with lvmlockd, the DLM, and working fencing.
Single-writer: a normal XFS volume whose activation and mount are owned by one Pacemaker resource group at a time. Only one node ever mounts it, so XFS is correct.

Anything else is the corruption case described in the lesson on two nodes mounting the same block device.

Fencing is a prerequisite here, not a follow-up task. The DLM will not grant locks while the cluster is unable to fence a node it has lost, so a GFS2 mount on an unfenced cluster hangs at the first membership change. Configure STONITH and prove it fences before the filesystem exists.

Concurrent access: shared VG plus GFS2

Read the step comments before running anything. Some steps run on every node, some run once from a single node, and doing a “one node only” step twice is how the VG metadata gets corrupted.

Cluster-wide riskshared LUN: shared VG + GFS2— Affects every node. A mistake here is whole-cluster data loss, not a single-host outage.

# 0. Fencing first. The DLM refuses to hand out locks without it.
sudo pcs property set stonith-enabled=true
sudo pcs stonith status

# 1. EVERY node: log in to the LUN, confirm multipath sees it.
sudo iscsiadm -m discovery -t sendtargets -p <iscsi-server>
sudo iscsiadm -m node -T <target-name> -p <iscsi-server> -l
sudo multipath -ll                  # expect the same mpatha on every node

# 2. EVERY node: set use_lvmlockd = 1 in /etc/lvm/lvm.conf, then let
#    Pacemaker own the lock managers as a cloned group so it starts
#    and stops them in the right order.
sudo pcs resource create dlm ocf:pacemaker:controld \
op monitor interval=30s on-fail=fence --group locking
sudo pcs resource create lvmlockd ocf:heartbeat:lvmlockd \
op monitor interval=30s on-fail=fence --group locking
sudo pcs resource clone locking interleave=true

# 3. ONE node only: create the VG as a SHARED VG.
sudo pvcreate /dev/mapper/mpatha
sudo vgcreate --shared vg_cluster /dev/mapper/mpatha
sudo lvcreate -L 100G -n shared vg_cluster

# 4. EVERY other node: start the lock space for the new shared VG.
sudo vgchange --lockstart vg_cluster

# 5. ONE node only: a real cluster filesystem.
#    -t is clustername:fsname, -j is one journal per node.
sudo mkfs.gfs2 -p lock_dlm -t mycluster:shared -j 3 /dev/vg_cluster/shared

# 6. EVERY node: activate the LV in SHARED mode, then mount.
sudo lvchange --activate sy vg_cluster/shared
sudo mount /dev/vg_cluster/shared /mnt/shared

Step 6 is the manual form, shown so you can see what the cluster is doing. In production Pacemaker owns it: an ocf:heartbeat:LVM-activate resource with vg_access_mode=lvmlockd and activation_mode=shared, then an ocf:heartbeat:Filesystem resource, both cloned and ordered after the locking clone. The volume never appears in /etc/fstab, because a node that mounts it at boot mounts it without the cluster’s permission.

Single-writer: the simpler answer

Most workloads do not need concurrent access. They need the volume to follow the service. That is cheaper to run and has a far smaller failure surface, and it is the pattern to reach for first.

# EVERY node: set system_id_source = "uname" in /etc/lvm/lvm.conf.
# LVM then stamps the VG with the owning node's ID and refuses to
# activate it anywhere else, so a second node cannot mount it by hand.

# ONE node only:
sudo pvcreate /dev/mapper/mpatha
sudo vgcreate vg_app /dev/mapper/mpatha
sudo lvcreate -L 100G -n data vg_app
sudo mkfs.xfs /dev/vg_app/data          # correct here: one mount, ever

# Pacemaker owns activation and the mount. Not a clone, so exactly
# one node runs the group at a time. Not in /etc/fstab.
sudo pcs resource create app_lvm ocf:heartbeat:LVM-activate \
  vgname=vg_app vg_access_mode=system_id --group app
sudo pcs resource create app_fs ocf:heartbeat:Filesystem \
  device=/dev/vg_app/data directory=/mnt/app fstype=xfs --group app

mkfs.xfs is correct in this second recipe and catastrophic in the first. The filesystem is not what changed. The number of nodes allowed to mount it is.

Cluster filesystems, for reference

GFS2: Red Hat’s cluster filesystem. Needs the DLM, fencing, and one journal per node. Grows online; shrinking is not supported.
OCFS2: Oracle’s cluster filesystem, the usual choice under Oracle RAC.
CephFS: distributed POSIX filesystem. It is its own storage layer, so there is no shared LUN to get wrong.

None of these removes the fencing requirement. A cluster filesystem coordinates locks between nodes it can talk to. It cannot coordinate with a node that has stopped answering, which is precisely the case fencing exists to handle.

Production considerations

SAN is a single point of failure: use a redundant SAN with multipath.
Network dependency: the SAN is its own network. Failure of the SAN fabric breaks all cluster nodes.
Performance: SAN throughput depends on the FC, iSCSI, or NVMe-oF transport.
Cost: SAN is expensive, especially FC.
Shared LUNs need a lock manager: lvmlockd for the VG and a cluster filesystem for the data, or a single-writer design. The SAN enforces neither.
No shared LUN in /etc/fstab: a boot-time mount bypasses the cluster and is the usual root cause of a dual-mount corruption.