Clustered filesystems concepts - POSIX over multiple hosts

A clustered filesystem provides shared POSIX semantics over multiple hosts. Multiple nodes can read and write the same files with proper locking. This lesson covers the options.

What a clustered filesystem is

A regular filesystem (ext4, xfs) is local. A clustered filesystem (GFS2, OCFS2, CephFS) is shared:

• Multiple nodes mount the same filesystem.
• POSIX semantics (locking, permissions, mtime).
• Distributed locking for consistency.
• Survives individual node failures.

The cluster filesystem needs:

• Shared storage (SAN, shared disks, or distributed storage).
• Cluster membership (DLM, corosync, or custom).
• Locking (distributed lock manager).

GFS2

GFS2 (Global File System 2) is Red Hat’s clustered filesystem:

• Requires Red Hat Cluster Suite or compatible.
• Shared storage required (typically SAN).
• POSIX-compliant.
• Online grow only, with gfs2_grow. GFS2 cannot be shrunk — there is no gfs2_shrink. Reclaiming space means building a smaller filesystem and migrating the data onto it.
• One journal per mounting node, fixed at mkfs time (mkfs.gfs2 -j N). Adding a node later needs gfs2_jadd first, or the new node simply cannot mount.
• Used with Pacemaker for cluster management.

For Red Hat shops, GFS2 is the standard.

OCFS2

OCFS2 (Oracle Cluster File System 2) is Oracle’s clustered filesystem:

• Used in Oracle RAC and other Oracle clusters.
• Shared storage required.
• POSIX-compliant.
• Simpler than GFS2 in some ways.

For Oracle shops, OCFS2 is the standard.

CephFS

CephFS is the distributed filesystem for Ceph:

• Distributed metadata (MDS).
• Distributed storage (OSD).
• POSIX-compliant.
• Scales to exabytes.
• No single point of failure.

CephFS is the modern choice for new deployments. It provides POSIX over a distributed object store.

Use cases

Production considerations

• Shared storage dependency: clustered filesystems require shared storage. The SAN or distributed storage is a single point of failure if not HA.
• Network dependency: cluster locking requires low-latency network.
• Locking contention: heavy concurrent writes can contend on locks.
• Recovery: a node that drops out of the cluster may need to fence before rejoining.

Knowledge check