Skip to main content
RunBook Academy

CephLXXXVI · Kubernetes RBDKubernetes RBD

RBD volumes and access modes

Intermediate⏱ ~17 minkubectlrbd

What you'll learn

  • Explain the access modes RBD supports
  • Understand why ReadWriteMany is unsafe for RBD
  • Use block mode where appropriate
  • Choose between RBD and CephFS by access requirement

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Not yet marked complete on this device.

Why this matters in production

RBD provides a block device, and a block device with a filesystem on it can be mounted by exactly one writer. Attempting otherwise corrupts data.

The access modes

ModeRBD with filesystemRBD block mode
ReadWriteOncesupportedsupported
ReadOnlyManysupported with caresupported
ReadWriteOncePodsupported; strictestsupported
ReadWriteManyunsafesupported only for cluster-aware consumers
spec:
  accessModes: [ReadWriteOnce]

ReadWriteOnce means one node, and a pod on another node cannot mount it. ReadWriteOncePod is stricter: one pod, which prevents two pods on the same node mounting it simultaneously.

Why ReadWriteMany is unsafe

Two nodes mount the same ext4 filesystem
  → each has its own in-memory metadata cache
  → node A allocates a block; node B does not know
  → node B allocates the same block
  → the filesystem is corrupted

A non-clustered filesystem assumes it is the only writer. Ceph does not prevent two nodes mapping the same image — that is a valid operation for a cluster-aware consumer — so the protection must come from the access mode.

# RBD's own protection
IMAGE=vm-disk-01
rbd info k8s-rbd/${IMAGE} | grep features
rbd status k8s-rbd/${IMAGE}

exclusive-lock prevents two clients writing simultaneously, but it does so by transferring the lock, not by refusing — which does not protect a filesystem.

Block mode

spec:
  volumeMode: Block
  accessModes: [ReadWriteOnce]
# in the pod
volumeDevices:
  - name: data
    devicePath: /dev/xvda

Block mode presents the raw device with no filesystem, which suits consumers that manage their own on-disk format — databases with raw device support, or clustered storage software.

# Substitute your own value before running:
PVC=data-postgres-0

kubectl get pvc "$PVC" -o jsonpath='{.spec.volumeMode}'

Choosing RBD or CephFS

RequirementUse
One writer, block semanticsRBD
One writer, best performanceRBD
Many readers and writersCephFS
Many pods on different nodesCephFS
Raw deviceRBD block mode
Shared configuration or contentCephFS
The question that decides it:
  will more than one pod write to this at the same time?
    yes → CephFS
    no  → RBD

Quiz

Knowledge check · 4 questions

  1. Q1. Why does the `exclusive-lock` feature not make ReadWriteMany safe for a filesystem?

  2. Q2. `ReadWriteMany` is safe for RBD volumes in block mode with a cluster-aware consumer.

  3. Q3. Handle a ReadWriteMany requirement.

    An application needs several pods across different nodes to write to shared storage. The team has requested a ReadWriteMany RBD volume.

  4. Q4. What single question decides between RBD and CephFS?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Never grant ReadWriteMany on an RBD volume carrying a filesystem — Ceph permits the mapping and the filesystem corrupts. Route shared-write requirements to CephFS, and use RBD block mode only where the consumer coordinates its own access.

Cross-course references

  • Kubernetes: access modes exist precisely to encode this constraint
  • Linux: mounting a non-clustered filesystem twice has always destroyed it