RBD image layout for Kubernetes volumes

Advanced⏱ ~17 minrbdkubectl

What you'll learn

Explain RBD object layout and its effect
Choose striping parameters where they matter
Understand thin versus thick provisioning
Recognise when the defaults are wrong

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

An RBD image is a sequence of RADOS objects, and the object size determines how a volume’s I/O distributes across the cluster.

The layout

A 100 GiB image with the default 4 MiB object size:
  → 25,600 objects named rbd_data.<id>.<index>
  → each placed independently by CRUSH
  → a 4 KiB write touches one object
  → a 64 MiB sequential write touches 16 objects

IMAGE=vm-disk-01
rbd info k8s-rbd/${IMAGE}
# order: 22 (4 MiB objects)
# block_name_prefix: rbd_data.2f9c74b0dc51

Object size	Objects for 100 GiB	Effect
4 MiB (default)	25,600	good distribution, more metadata
8 MiB	12,800	fewer objects, coarser distribution
1 MiB	102,400	very fine distribution, much more metadata

The default suits almost everything. Changing it is warranted only for specific access patterns.

Striping

rbd create --size 100G --stripe-unit 4M --stripe-count 4 pool/image

Without striping: sequential writes fill one object, then the next
                  → one OSD at a time for a sequential stream
With stripe-count 4: writes round-robin across 4 objects
                  → four OSDs in parallel

Workload	Striping
Random small I/O	default; striping adds nothing
Large sequential	`stripe-count 4` or more helps
Databases	default; access is already distributed
Video, backups	striping helps throughput

Ceph-CSI does not expose striping parameters in the StorageClass, so striped images require creating them outside CSI and importing as static PVs.

Thin and thick provisioning

parameters:
  thickProvision: "false"     # default

	Thin	Thick
Provisioning time	instant	proportional to size
Capacity consumed initially	none	the full size
Capacity reporting accuracy	usage grows over time	accurate from the start
Over-commitment possible	yes	no
First-write latency	allocation on first write	pre-allocated

Thin is the default and almost always correct. Thick provisioning writes zeros across the whole image at creation, which for a 1 TiB volume takes a long time and consumes the capacity immediately.

# actual consumption of a thin image
IMAGE=vm-disk-01
rbd du k8s-rbd/${IMAGE}

When the defaults are wrong

Situation	Change
Very large sequential throughput requirement	striping, via a static PV
Capacity must be guaranteed at provision time	`thickProvision: true`
Extremely small volumes, many of them	consider CephFS instead
Object size mismatched to the workload’s I/O size	rarely worth changing

Quiz

Knowledge check · 4 questions

Q1. Why does striping help sequential throughput but not random I/O?

Without striping a sequential stream fills one object at a time and is bounded by one OSD; random I/O already touches objects scattered across the cluster Random I/O is served largely from the OSDs' BlueStore cache, so the number of objects it spans makes little difference Striping applies only on the read path, since writes have to be assembled in order before they can be placed into an object and handed to that PG's primary OSD Random I/O takes a different code path in librbd, one that ignores the stripe unit and stripe count settings entirely
Q2. Thin provisioning means capacity planning can use the sum of PVC sizes.

True False
Q3. Plan capacity for a thin-provisioned cluster.

A Kubernetes cluster has PVCs totalling 400 TB against a Ceph cluster with 120 TB usable. Actual usage is 48 TB. The team is unsure whether this is a problem.
Q4. Why does Ceph-CSI not expose striping parameters in the StorageClass?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Track actual usage rather than committed PVC size for capacity planning — thin provisioning permits over-commitment by design and the sum of claims says nothing about when the cluster fills. Reach for striping only for large sequential workloads, and note it requires creating the image outside CSI.

Cross-course references

Kubernetes: resource requests versus actual usage present the same over-commitment question
Linux: thin LVM pools require the identical usage-based monitoring