Designing StorageClasses for Ceph

Advanced⏱ ~18 minkubectlceph

What you'll learn

Configure a Ceph StorageClass correctly
Choose the parameters for a workload
Decide how many classes to define
Recognise the immutable decisions

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

A StorageClass encodes the storage policy for every volume created from it, and several of its parameters cannot be changed for volumes already provisioned.

A complete StorageClass

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ceph-rbd-fast
provisioner: rbd.csi.ceph.com
parameters:
  clusterID: b3d5f2a1-...
  pool: k8s-rbd-nvme
  imageFeatures: layering,exclusive-lock,object-map,fast-diff,deep-flatten
  csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret
  csi.storage.k8s.io/provisioner-secret-namespace: ceph-csi
  csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret
  csi.storage.k8s.io/controller-expand-secret-namespace: ceph-csi
  csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret
  csi.storage.k8s.io/node-stage-secret-namespace: ceph-csi
  csi.storage.k8s.io/fstype: ext4
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer

The parameters that matter

Parameter	Effect
`pool`	which Ceph pool, and therefore which device class and durability
`imageFeatures`	RBD features; must suit the client
`csi.storage.k8s.io/fstype`	the filesystem created on the image
`reclaimPolicy`	`Delete` removes the image when the PVC is deleted
`allowVolumeExpansion`	whether PVCs can be grown
`volumeBindingMode`	when the volume is provisioned
`mounter`	`rbd` (krbd) or `rbd-nbd`

volumeBindingMode: WaitForFirstConsumer delays provisioning until a pod is scheduled, which is what allows topology constraints to work.

Choosing parameters for a workload

Workload	Pool	fsType	Features
Databases	NVMe pool	`xfs`	full set
General applications	SSD pool	`ext4`	full set
Bulk data	HDD pool	`xfs`	full set
Old kernels	any	`ext4`	`layering` only

# what the node kernel supports
uname -r

# POOL is the StorageClass `pool` parameter; IMAGE is the CSI-provisioned
# image name from `rbd ls "$POOL"`. Substitute your own:
POOL=kubernetes
IMAGE=csi-vol-8f3a2b1c-4d5e-4f70-8192-a3b4c5d6e7f8

rbd feature disable "$POOL/$IMAGE" object-map fast-diff  # if mapping fails

How many classes

One per meaningful policy difference:
  ceph-rbd-nvme       fast pool, Delete
  ceph-rbd-hdd        bulk pool, Delete
  ceph-rbd-retain     fast pool, Retain — for data that must survive PVC deletion
  ceph-fs             CephFS, for ReadWriteMany

Four classes covering device class, reclaim policy, and access mode. More than that usually means encoding something that belongs elsewhere.

The immutable decisions

Decision	Changeable for existing volumes?
`pool`	no — the image is in that pool
`fstype`	no — the filesystem is created at provision time
`imageFeatures`	partially, via `rbd feature` on each image
`reclaimPolicy`	yes, on the PV object
`allowVolumeExpansion`	affects new operations only

# reclaim policy can be changed per PV
# PV name from `kubectl get pv`; substitute your own:
PV=pvc-8f3a2b1c-4d5e-4f70-8192-a3b4c5d6e7f8

kubectl patch pv "$PV" -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

A StorageClass itself is immutable in most fields — changing policy means creating a new class and migrating.

Quiz

Knowledge check · 4 questions

Q1. Why is `volumeBindingMode: WaitForFirstConsumer` required for topology-aware placement?

It reduces provisioning latency, since the image is created while the scheduler is still filtering candidate nodes It enables volume expansion, which requires the volume to be bound to a node before the resizer can act on it With Immediate binding the volume is created before the scheduler picks a node, so the provisioner cannot honour topology constraints The CSI specification requires it for any driver that reports topology keys, and the provisioner rejects Immediate binding from such a driver
Q2. A StorageClass's `pool` parameter can be changed to migrate existing volumes to a different pool.

True False
Q3. Design StorageClasses for a cluster.

A Kubernetes cluster will host databases, general applications, and bulk data on Ceph. Some data must survive accidental PVC deletion.
Q4. Which StorageClass decisions cannot be changed for already-provisioned volumes?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Define a Retain-policy StorageClass for anything whose loss would matter — Delete is the common default and a namespace deletion destroys the underlying images. Set volumeBindingMode: WaitForFirstConsumer; it is a prerequisite for topology awareness rather than an optimisation.

Cross-course references

Kubernetes: StorageClass design mirrors any policy-per-tier storage system
Linux: filesystem type chosen at creation is equally immutable