Topology-aware volume placement

Advanced⏱ ~18 minkubectlceph

What you'll learn

Configure topology-aware provisioning
Map Kubernetes topology to Ceph failure domains
Verify placement is correct
Recognise when topology awareness matters

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

If Kubernetes nodes and Ceph OSDs are spread across zones, a pod in one zone reading data placed in another crosses the inter-zone link for every operation.

When it matters

Situation	Topology awareness
Single rack or zone	irrelevant
Zones with high inter-zone latency	important
Zones with metered or limited bandwidth	important
Ceph pools with zone-level failure domains	important
Uniform low-latency fabric	marginal

The benefit is proportional to the inter-zone cost. On a flat fabric it adds complexity for nothing.

Configuring it

# StorageClass
allowedTopologies:
  - matchLabelExpressions:
      - key: topology.kubernetes.io/zone
        values: [zone-a, zone-b, zone-c]
volumeBindingMode: WaitForFirstConsumer

# the CSI driver must be told about the topology
# in the node plugin DaemonSet
args:
  - "--domainlabels=topology.kubernetes.io/zone"

kubectl get nodes -L topology.kubernetes.io/zone

Mapping to Ceph failure domains

# Ceph side: a CRUSH hierarchy matching the zones
ceph osd crush add-bucket zone-a rack
ceph osd crush add-bucket zone-b rack
ceph osd crush add-bucket zone-c rack
ceph osd crush move ceph-01 rack=zone-a
ceph osd crush rule create-replicated zone-rule default rack
ceph osd pool set k8s-rbd crush_rule zone-rule

Kubernetes zone-a  ↔  CRUSH rack zone-a
Kubernetes zone-b  ↔  CRUSH rack zone-b
Kubernetes zone-c  ↔  CRUSH rack zone-c

The names must correspond and the physical placement must match both.

Verifying placement

# where is the pod?
NAME=acme
NODE=stor-04
IMAGE=vm-disk-01
OBJECT_NAME=acme
kubectl get pod ${NAME} -o jsonpath='{.spec.nodeName}'
kubectl get node ${NODE} -L topology.kubernetes.io/zone

# where is the data?
rbd info k8s-rbd/${IMAGE} | grep block_name_prefix
# then map the objects to OSDs
ceph osd map k8s-rbd ${OBJECT_NAME}

# which zone are those OSDs in?
OSD_ID=12
ceph osd find ${OSD_ID}

If the pod’s zone appears among the acting set’s zones, at least one replica is local.

What topology awareness does and does not do

Does	Does not
Provision the volume in a zone the pod can reach	guarantee reads come from the local replica
Prevent scheduling a pod where its volume is unreachable	control which replica serves a read
Align with `allowedTopologies` constraints	reduce write traffic

Reads are served by the primary OSD, which CRUSH chooses without regard to client location. Topology awareness ensures the volume is reachable and correctly placed, not that traffic is local.

Quiz

Knowledge check · 4 questions

Q1. What happens if `allowedTopologies` is set but the CSI driver has no `--domainlabels` argument?

The driver reports no topology, so the scheduler has nothing to match and the constraint silently has no effect Provisioning fails with an error naming the missing topology, which is how the misconfiguration is normally found The default zone is used, which is whichever zone the provisioner pod itself happens to be running in The StorageClass is rejected at admission, since allowedTopologies requires a driver that advertises topology
Q2. Topology-aware provisioning ensures reads are served from a local replica.

True False
Q3. Configure topology awareness across zones.

A Kubernetes cluster spans three zones with significant inter-zone latency. Ceph OSDs are in the same three zones. Volumes are currently provisioned without topology constraints.
Q4. When does topology-aware provisioning add complexity for no benefit?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Configure both halves — allowedTopologies on the StorageClass and --domainlabels on the CSI node plugin — or the constraint silently does nothing. Set expectations that topology awareness places volumes reachably; it does not localise reads, which CRUSH directs to the primary regardless of client location.

Cross-course references

Kubernetes: topology spread constraints require accurate node labels in the same way
Linux: data locality requires both placement and routing to be aware of it