Volume density limits per node

Advanced⏱ ~17 minkubectlrbd

What you'll learn

Identify the limits on volumes per node
Configure the CSI volume limit
Recognise the symptoms of exceeding a limit
Plan density for high-volume workloads

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

There is no single hard limit on RBD volumes per node, but there are several practical ones, and hitting them produces pods that will not schedule or will not mount.

The limits

Limit	Source	Typical value
CSI reported volume limit	`--maxvolumesperode` on the node plugin	configurable
Kernel memory per mapped image	krbd	grows with count
File descriptors in the node plugin	process limit	configurable
Connections to OSDs	one set per mapped image	grows with count
Device node allocation	`/dev/rbd*`	large
Mount points	kernel	large

# in the node plugin DaemonSet
args:
  - "--maxvolumesperode=64"

# Substitute your own value before running:
NODE=worker-03

kubectl get csinode "$NODE" -o jsonpath='{.spec.drivers[?(@.name=="rbd.csi.ceph.com")].allocatable.count}'

The reported limit is what the scheduler uses: pods requiring a volume are not scheduled onto a node already at its limit.

Symptoms of exceeding a limit

Symptom	Limit
Pod stays Pending with a volume-limit predicate failure	CSI reported limit
Mapping fails with out-of-memory errors	kernel memory
`too many open files` in the plugin log	file descriptors
Mapping becomes progressively slower	connection count
Node becomes unstable at high density	kernel memory

PENDING_POD=pending_pod
kubectl describe pod ${PENDING_POD} | grep -A5 Events
# 0/12 nodes are available: 12 node(s) exceed max volume count

# the ceph-csi node plugin pod on the node in question:
PLUGIN_POD=csi-rbdplugin-7x4kd

# how many are mapped on a node
kubectl -n ceph-csi exec "$PLUGIN_POD" -- rbd showmapped | wc -l

Configuring the limit

Setting it too low: pods do not schedule despite capacity
Setting it too high: nodes become unstable before the scheduler notices

# the ceph-csi node plugin pod on the node in question:
PLUGIN_POD=csi-rbdplugin-7x4kd

# measure actual resource use at a given density
kubectl -n ceph-csi exec "$PLUGIN_POD" -- cat /proc/self/limits | grep -i files
free -g   # on the node

The right value depends on node memory, kernel version, and the volume sizes. Measuring at increasing density on a test node is the only reliable way to determine it.

Planning for high density

Approach	Effect
Fewer, larger volumes	far fewer mappings
CephFS instead of RBD	one mount serves many pods
Higher `maxvolumesperode` with more node memory	more volumes per node
More nodes	spreads the mappings
`rbd-nbd` instead of krbd	userspace; different resource profile

A workload with 500 pods each needing 1 GiB:
  500 RBD volumes across the fleet, mapped individually
  or one CephFS volume with 500 subdirectories

The second is dramatically cheaper in mappings, connections, and kernel memory, at the cost of shared-filesystem semantics.

Quiz

Knowledge check · 4 questions

Q1. Why does mapping many RBD images consume significant node resources?

The node plugin holds a full copy of each image allocation map in memory so it can serve reads without contacting an OSD The kernel serialises requests across every mapped image through one shared krbd queue, so each extra image adds queueing delay Each mapped image reserves its provisioned size in the node page cache, so a hundred large images pin most of the node memory Each mapping creates its own kernel block device, client session, and set of OSD connections with their socket buffers
Q2. There is a hard maximum number of RBD volumes a node can map.

True False
Q3. Plan storage for a high pod density workload.

A workload will run 500 pods each needing 1 GiB of persistent storage. The team plans to use 500 RBD PVCs.
Q4. What symptom indicates the CSI volume limit has been reached?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Reach for CephFS when a workload needs many small per-pod volumes — each RBD mapping carries its own kernel device, session, and OSD connections, while one CephFS mount serves every pod on a node. Set maxvolumesperode from measurement on a test node rather than from a guess.

Cross-course references

Kubernetes: per-node resource limits become scheduling constraints across many resource types
Linux: per-device kernel state limits practical device counts on any system