Managing compute and storage contention

Why this matters in production

Compute and storage on one host contend for four resources, and each needs a different mechanism to bound.

The contended resources

The device row is the one to avoid entirely: VM disks and OSD data on the same physical devices is a configuration that should not exist.

Measuring contention

# CPU: is the OSD getting scheduled?
top -H -p $(pgrep -d, ceph-osd) -b -n 1 | head -20
mpstat -P ALL 1 5

# memory
free -g
ceph config get osd osd_memory_target
ceph daemon osd.0 dump_mempools | python3 -c '
import sys,json; d=json.load(sys.stdin)["mempool"]["by_pool"]
print("total:", sum(v["bytes"] for v in d.values()) // 1048576, "MiB")'

# is the host swapping at all?
vmstat 1 5 | awk 'NR>2 {print $7, $8}'

# correlate: VM CPU against OSD latency
ceph osd perf | sort -k2 -rn | head -3

Bounding the OSD side

ceph config set osd osd_memory_target 4294967296
ceph config set osd osd_max_backfills 1
ceph config set osd osd_mclock_profile high_client_ops

# CPU affinity via systemd
systemctl set-property ceph-osd@0.service AllowedCPUs=0-7
systemctl set-property ceph-osd@0.service CPUWeight=200

CPUWeight raises the OSD’s share under contention without pinning, which is usually preferable to hard affinity.

Bounding the VM side

# CPU limits per VM
qm set 100 --cpulimit 4 --cpuunits 100

# memory without ballooning surprises
qm set 100 --memory 16384 --balloon 0

# I/O limits per disk
qm set 100 --scsi0 ceph-vms:vm-100-disk-0,mbps_rd=200,mbps_wr=100,iops_rd=5000,iops_wr=2000

The per-disk I/O limits are what prevent one VM saturating the storage for every other, and they are rarely set by default.

Verifying the limits work

# generate load in a VM and watch OSD latency
# in the guest:
fio --name=load --rw=randwrite --bs=4k --iodepth=64 --runtime=300 --time_based

# on the host, concurrently:
watch -n 5 'ceph osd perf | sort -k2 -rn | head -3'

If OSD latency rises materially while one VM generates load, the limits are not effective and the numbers need tightening.

Acceptable: one VM at full allowed rate raises OSD latency modestly
Not acceptable: one VM raises cluster-wide latency for every other VM

Quiz

Production discipline

Set per-disk mbps and iops limits on every VM as a baseline; without them one VM can saturate the shared OSDs for the entire cluster. Use CPUWeight rather than hard affinity for OSDs, and never place VM disks on the same physical devices as OSD data.

Cross-course references

• Kubernetes: per-pod I/O limits are equally necessary and equally often omitted
• Linux: cgroup weights suit contention better than hard CPU pinning