Block size and the layers that have one

Why this matters in production

Each layer rounds I/O up to its own unit, and the units multiply. A guest write of 512 bytes can become a 4 KiB filesystem block, a 4 KiB BlueStore allocation on each of three OSDs, and — with a snapshot present — a 4 MiB object clone. Knowing where the multiplication happens is how you stop it.

The layers

Where amplification occurs

Sub-block application writes. A 512-byte write to a 4 KiB filesystem block means reading the block, modifying it, writing it back — inside the guest, before Ceph sees anything.

Misaligned partitions. A partition starting at an offset that is not a multiple of the filesystem block size makes every block span two underlying blocks:

# check alignment — start sector should be a multiple of 8 for 4 KiB
fdisk -l /dev/rbd0
parted /dev/rbd0 align-check optimal 1

Modern tooling aligns correctly by default; this is a legacy-image concern.

Snapshot copy-on-write. With a snapshot present, the first write to any object clones it, at object granularity.

EC partial-stripe writes. On an EC data pool, a write smaller than the stripe takes the read-modify-write path.

Choosing sizes

# filesystem block size at creation, matching the application
mkfs.xfs -b size=4096 /dev/rbd0
mkfs.ext4 -b 4096 /dev/rbd0

# RBD object size, fixed at creation
rbd create --size 1T --object-size 4M rbd-vms/standard
rbd create --size 10T --object-size 16M rbd-bulk/archive

That last row is the non-obvious one: on an image with frequent snapshots, a smaller object size reduces the copy-on-write cost per write, at the price of more objects and more metadata.

Measuring the real distribution

# in the guest
iostat -x 5                     # avgrq-sz, in sectors

# with blktrace for a detailed distribution
btrace /dev/rbd0 | awk '$6=="Q"{print $10}' | sort -n | uniq -c

Design against the measured distribution, not against an assumption about what the application does.

Quiz

Production discipline

Measure the request-size distribution before choosing an object size, since the parameter is fixed at creation and reasoning about expected behaviour is unreliable. Record the reasoning behind any non-default choice on the image itself, so it survives as a decision rather than looking like an anomaly.

Cross-course references

• Kubernetes: StorageClass parameters can encode object size per volume class
• Linux: filesystem block size and partition alignment are the same layered concern