Image layout: objects, striping, and sparseness

Why this matters in production

The default layout is right for almost everything, which makes the striping parameters look like an obvious tuning opportunity they usually are not. Knowing what they actually change prevents both a wasted experiment and a missed opportunity on the one workload where they matter.

The default layout

Simple sequential mapping. Object N covers image bytes [N × 4 MiB, (N+1) × 4 MiB):

rbd info rbd-vms/vm-disk-01
# size: 100 GiB in 25600 objects
# order: 22 (4 MiB objects)
# block_name_prefix: rbd_data.1f2a3b

A 1 MiB write at image offset 0 touches one object. A 16 MiB write at offset 0 touches four, distributed by CRUSH.

Fancy striping

With --stripe-unit and --stripe-count, consecutive chunks are spread across several objects before advancing:

rbd create --size 1T \
    --stripe-unit 64K --stripe-count 16 \
    --object-size 4M \
    rbd-vms/striped-image

default (stripe-count 1):
  bytes 0–4 MiB    → object 0
  bytes 4–8 MiB    → object 1

stripe-unit 64K, stripe-count 16:
  bytes 0–64K      → object 0
  bytes 64K–128K   → object 1
  ...
  bytes 960K–1M    → object 15
  bytes 1M–1M+64K  → object 0, next stripe

A 1 MiB write now spans 16 objects — probably 16 OSDs — instead of one.

When it helps, and when it does not

Helps: a single-threaded process doing large sequential I/O. Without striping, that thread’s writes hit one object and therefore one primary OSD at a time, capping throughput at one device.

Does not help: anything with concurrency. Many VMs, or one VM with queue depth, already spread across objects naturally, and striping adds metadata overhead for no gain.

Actively hurts: small random I/O. A 4 KiB write still touches one object, but the image now has more objects with more metadata, and recovery has more to track.

Inspecting real usage

rbd du rbd-vms/vm-disk-01
# NAME          PROVISIONED  USED
# vm-disk-01        100 GiB   18 GiB

rados -p rbd-vms ls | grep -c rbd_data.1f2a3b
rbd info rbd-vms/striped-image | grep -E 'stripe'

rbd du shows provisioned versus actually allocated — the thin provisioning ratio, and the number that matters for capacity planning rather than the sum of image sizes.

Quiz

Production discipline

Default to the standard layout and require a measurement on the real workload before deviating — the parameters are fixed at creation and the benefit is confined to a narrow class of workload. Track both rbd du and provisioned totals per pool so over-commitment is a monitored position rather than an accident.

Cross-course references

• Kubernetes: PVC requested size versus actual usage is the same thin-provisioning gap
• Linux: LVM thin pools require exactly this dual accounting