Capacity efficiency and the real cost of a profile

Why this matters in production

Capacity efficiency is the reason EC exists, and it is the number that gets quoted in budget meetings. The headline arithmetic is easy and the exceptions are not, and a capacity plan that uses the headline number for a pool full of small objects will be badly wrong.

The headline arithmetic

Raw bytes per user byte is (k+m)/k. Overhead is m/k.

Against size 3, a 4+2 pool doubles usable capacity for comparable — in some respects better — failure tolerance. On a petabyte of raw disk that is 334 TB of difference.

Same overhead, different durability

4+2 and 6+3 both cost 1.5× raw. They are not equivalent:

• 4+2 loses any 2 of 6 chunks → survives, needs 6 failure domains
• 6+3 loses any 3 of 9 chunks → survives, needs 9 failure domains

6+3 is strictly more durable at identical capacity cost. The price is that it needs more failure domains and touches more OSDs per operation, which costs latency and makes recovery broader.

Where the savings disappear

BlueStore allocates in units of bluestore_min_alloc_size — 4 KiB on recent releases for both HDD and SSD, historically 64 KiB on HDD. Each EC chunk is allocated independently.

A 16 KiB object on a 4+2 profile splits into 4 KiB data chunks:

16 KiB object → 4 data chunks of 4 KiB + 2 coding chunks of 4 KiB
              = 6 allocations of 4 KiB = 24 KiB raw
              = 1.5× — as expected

But on an older cluster with 64 KiB min_alloc_size:

16 KiB object → 6 chunks, each rounded up to 64 KiB
              = 384 KiB raw for 16 KiB of data
              = 24× overhead

The general rule: EC’s capacity advantage requires objects substantially larger than k × min_alloc_size. For 4+2 at 4 KiB that is 16 KiB and up — fine for most workloads. On legacy 64 KiB allocations it is 256 KiB and up, which excludes a great deal of real data.

ceph config get osd bluestore_min_alloc_size_hdd
ceph config get osd bluestore_min_alloc_size_ssd

Quiz

Production discipline

Base EC capacity plans on measured average object size from the actual workload, not on the profile arithmetic alone, and record both in the plan so the assumption is visible. Where a workload is dominated by small objects, say so explicitly in the capacity document — it is the most common reason an EC projection misses.

Cross-course references

• Kubernetes: requests versus actual usage has the same headline-versus-reality gap
• Linux: filesystem block size and small-file waste is the identical phenomenon