Endurance and DWPD — sizing flash for years of writes

Why this matters in production

Flash wears out by being written. Unlike a mechanical failure, this one is predictable, which means it can be planned — and if it is not planned, a cluster built from identical drives commissioned on the same day will reach end of life at roughly the same time. That is a correlated failure with a calendar date, and it is entirely avoidable.

The two ratings

DWPD — drive writes per day. How many times the drive’s full capacity can be written each day for the warranty period, usually five years. A 1 DWPD 4 TB drive tolerates 4 TB of writes per day for five years.

TBW — terabytes written. The same figure expressed as a total.

TBW = DWPD x capacity x 365 x warranty_years

A 1 DWPD 4 TB drive with a five-year warranty is about 7,300 TBW. Typical classes:

The Ceph multiplier

Client writes are not device writes. Three factors sit between them:

device writes = client writes
                x replication factor (or EC expansion)
                x write amplification (BlueStore + flash internal)

A worked example:

Client ingest:              2 TB/day
Replication:                3x            → 6 TB/day
BlueStore metadata overhead: ~1.1x        → 6.6 TB/day
Flash write amplification:  ~1.2x         → 7.9 TB/day
Spread across 24 x 4 TB drives            → 330 GB/drive/day
As DWPD on a 4 TB drive                   → 0.08 DWPD

Comfortable for a 1 DWPD drive with a wide margin. That margin is typical for steady-state clusters, and the reason to compute it anyway is the activities that are not steady state.

Monitoring consumed endurance

# NVMe
nvme smart-log /dev/nvme0n1 | grep -E 'percentage_used|data_units_written'

# SATA/SAS
smartctl -A /dev/sda | grep -Ei 'wear|percent|total_lbas_written'

percentage_used on NVMe is the drive’s own estimate of consumed endurance. 100 does not mean the drive stops — it means the warranted endurance is exhausted and failure probability rises. Treat 80 as the point to plan replacement and 90 as the point to have ordered.

Ceph’s device health module can collect and store this:

DEVID=devid
ceph device ls
ceph device get-health-metrics ${DEVID}

Choosing the class

Match the class to the pool’s write intensity, not to the cluster average. In a mixed cluster the WAL/DB devices and metadata pools see far more writes per terabyte than the bulk data OSDs, so they justify a higher DWPD class even though they are a small fraction of the capacity. Buying one class for the whole cluster either overspends on the bulk tier or under-specifies the metadata tier — usually both.

Quiz

Production discipline

Compute the write budget before purchase — client writes times replication times amplification, divided across the fleet — and match the DWPD class to each pool’s write intensity rather than to a cluster average. Monitor percentage_used as a fleet distribution so a cohort approaching its limit is visible months ahead. And stagger commissioning and replacement deliberately: identical drives bought together and written evenly will wear out together, which is the one correlated failure you can put in a calendar.

Cross-course references

• Ceph: Part XCIV (Hardware Replacement) for the staged replacement procedure.
• Ceph: Part LVIII (Recovery) for the write volume a replacement generates.
• Observability: fleet-wide distribution metrics are the right shape for wear monitoring.