Verifying replacement hardware before deployment

Intermediate⏱ ~17 minsmartctllsblkethtool

What you'll learn

Verify compatibility before deployment
Check firmware and its consistency
Identify differences from the existing fleet
Decide whether a difference is acceptable

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Replacement hardware that differs from the fleet produces problems that appear as Ceph problems and are diagnosed as such for a long time.

Compatibility

# the drive
smartctl -i /dev/sdX | grep -E 'Model|Firmware|Rotation|Capacity|Form Factor'
lsblk -o NAME,SIZE,ROTA,MODEL,SERIAL /dev/sdX

Property	Must match	Why
Capacity	ideally; the weight is set from it	mismatched capacities need weight care
Interface	yes	SATA in a SAS-only backplane may not work
Sector size	yes	512e versus 4Kn affects alignment
Rotation rate	for class detection	affects device class
Form factor and connector	yes	physical fit

# sector size, which is easy to overlook
smartctl -i /dev/sdX | grep -i 'Sector Size'
blockdev --getss --getpbsz /dev/sdX

A 4Kn drive in a fleet of 512e drives is functional and behaves differently enough to be worth knowing about.

Firmware

smartctl -i /dev/sdX | grep -i firmware
nvme id-ctrl /dev/nvmeXn1 | grep -i '^fr'

# compare against the fleet
for d in /dev/sd{b..m}; do
  printf '%-10s ' "$d"
  smartctl -i "$d" | awk -F: '/Firmware Version/ {gsub(/ /,"",$2); print $2}'
done | sort -k2 | uniq -c -f1

Firmware differences within a fleet:
  usually harmless
  occasionally the cause of behaviour differences
  worth recording so a later investigation can consider it

Known-bad firmware revisions exist for many drive models, and checking the vendor’s advisories before deploying is worth the minutes.

Differences from the fleet

# a comparison against an existing drive
for f in Model Firmware Rotation 'Sector Size' Capacity; do
  printf '%-14s new=%-24s old=%s\n' "$f" \
    "$(smartctl -i /dev/sdX | grep -i "$f" | cut -d: -f2- | xargs)" \
    "$(smartctl -i /dev/sdb | grep -i "$f" | cut -d: -f2- | xargs)"
done

Difference	Acceptable?
Newer firmware, same model	usually
Different model, same specifications	usually; verify performance
Larger capacity	yes, with the CRUSH weight set correctly
Smaller capacity	yes, with the weight set correctly; reduces the host’s total
Different sector size	verify; may affect performance
Different rotation rate	affects device class; verify detection
Consumer rather than enterprise	no for OSD use

Deciding

A difference is acceptable when:
  it is understood
  its effect is bounded
  it is recorded so a later investigation can consider it

It is not acceptable when:
  it changes the device class
  it lacks power-loss protection where the fleet has it
  its endurance rating is below the workload's requirement

# record it
echo "$(date -I) osd.44 replaced: $(smartctl -i /dev/sdX | grep -E 'Model|Serial|Firmware' | tr '\n' ' ')" \
  >> /var/log/ceph-hardware.log

Quiz

Knowledge check · 4 questions

Q1. Why does a drive without power-loss protection become a persistent latency outlier?

It uses slower NAND, because power-loss protection tends to be fitted only to drives built with the faster grades of flash in the first place It fails more often, and the internal retries a marginal drive performs are what show up as added latency BlueStore's flushes must write through to media rather than being acknowledged from cache, which is often an order of magnitude slower It reports its SMART latency counters on a different scale, so the outlier is an artefact of the measurement
Q2. A larger replacement drive automatically receives a correct CRUSH weight.

True False
Q3. Verify a replacement drive before deployment.

A replacement drive has arrived. It is a different model from the fleet but the same nominal capacity and interface.
Q4. Which hardware differences are not acceptable for an OSD replacement?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Confirm a replacement has power-loss protection where the fleet does — without it BlueStore’s flushes write through to media and the drive becomes a permanent latency outlier that looks like a failure. Verify the CRUSH weight matches the capacity after any non-identical replacement.

Cross-course references

Kubernetes: heterogeneous node hardware produces workload behaviour differences
Linux: mixing drive specifications in an array produces the slowest member’s behaviour