Checksums — the promise that data comes back as it went in

Intermediate⏱ ~15 minceph

What you'll learn

Explain what BlueStore checksums protect against
Describe when checksums are computed and verified
Choose a checksum algorithm with an understanding of the trade
Interpret a checksum error correctly

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Silent data corruption — a device returning data that differs from what was written, with no error — is the failure mode storage systems exist to prevent. BlueStore’s answer is a checksum on every write, verified on every read.

How it works

write:  compute checksum over each block → store alongside metadata
read:   recompute checksum → compare → return data or raise an error

The checksum lives in RocksDB with the object’s extent map, so it is stored separately from the data it protects. A device error corrupting data will not correspondingly corrupt the checksum.

# POOL is the pool to set the per-pool override on; substitute your own:
POOL=rbd-vms

ceph config get osd bluestore_csum_type
ceph osd pool set "$POOL" csum_type crc32c

Algorithm	Cost	Detection
`crc32c` (default)	very low, hardware-accelerated	excellent for device errors
`xxhash32` / `xxhash64`	low	comparable
`none`	zero	none — do not use

crc32c is the right choice for essentially all workloads. Setting none removes the guarantee entirely for a saving that does not register on modern CPUs.

A checksum error is a read failure, not a silent return

When verification fails, BlueStore does not return the data. It returns an error, and the OSD logs it:

_verify_csum bad crc32c/0x1000 checksum at blob offset 0x0, got 0x...

For a replicated pool the client’s read is served from another replica and the client sees nothing wrong. The cluster then flags the inconsistency at the next scrub.

This is the behaviour you want, and it means a checksum error in the logs is a device that is returning wrong data — not a Ceph problem. The correct response is to check device health and plan replacement, not to repair the PG and move on.

What checksums do and do not cover

Covered: bit flips on the device, misdirected writes landing at the wrong offset, torn writes, firmware bugs returning wrong data, and degradation of stored data over time.

Not covered: corruption that occurs before BlueStore computes the checksum. If data is corrupted in host memory before the write, the checksum is computed over the corrupted data and will verify correctly forever.

That gap is why ECC memory matters on OSD hosts. It is the one place in the chain BlueStore cannot protect, and non-ECC memory on a storage node undermines an otherwise complete integrity story.

How checksums and scrub divide the work

Checksums verify on read: they catch corruption in data that is being accessed. Data that is never read is never checked.

Scrub exists to close that gap. A shallow scrub compares object metadata across replicas; a deep scrub reads all the data and verifies checksums, which is what finds corruption in cold data before someone needs it.

The division matters for how you interpret each. A checksum error found on a client read means that object was being used. A checksum error found by deep scrub means the corruption existed and nobody had noticed — possibly for months.

It also explains why deep scrub is expensive and why disabling it is a poor trade. Deep scrub is the only mechanism that examines data nobody is reading, and cold data is precisely where silent corruption accumulates unnoticed.

Responding to a checksum error

Identify the OSD and device from the log line.
Check device health: smartctl -a, ceph device get-health-metrics.
Determine scope: one error or many? Rising?
If the device shows pending sectors, media errors, or repeated checksum failures, plan replacement.
Repair the affected PG only after understanding which copy is authoritative.

A single isolated checksum error on an otherwise healthy device can be a cosmic-ray event. Repeated errors on one device are the device.

Quiz

Knowledge check · 4 questions

Q1. BlueStore detects a checksum mismatch reading an object from a replicated pool. What does the client see?

Nothing wrong — the read is served from another replica A read error, which the application must handle Corrupted data, with the mismatch logged for later investigation A delay while the OSD repairs the block from a replica
Q2. BlueStore checksums protect against data corrupted in host memory before the write is issued.

True False
Q3. A team proposes disabling deep scrub to reduce cluster load, arguing that checksums already protect the data. Respond.

96-OSD cluster serving RBD and an RGW archive. Deep scrub causes noticeable latency during business hours. The proposal is to set nodeep-scrub permanently, reasoning that BlueStore verifies checksums on every read so corruption will be caught when data is accessed. The archive pool holds data that is written once and read very rarely.
Q4. Explain how checksums and deep scrub divide the integrity work between them.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Leave bluestore_csum_type at crc32c and never set it to none — the saving does not register on modern CPUs and the guarantee is the point. Treat a checksum error in the logs as evidence about a device rather than about Ceph, and investigate device health before repairing the PG. Insist on ECC memory for OSD hosts, since pre-checksum corruption is the one gap. And constrain deep scrub timing rather than disabling it, because cold data is where silent corruption accumulates and deep scrub is the only thing that looks.

Cross-course references

Ceph: Part LXI (Scrubbing) for scheduling and tuning scrubs.
Ceph: Part LXII (Inconsistent PGs) for what happens after detection.
Ceph: Part III (Storage Hardware) for the devices that cause these errors.