Media lifecycle, bit rot and decommissioning

The tape lesson ended on a clock rather than a rate: the LTO Program states that LTO-9 drives read and write LTO-8 and LTO-9 media only, so any retention longer than that read window is a migration schedule wearing a shelf’s clothing. That clock is not a peculiarity of tape. Every medium in the estate runs one, and a second clock runs alongside it on the bytes themselves, which degrade quietly and produce no event when they do. Long retention is where both clocks matter, because it is the only place where a copy is expected to sit unread for years and still be worth something at the end.

Ten bytes, and every piece of metadata is still correct

This course measured the failure mode rather than describing it. A restic repository holding two snapshots of a 60 MiB source tree was damaged the way storage actually fails — not by deleting a file, which any inventory would notice, but by overwriting ten bytes at the midpoint of the largest data pack, 17374653 bytes of bulk file data. Afterwards the file was still 17374653 bytes. Its name was unchanged, its inode was unchanged, its link count was unchanged, and mtime was the only filesystem-visible difference — a field that moves for every legitimate write as well. Nothing watching for missing or short files had anything to fire on.

restic check then reported no errors were found and exited 0. On the same repository, seconds later, restic check --read-data exited 1 and named the damaged pack. The gap between those two results is not thoroughness or care. It is that only one of the two commands read the stored bytes; the other verified the structure and the metadata, all of which were still perfectly true. Then the restore settled the question of whether any of this was academic.

$ restic restore 3fe43af4 --target /work/restore2

restoring snapshot 3fe43af4 of [/work/prod] at 2026-08-28 13:27:02.65376235 +0000 UTC by root@8211a08b55c3 to /work/restore2
ignoring error for /work/prod/db/data.bin: decrypting blob <data/9a6d59cf> from pack 2c3be6d1c75844d268248b7a2a90e42f5d325095bd381b31c25bcc3d0179951f failed: ciphertext verification failed
Summary: Restored 6 / 7 files/dirs (59.401 MiB / 60.000 MiB) in 0:00
Fatal: There were 1 errors
>>> exit code: 1

--- verifying whatever was restored, file by file ---
./app/app.conf: OK
./app/orders.csv: OK
./db/data.bin: FAILED
md5sum: WARNING: 1 computed checksum did NOT match
>>> verification exit code: 1

The corruption above was deliberate, and that is the only artificial thing about it. A cell that has lost charge, a sector whose error-correcting code no longer corrects, a tape with an unrecoverable read error, a controller that acknowledged a write it never completed: every one of them arrives at the filesystem as bytes that differ from what was written, under metadata that is entirely intact. The signature is the same one because there is only one.

What makes this a lifecycle problem rather than an incident is exposure time. Production data is read constantly, so a bad block there is usually found by the application within days. A recovery point written once and read never is a bet that nothing has happened to it for the whole retention period, and seven years of that bet, taken silently, is the risk long retention actually carries.

A checksum detects; only a second copy repairs

Checksummed filesystems exist to close exactly this gap, and it is worth being precise about which half of the problem they close. ZFS and Btrfs store a checksum for each block along with the metadata that points at the block, so an ordinary read recomputes and compares before the data is handed to the caller. A block that no longer matches is returned as an error rather than as plausible bytes, which is the whole difference between corruption you can see and corruption you cannot.

A scrub, described in zpool-scrub(8) and in the equivalent btrfs-scrub(8) documentation, is the deliberate version of that read: it walks the stored data and forces the comparison to happen on blocks nobody is asking for. Where the pool holds a second copy of a block that fails its checksum — a mirror, a parity group, or an extra copy of the same data — the good copy can be used to rewrite the bad one, and detection becomes repair. Where the pool holds no second copy, the scrub reports the error, tells you which file it belongs to, and stops there. That is still a large improvement over silence, and it is not a repair. The redundancy is what repairs; the checksum only ever finds.

Backup repositories reach the same place by a different route. In a restic repository a blob is identified by the hash of its own contents, so re-reading a pack is arithmetically the same act as verifying it, which is why the verifying command is spelled --read-data. The repository itself carries no redundancy that could substitute a good copy for a bad one, so at repository level the tool can only report. Repair comes from the architecture instead: in the capture, the second and undamaged copy of the same repository was the thing that made the damage survivable, and it earned that status by passing a full data read of its own.

$ restic check --read-data

using temporary cache in /tmp/restic-check-cache-1847567736
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
[0:00] 100.00%  2 / 2 snapshots
read all data
[0:00] 100.00%  7 / 7 packs
no errors were found
>>> exit code: 0

The line that carries the claim is [0:00] 100.00% 7 / 7 packs. Everything above it was also produced by the run that passed on the damaged repository. Only the pack count establishes that every stored byte was read and compared, and it is the reason this output is evidence while the cheaper one is not.

Sizing the rotation so every copy carries a date

The policy sentence this material is trying to earn is a specific one: every copy of every repository is read in full at least once every N days. That form forces two admissions — verification is per copy rather than per system, and N is a number you have to defend rather than an aspiration.

N comes out of capacity and read throughput, using the throughput figure measured on the actual path rather than a datasheet. Take a stated architecture: 200 TB per copy, three copies, and a verification stream that sustains 300 MB/s on the path each copy lives on. One full pass reads 200 TB at 300 MB/s, which is roughly 667,000 seconds, or about 7.7 days of continuous reading. Three copies is about 23 days. Spread across a 90-day window that is roughly 26% of the wall clock, or a little over six hours of verification every night. The numbers are entirely a consequence of those assumptions; change the throughput and the answer changes with it.

The artefact the rotation must produce is a per-copy date, not a stream of green job results. A schedule that records when each copy was last read in full lets you answer the only question that matters during an incident — how old the last proof is for the copy you are about to restore from.

# One slice of the estate per night, so every copy is read inside the window.
SLICE="$(date +%u)"                       # 1..7: today's position in the rotation
LIST="/etc/backup/verify-slice-${SLICE}.txt"
mapfile -t REPOS < "$LIST"
for REPO in "${REPOS[@]}"; do
  if restic -r "$REPO" check --read-data; then
    date -Is > "/var/lib/backup/verified/$(basename "$REPO")"
  fi
done

Where a full pass is genuinely unaffordable — media whose reads are billed, or a tape library with more cartridges than drive hours — the honest response is to verify a stated sample and publish the sample size, so the claim stays proportional to the evidence. The dishonest response is to keep running the structural check and let the dashboard imply that the data was read.

Format obsolescence is the second clock

A medium can be perfectly healthy and still hold data the estate cannot get back, because reading it requires three things to exist at the same time: a device that can read the physical medium, a host and driver stack that can talk to the device, and software that understands the format written on it. For encrypted copies there is a fourth, and Part IX of this course covers what happens when the key is the missing one. Data that survives on media the estate can no longer read is not partially recoverable. It is gone, with a shelf full of confirming artefacts.

Tape makes this concrete because the vendors publish the window: LTO-9 drives read and write LTO-8 and LTO-9 media only, so a seven-year retention on tape crosses at least one drive generation and must be planned to. Disk-based estates get no such published statement and have the same problem. A repository written by a tool whose format the current binaries no longer accept, a proprietary catalogue whose vendor no longer exists, a filesystem image needing a kernel module that no longer ships — each ends with bytes that are present and unreadable. Deduplicated repositories carry the largest surface here: the format is not merely the container but the chunker, the index and the encryption scheme together.

The response to both clocks is the same operation. Rewrite the data onto media and into a format that is current, before either the stated life of the media or the read window of the format runs out. That rewrite is complete when the new copy has been read back and compared against a hash recorded before the move — not when the copy job exits zero, which is the same claim that the damaged repository above was making. The ZFS capture shows the shape of a migration that ends in evidence.

$ zfs clone rbdrbkp/ledger@0900 rbdrbkp/restore0900

ORDER-1001,4500.00
ORDER-1002,1250.00
md5 recovered:     9eb4e2ad8e08e1dcaaf87ababab964b0
md5 recorded 09:00: 9eb4e2ad8e08e1dcaaf87ababab964b0
RECOVERED - byte-identical to the 09:00 ledger

Decommissioning is a security operation with its own record

A retired production disk holds one system’s data. A retired backup disk holds every recovery point it ever stored, which usually spans many systems and several years, and its sensitivity is the maximum of everything that ever landed on it rather than whatever the host it was last attached to happened to run. That single asymmetry is why media retirement in a backup estate belongs to security rather than to capacity management.

NIST SP 800-88 Rev. 1 gives the vocabulary. It organises sanitization into three categories — Clear, Purge and Destroy — and chooses between them on the confidentiality of the data and on whether the media will leave organisational control. It also treats verification and documentation as part of sanitization rather than as paperwork bolted on afterwards, to the point of including a sample certificate form. The operational reading of that is simple: a decommissioning that produced no record did not happen, because in six months nothing will distinguish it from a disk that went into a skip.

The record names the medium by serial number, states what it held — which repositories, which systems, which span of recovery points — states the method used, names who verified it, and records where the medium went and when. Two dates meet in it and are frequently confused. Retention expiry is a property of the data; media retirement is a property of the hardware. Destroying a medium early destroys recovery points still inside their retention window, and keeping one late keeps data past a deletion obligation.

Encryption changes the economics here, under one exact condition. If a medium was encrypted before anything was written to it, destroying every copy of the key retires the data without touching the medium — the technique SP 800-88 discusses as cryptographic erase. The condition is total: escrowed copies count, the copy in the key management system counts, and a key that still exists anywhere means the medium has not been sanitised. Part IX built the escrow arrangements that make key custody survivable; decommissioning is where that inventory has to prove it is complete.

Production discipline

1. Schedule a reader for every copy, and write the window down. On the stated architecture above — 200 TB per copy, three copies, 300 MB/s sustained — one pass is about 7.7 days and three copies about 23 days, which fits a 90-day window at roughly six hours a night. A window you cannot meet should be published as the larger number you can.
2. Record the last full read per copy, not per job. The measured contrast is the whole reason: plain restic check reported no errors were found at exit 0 on a repository that then returned 6 of 7 files and 59.401 MiB of 60.000 MiB, while restic check --read-data on the healthy copy reported [0:00] 100.00% 7 / 7 packs.
3. Keep detection and repair in separate columns. A checksum finds the bad block; only a redundant copy replaces it. The damaged repository was survivable because a second, independently stored copy passed a full data read of its own, not because any command could repair the first one.
4. Give every retained format a migration date and prove the read path. LTO-9 drives read and write LTO-8 and LTO-9 media only, and a migration is finished when the new copy has been read back and compared — as in the capture, where the recovered ledger returned md5 9eb4e2ad8e08e1dcaaf87ababab964b0 against the value recorded at 09:00.
5. Decommission under a record that names the medium and the method. Serial number, contents, sanitization method, verification, custody and date, with the medium classified by the most sensitive data it ever held; treat cryptographic erase as valid only when every copy of the key, escrow included, is destroyed.

Cross-course references

• Ceph & Distributed Storage for Production Sysadmins — Part LXI (Scrubbing) runs this lesson’s rotation at cluster scale, where the tuning question is how much throughput the verification may consume — the same trade this lesson settles with a 90-day window and six hours a night.
• Ceph & Distributed Storage for Production Sysadmins — Part LXII (Inconsistent PGs) is detection without automatic repair in practice: a scrub has found copies that disagree and an operator must decide which is authoritative, which is the cluster-scale form of this lesson’s point that the checksum finds and only redundancy repairs.
• Linux for Production Sysadmins — Part LXIX (Hardware Health) covers the device-level signals — SMART attributes, reallocated sectors, error counters — that say a medium is nearing the end of its life, and those signals are what should trigger the migration this lesson schedules, rather than waiting for the stated life to expire on the calendar.

Quiz