Fourteen days of hourly snapshots vanished with the array

Reported symptoms

At 02:14 an enclosure fault takes the primary array offline. It does not come back. At 06:40 the vendor confirms the pool cannot be reconstructed.

Nine production systems lived on that array. All nine appear in the protection inventory with the same entry: array snapshot, hourly, fourteen days retained, status OK. The status has been OK for eleven consecutive months.

The recovery call opens on the arithmetic that follows from that: fourteen days at hourly granularity is 336 recovery points per system, and the only decision is which hour to return to.

Forty minutes in, somebody asks which device those snapshots are stored on. Nobody can name one that is not the failed array.

Evidence provided

The storage configuration, first. Every recovery point in the estate is a child of the pool that failed.

$ zfs list -r -t all -o name,used,refer rbdr-prod

NAME                              USED  REFER
rbdr-prod                        7.41T   205K
rbdr-prod/vm-ledger              1.92T  1.44T
rbdr-prod/vm-ledger@hourly-0100  6.11G  1.44T
rbdr-prod/vm-ledger@hourly-0200  5.87G  1.44T
rbdr-prod/vm-billing             2.30T  1.71T
rbdr-prod/vm-billing@hourly-0100 9.02G  1.71T
rbdr-prod/vm-billing@hourly-0200 8.44G  1.71T

Second, the control that reported this as protected.

$ column -t -s, rbdr-protection-inventory.csv

system        tier  protection_method  schedule  retention  status
rbdr-ledger   1     array snapshot     hourly    14 days    OK
rbdr-billing  1     array snapshot     hourly    14 days    OK
rbdr-idm      1     array snapshot     hourly    14 days    OK
rbdr-reports  3     array snapshot     hourly    14 days    OK

Third, what the business was told, quoted from the recovery objectives circulated at the last service review:

Recovery points are retained for fourteen days at hourly granularity for
all Tier 1 systems. Recovery to any hour within that window is available
on request.

Fourth, the scheduler. There is no job writing any of the nine systems to a destination outside the array. The only export that leaves it is a monthly audit extract covering two systems.

Fifth, the array event log for the hours before the fault.

$ grep -E 'snapshot|enclosure|pool' rbdr-array-events.log | tail -6

2026-08-15T01:00:03Z INFO  snapshot create rbdr-prod/vm-ledger@hourly-0100 ok
2026-08-15T01:00:04Z INFO  snapshot destroy rbdr-prod/vm-ledger@hourly-0100-14d retention ok
2026-08-15T02:00:03Z INFO  snapshot create rbdr-prod/vm-ledger@hourly-0200 ok
2026-08-15T02:00:04Z INFO  snapshot destroy rbdr-prod/vm-ledger@hourly-0200-14d retention ok
2026-08-15T02:14:01Z ERROR enclosure 0 lost communication with both controllers
2026-08-15T02:14:11Z ERROR pool rbdr-prod suspended: no healthy paths

Sixth, the executed capture of the same arrangement on a disposable device.

$ pvs; vgs; lvs

Both origin and snapshot are extents in ONE volume group on ONE PV.
pvs before:
PV         VG     PSize   
/dev/loop5 rbdrvg 1020.00m

Simulating permanent loss of the underlying device:
re-attached the same (now destroyed) backing store as /dev/loop5

--- what survived? ---
vgs:
lvs:

Work the evidence before reading on

1. The inventory was accurate in every field it contained. What question was it answering, and what question did the incident ask?
2. The event log shows snapshot destroy lines. Do they support or eliminate the theory that an operator removed the snapshots?
3. The objectives document says fourteen days. Fourteen days of protection against what?
4. If the estate had held 336 recovery points per system for eleven months, how many did it hold against the failure that actually occurred?

Root cause

The snapshots were allocations inside the storage they protected

A snapshot records the blocks a dataset occupied at a point in time. Those blocks stay in the pool that holds the live data, are addressed by the same metadata, and are reachable only through the same storage stack. That is what makes a snapshot near-instant and near-free at the moment it is taken, and it is the same property that removes it when the pool goes.

The executed LVM capture above is the estate in miniature: origin and snapshot were extents in one volume group on one physical volume, and after the backing device was destroyed both vgs and lvs printed nothing at all. Not a corrupted snapshot, not a partial one. No rows.

The inventory was keyed on tool rather than on failure model

The protection inventory asked each system one question — what protects this? — and accepted a tool name as the answer. “Array snapshot, hourly, 14 days” is a complete and correct answer to that question, so all nine rows were legitimately green and stayed green through eleven monthly reviews.

The question it never asked was what the system survives. With no column for where the copy lands, a system whose only copy lives on the array it runs from is unrepresentable as a gap. The control was not bypassed or overridden; it was answering a question whose answer could not come out wrong.

The recovery objectives inherited the defect. A retention and a granularity with no failure model attached describes protection against an accidental deletion or an application corrupting its own data, and describes nothing about hardware loss. Nobody was misled about the number. They were misled about what it counted.

Resolution

There is no restore in this incident, and that is the finding rather than a gap in the write-up. Two systems recovered to an eleven-day-old audit extract; seven were rebuilt from configuration management to their build state with no application data. Record the recovery point actually reached per system before anything else — that measurement is what the estate change is argued from.

The structural fix is not a different snapshot technology. It is a second failure domain for the same snapshots, and the ZFS capture shows the difference the copy makes under the identical event.

$ zpool import; zfs list -r -t all rbdrbkp

--- zpool import (can ZFS find anything to import?) ---
no pools available to import

--- can the 09:00 snapshot be reached? ---
cannot open 'rbdrprod': dataset does not exist

--- what the INDEPENDENT backup pool still holds ---
NAME                  USED  REFER
rbdrbkp               100M    24K
rbdrbkp/ledger        100M  50.1M
rbdrbkp/ledger@0900  50.0M  50.1M
rbdrbkp/ledger@0930     0B  50.1M

The snapshot mechanism in that capture is the same one the array used. What differs is that zfs send had already written the points to a pool on a different device, so losing production left them readable.

Then repair the inventory, which is the defect that would otherwise recreate this state on the replacement array: add destination, failure domain and last-proven-restore columns, and let a row be green only when the destination is not the storage being protected.

Verification

Ask each system a question rather than reading a status: name the device the recovery point is stored on, and confirm it is not the device being protected. Matching names mean the row is a gap whatever its status column says.

For every Tier 1 system, zfs list -H -o name -t snapshot against the backup pool returns the expected points, and zpool status shows that pool on different physical devices from production. A restore is then performed from the off-array copy into isolated infrastructure and compared by checksum against a value recorded beforehand, never against the source at restore time.

The revised objectives document names a failure model beside every number, and the fourteen-day figure states which failures it applies to.

Finally, run this incident as a tabletop against the new inventory: mark the production array destroyed and confirm every Tier 1 row still names a readable device. Nine rows that named the array before should now name something else.

Prevention

Apply the snapshot test to every row. Assume the storage the data lives on is gone, then ask what is still readable. A snapshot is not a backup until it has been copied somewhere with an independent failure domain.

Key the inventory on the failure model. Tool, schedule and retention were all correct here. Destination, failure domain and last proven restore are the columns that could have been wrong, which is why they are the ones worth having.

Never publish a retention without a failure model. Fourteen days is not a recovery capability; fourteen days against deletion and corruption, plus a stated figure against loss of the array, is one.

Alert on the age of the off-array copy, not on the snapshot schedule. A snapshot schedule running perfectly reports that the array is healthy, which is information the estate already has.

Count recovery points per failure domain. Nine systems holding 336 snapshots each on one array is one recovery point against hardware loss, and every review should present the arithmetic that way.