Rollback: the operation that destroys newer data · Backup & DR

The snapshot test ended with a snapshot nobody could reach, because the pool holding it had gone. The failure in this lesson is the reverse. The snapshot is present, it holds exactly the state somebody wants, and the act of using it is what causes the loss. Rollback is the only snapshot operation that writes to production, it is reached for by people who believe they are reading, and the distinction decides whether an incident ends with one file recovered or with a morning of unrelated work gone.

Three operations wearing one sentence

“Restore it from the snapshot” gets said in incident channels to mean three operations that differ in what they touch, what they cost and whether they can be undone. Separating them is most of the skill.

The first is reading from a snapshot. A clone, a read-only subvolume, or the .zfs/snapshot path covered in the ZFS lesson all present the historical state as something you can open, diff and checksum while the live data carries on unchanged. Two points in time exist side by side and nothing in production is written. It is the operation to reach for first, because it is the only one that answers “what did this actually look like?” without also committing to an answer.

The second is extracting objects out of a snapshot. Having read the old state, you copy the specific files that need to come back into the live dataset. Production changes, so this is a write — but its extent is exactly the set of paths you named, it is visible in the copy command itself, and the previous contents of those paths are still in every snapshot taken since. In the great majority of incidents this is what was actually wanted, and it is what “restore that file from the snapshot” means when a person says it.

The third is rolling the dataset back. The dataset stops presenting its current contents and presents the snapshot’s contents instead. Nothing is selected, nothing is merged and nothing is asked. The unit of the operation is the dataset, and its boundary in time is the snapshot — which is to say the extent of the change is defined by a clock rather than by the fault you are responding to.

That last sentence is the whole lesson. A rollback is not scoped to the incident, because a snapshot does not know what the incident was.

What “everything since the snapshot” actually contains

The interval between the snapshot and now is rarely as empty as it feels at 03:00. It is worth enumerating what typically lives in it, because the enumeration is what turns an obvious decision into an uncomfortable one.

If the dataset is shared — and datasets are shared far more often than their owners remember, because the layout was decided at provisioning time by somebody sizing volumes rather than modelling recovery — the interval contains other services’ writes. Uploaded files, spooled mail, queued jobs waiting for a worker, a log the auditors care about, an application’s own state directory. None of it is related to the fault. All of it goes.

The interval also contains changes made by the response itself. A rollback proposed forty minutes into an incident discards the diagnostic output someone redirected to a file on that dataset, and the configuration change that stopped the bleeding. The response window is exactly the period an operator is least likely to inventory and most likely to have written into.

Then there is the state that lives on the other side of a boundary the filesystem cannot see. A dataset holding user-uploaded documents almost always has a database somewhere holding rows that point at them. Roll the filesystem back four hours and the database does not move: rows written in those four hours now reference files that no longer exist, and files restored by the rollback are referenced by rows that were deleted. Neither system reports an error. The application starts, serves traffic, and is wrong in a way that surfaces days later as user complaints about missing attachments.

`zfs rollback` makes you destroy the record of what it discards

The mechanism is worth reading precisely rather than paraphrasing, because the flags are where the irreversibility enters.

zfs-rollback(8) states that when a dataset is rolled back, all data that has changed since the snapshot is discarded and the dataset reverts to the state at the time of the snapshot. By default the command refuses to roll back to a snapshot other than the most recent one. To reach an older snapshot, all intermediate snapshots and bookmarks must be destroyed, which is what -r does. -R goes further and destroys the clones of those snapshots as well, and -f is used with -R to force an unmount of clone filesystems that are to be destroyed.

Read that ordering carefully. The newer snapshots are the only remaining representation of the interval the rollback is about to discard. -r is therefore not a convenience flag that suppresses a warning; it is the price of the operation, and paying it is what converts a reversible decision into a final one.

Destructivethe shape of the command, and the flag that does the damage— DOCUMENTATION VERIFIED against zfs-rollback(8). No output prop: this course captured no rollback run, so only the command form is shown, with the flag behaviour quoted from the manual page.

$ zfs rollback -r rbdrprod/ledger@0900

The refusal that -r overrides is a genuine safety property, and so is the one that -R overrides. The ZFS lesson established from zfs-destroy(8) that ZFS will not destroy a snapshot while a clone of it exists, and -R is documented as the option that destroys the clones of the newer snapshots as well — so an operator who cloned the current state before proposing the rollback has, as a side effect, armed a dependency that a plain -r has to be escalated past.

There is one more clause that catches people out. zfs-rollback(8) notes that -r and -R do not recursively destroy the child snapshots of a recursive snapshot: only direct snapshots of the specified filesystem are destroyed by either option, and to completely roll back a recursive snapshot you must roll back the individual child snapshots. A parent filesystem with children — which is the ordinary shape of a ZFS estate — therefore does not roll back as a unit. Roll back the parent and the children stay where they are, leaving one namespace holding two different points in time.

Why a rollback cannot be undone once the newer snapshots are gone

Copy-on-write is what makes both the snapshot and the rollback cheap, and it is also what makes the rollback final.

A write in a copy-on-write filesystem never modifies a live block. It allocates a new block, writes into it, and updates the metadata that points at it. The old block is released only when nothing references it any more. A snapshot is a reference to the metadata as it stood at an instant, so while the snapshot exists, both the old blocks and the new ones are allocated and reachable. The pool holds the union of the two states, which is why you can read from a snapshot and from the live dataset at the same time.

A rollback does not copy anything back. It discards the newer metadata tree and re-points the dataset at the older one. Every block that only the newer tree referenced loses its last holder at that moment and returns to free space, where it is available to the next write that needs an allocation. Nothing sits underneath to be recovered from, because there is no layer beneath the allocator that remembers what a freed block used to hold.

This is why the flag ordering matters so much. The newer snapshots are precisely the references that would have kept those blocks alive, so a rollback to a non-latest snapshot cannot proceed until they are gone — and once they are gone the blocks have no holder left. The command does not become dangerous when it runs; it becomes dangerous when -r clears the path for it.

The corollary is the useful one. State that has been copied into a different dataset, or better a different pool, sits in an allocation domain the rollback cannot reach. It is not a reference to the same blocks, it is a second set of blocks, and it survives for the same reason the replicated copy survived losing the production pool in the previous lessons. That is the only fallback with a mechanism behind it rather than a hope.

One last consequence worth writing into the change record: -r destroys bookmarks as well as snapshots. Anything downstream that referred to those names — a replication schedule, a retention job, a runbook step — is now pointing at objects that do not exist, and will report that fact on its own schedule rather than yours.

Btrfs has no rollback verb, and the composition is the safer shape

The subcommands btrfs-subvolume(8) documents are the ones you would expect — create, snapshot, delete, list, show, sync, set-default and get-default among them — and no rollback appears anywhere in that set. What Btrfs administrators call a rollback is therefore a composition of the operations that do exist, and the composition has a property the ZFS command does not.

The shape is to take a writable snapshot of the read-only snapshot holding the state you want, move the live subvolume out of the way under a new name, and put the new subvolume in its place. For a root filesystem the equivalent step is btrfs subvolume set-default, which takes effect at the next mount.

LIVE=/mnt/prod/ledger
GOOD=/mnt/prod/ledger-0900
KEEP="$LIVE-superseded-$(date -u +%Y%m%dT%H%M%SZ)"

btrfs subvolume snapshot "$GOOD" /mnt/prod/ledger-restored
mv "$LIVE" "$KEEP"
mv /mnt/prod/ledger-restored "$LIVE"

Nothing in that sequence destroys anything. The superseded state is still a subvolume with a name, still mountable, still readable, and it stays that way until somebody runs btrfs subvolume delete against it. The reversibility that ZFS requires you to engineer in advance, Btrfs gives you as a by-product of having no rollback verb — provided the live subvolume is renamed rather than deleted, which is the single decision the whole property rests on.

Two prices come with it. The first is capacity: extents referenced only by the superseded subvolume are not freed while it exists, so an estate that never closes out its swaps accumulates -superseded subvolumes holding space open, in the same way the forgotten clone pinned its origin snapshot in the ZFS lesson. The second is replication: an incremental btrfs send -p names a parent subvolume, and in this course’s send/receive capture that parent — ledger-0900 — was present on the backup filesystem before ledger-1000 was sent against it. A swap that eventually deletes the old live subvolume removes the subvolume the sending side had been naming, so the next incremental has to be re-established rather than continued.

Making the decision reversible before you make it

The procedure is short, and every step exists to stop the previous one being skipped under pressure.

Start by naming exactly what has to come back: which paths, at which point in time, and how you will know afterwards that it is correct. A checksum, a row count, a person who can look at the thing and confirm it. If that list cannot be written down, the incident is not yet understood well enough to justify an irreversible operation.

Then ask whether returning those objects genuinely requires the dataset to go back. Usually it does not, and the extraction path is available:

DATASET=rbdrprod/ledger
SNAP="$DATASET@0900"
CLONE=rbdrprod/inspect0900

zfs clone "$SNAP" "$CLONE"
FROM=$(zfs get -H -o value mountpoint "$CLONE")
INTO=$(zfs get -H -o value mountpoint "$DATASET")

cp -a "$FROM/orders.csv" "$INTO/orders.csv.from-0900"
zfs destroy "$CLONE"

zfs-clone(8) creates a writable dataset whose initial contents are the snapshot’s, sharing blocks with it rather than copying them, so this costs almost nothing and touches production only at the path in the cp. The recovered file arrives under a new name so that the operator, not the command, decides when it replaces the live one.

Rollback is the right operation when the dataset is the object: a virtual machine disk, a dataset that exists to hold one application’s entire state, an in-place upgrade that failed halfway. When that is the case, put the current state somewhere the rollback cannot reach before running it — and note that a local snapshot does not qualify, because -r destroys it along with everything else newer than the target.

DATASET=rbdrprod/ledger
TARGET="$DATASET@0900"
PRE="$DATASET@pre-rollback"

zfs snapshot "$PRE"
zfs send "$PRE" | zfs receive rbdrbkp/ledger-pre-rollback

zfs rollback -r "$TARGET"

The zfs send in the middle is the step that makes the sequence a decision rather than a commitment. Without it the two lines around it are a rollback with extra ceremony.

Finally, treat rollback on a shared dataset as a service-affecting change and authorise it accordingly. It takes back writes belonging to people who were not consulted, and it does so on the authority of whoever happened to have the shell open. The authorisation a restore gets — a named approver, a stated window, a stated blast radius, a stated way back, and the affected teams told before rather than after — is the correct floor, because the blast radius of a rollback is measured the same way and the way back is thinner.

Production discipline

Name the objects before naming the operation. Write down which paths have to come back, to which point in time, and what evidence will show the result is correct. An operation chosen before that list exists is a guess with root privileges.
Default to extraction, and make rollback justify itself. Clone the snapshot, copy out what you named, destroy the clone. Reserve rollback for the cases where the dataset genuinely is the unit of recovery, and say out loud which case you are in.
Move the current state out of reach before you roll back. A snapshot of the present is destroyed by the same -r that permits the rollback, so replicate it to another dataset or pool first and confirm it arrived.
Enumerate what else lives in the dataset. List the services, paths and downstream systems that will be taken back with the target, including databases holding references to files the rollback will remove, and get that list in front of the people who own them.
Authorise a rollback as a change, not as a command. Named approver, stated window, stated blast radius, stated reversal path, affected teams informed beforehand — the same floor a restore gets, applied to an operation that is harder to walk back.

Cross-course references

Linux for Production Sysadmins — Part XVI (LVM) covers the volume manager whose snapshots this course measured being invalidated once their copy-on-write exception store filled; the same conclusion governs rollback, because there as here the object holding the older state lives in the same allocation domain as the origin, so a fallback has to be built outside the volume group rather than assumed inside it.
Proxmox VE for Production Operators — Part IX (Virtual Machines) covers the case where rollback is genuinely the right operation, because a guest disk is a dataset that holds one machine’s entire state; the decision procedure here is what tells you whether the estate you are looking at is that case or a shared dataset wearing the same storage.
PostgreSQL for Production Sysadmins — Part XIII (Backup, Archiving and Point-in-Time Recovery) supplies the other half of the cross-boundary problem raised above: rolling a filesystem back four hours while the database stays where it is produces dangling references in both directions, and recovering the pair to one consistent instant is a point-in-time recovery problem rather than a snapshot one.