Rollback strategies - safe deployment reversals · Linux

Every deployment should have a tested rollback procedure. If you cannot roll back, the change is a one-way door.

Package rollback

RHEL family

# View history - find the transaction id
sudo dnf history list

# Confirm what that transaction actually did
sudo dnf history info 42

# Invert exactly that transaction
sudo dnf history undo 42

# Invert the most recent transaction
sudo dnf history undo last

dnf history records every transaction, including the packages and versions involved.

undo and rollback are not synonyms

dnf has two subcommands here and they do different things:

Command	What it does
`dnf history undo <id>`	Performs the opposite of the operations in that one transaction
`dnf history rollback <id>`	Undoes every transaction performed after `<id>`, returning the system to its state at that point

Two consequences follow, and both bite during an incident.

dnf history rollback last is a no-op. last is the most recent transaction, and there is nothing after it, so dnf has nothing to undo. It exits successfully. An operator who runs it reports the bad upgrade reverted while the regressed package is still installed and the outage continues.

dnf history rollback 42 on a host that has had thirty transactions since #42 reverts all thirty — including the security updates that had nothing to do with your change. That is a much larger change than the one you are trying to back out, applied to a production host mid-incident.

Use undo to back out a single change. Reach for rollback only when you genuinely intend to return the whole package set to a known point in time, and then read dnf history info for every transaction in the range first.

Debian / Ubuntu

# Substitute your own values before running:
PACKAGE=nginx
OLD_VERSION=1.24.0-1

# What changed, from which version to which. history.log records the
# transaction; dpkg.log records individual package state transitions and is
# far harder to read under pressure.
grep -A3 '^Start-Date' /var/log/apt/history.log | tail -40

# Is the previous version still obtainable? Check BEFORE you need it.
apt-cache policy "$PACKAGE"
ls /var/cache/apt/archives/"$PACKAGE"_*.deb

# Downgrade, then freeze - a downgrade alone is undone by the next upgrade
sudo apt-get install -y --allow-downgrades "$PACKAGE"="$OLD_VERSION"
sudo apt-mark hold "$PACKAGE"
apt-mark showhold
dpkg -l "$PACKAGE" | tail -1        # confirm the version actually in place

The apt rollback fails in three different ways, all of them at the worst moment

apt has no transaction rollback. What it has is “install an older version”, and each of the three steps above exists because one of them fails silently otherwise.

-y without --allow-downgrades aborts. apt refuses with E: Packages were downgraded and -y was used without --allow-downgrades. In a script this is a non-zero exit at the exact moment you needed the rollback to work.

Without apt-mark hold, the rollback is temporary. The package’s candidate version is still the newer one, so the next apt upgrade — or the next unattended-upgrades run, which needs no operator at all — reinstalls the regression. The rollback silently undoes itself, typically overnight.

The old version may simply not exist any more. Ubuntu’s -security and -updates pockets supersede rather than retain, so the version you want is often gone from the archive within days. /var/cache/apt/archives keeps it only until something runs apt clean. Check apt-cache policy before you plan around a downgrade; if the answer is “not available”, the recovery path is a restore or a rebuild, not a downgrade, and you want to know that now rather than during the incident.

One more thing a downgrade is not: a clean reversal. It runs the new package’s prerm against a system the old package’s postinst will then configure. For anything holding on-disk state — a database server, a schema-migrating application — assume the downgrade will not undo what the upgrade did to the data.

RHEL’s dnf history undo is a genuine transaction reversal and is the closer analogue of a rollback; the section above covers it.

Configuration rollback

Configuration management tools make this easy:

# Substitute your own values before running:
COMMIT=a1b2c3d

# Ansible
ansible-playbook -i prod rollback.yml

# Revert a specific commit
git revert "$COMMIT"
ansible-playbook -i prod playbook.yml

The principle: configuration is in version control. Revert the change, apply the previous version.

Container rollback

# Substitute your own values before running:
PREVIOUS_TAG=v1.4.2

# Kubernetes
kubectl rollout undo deployment/myapp

# To a specific revision
kubectl rollout undo deployment/myapp --to-revision=3

# Docker
docker run myapp:"$PREVIOUS_TAG"

Container rollback is fast and safe; the previous image is known-good.

Database migration reversal

For schema changes, the rollback must include:

Revert the schema (drop new columns, drop new tables).
Revert the application code that used the new schema.

Test the rollback in staging:

Apply migration.
Verify application works.
Roll back migration.
Verify application still works (with old code).

A migration that cannot be rolled back is dangerous. Prefer forward-only migrations (add columns, never drop) for production.