Repository snapshots - making a wave reproducible

The canary ran on Monday. Twenty hosts took the update, health checks were green for 24 hours, and the change was approved to proceed. Production ran on Thursday.

Production installed a different set of packages, because upstream published four new versions between Monday and Thursday and nothing in the process said otherwise. The canary validated a package set that no longer exists.

This is not a hypothetical failure. It is the default behaviour of every wave strategy built on top of a live repository, and it quietly removes the value from the most expensive part of the process.

Demonstrating the drift

You do not need an incident to see it. Ask two hosts in different waves what they would install:

$ apt-get --simulate upgrade | grep '^Inst' | head

Inst libssl3 [3.0.11-1~deb12u2] (3.0.14-1~deb12u2 Debian-Security:12/stable [amd64])
Inst openssl [3.0.11-1~deb12u2] (3.0.14-1~deb12u2 Debian-Security:12/stable [amd64])
Inst curl [7.88.1-10+deb12u5] (7.88.1-10+deb12u7 Debian-Security:12/stable [amd64])

# Capture the plan as a comparable artefact, per wave
apt-get --simulate upgrade | awk '/^Inst/ { print $2, $4 }' | sort > /tmp/plan.txt

# RHEL family
dnf --assumeno upgrade 2>/dev/null | sed -n '/^Upgrading:/,/^Transaction Summary/p'

If the canary plan and the production plan differ, the canary result does not transfer. That diff is the measurement, and most teams have never taken it.

The snapshot-and-promote model

The fix is to stop treating the repository as a moving target. Take a point-in-time copy, give it a name, and let each tier resolve to a different name.

upstream (moves continuously)
        |
        v  mirror + snapshot, weekly
   snap-2026-08-11  -->  published as  "dev"
   snap-2026-08-04  -->  published as  "staging"
   snap-2026-07-28  -->  published as  "production"

Three properties fall out of this, and they are the whole point:

• Promotion is a metadata operation. Moving staging forward means republishing the staging distribution to point at a different snapshot. No host is touched.
• The wave is reproducible. Production installs exactly what staging installed, because it resolves the identical package set - not the same version numbers by coincidence.
• Rollback is instant for hosts that have not patched yet. Republish the previous snapshot and the remaining waves stop receiving the bad package.

Note the limit on that last one. Rolling the snapshot back does not un-install anything from hosts that already applied it; those need the package rollback procedure from earlier in this part. What the snapshot rollback buys is that the blast radius stops growing while you work.

Building snapshots

Debian family

For ad-hoc reproduction of a past state, Debian publishes an archive of the archive:

Types: deb
URIs: https://snapshot.debian.org/archive/debian/20260728T000000Z/
Suites: bookworm
Components: main
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg
Check-Valid-Until: no

That service is excellent for reproducing a build and unsuitable as a fleet’s production mirror - it is a shared community resource, it is slow under load, and your fleet hitting it on every apt run is not a reasonable thing to do.

For a fleet, mirror it yourself. aptly is the common choice because its model is snapshots rather than directories:

# One-time: define what you mirror
aptly mirror create -architectures=amd64 bookworm-security \
    http://security.debian.org/debian-security bookworm-security main

# Each cycle: pull, freeze, publish under a tier name
aptly mirror update bookworm-security
aptly snapshot create snap-2026-08-11 from mirror bookworm-security
aptly publish snapshot -distribution=dev snap-2026-08-11

# Promotion: repoint an existing published distribution
aptly publish switch staging snap-2026-08-04

aptly publish switch is the promotion. It rewrites the published metadata in place; hosts pick up the new package set at their next apt update and nothing about their configuration changed.

Hosts point at the tier, never at a snapshot:

Types: deb
URIs: https://packages.example.com/debian
Suites: production
Components: main
Signed-By: /etc/apt/keyrings/example-archive.gpg

RHEL family

reposync plus createrepo_c gives the same shape with more assembly:

sudo dnf reposync --repoid=rhel-9-baseos --download-metadata \
    --newest-only --download-path=/srv/repo/snap-2026-08-11

sudo createrepo_c --update /srv/repo/snap-2026-08-11/rhel-9-baseos

# Promotion is a symlink swap, served by the web server
sudo ln -sfn /srv/repo/snap-2026-08-04 /srv/repo/staging

Pulp and Katello implement the model properly with content views and lifecycle environments, and deduplicate the package pool across snapshots. If the fleet is large enough that disk is a concern, that is the reason to use them: a naive cp per snapshot stores the same packages many times, while snapshot-aware tools store one copy and many sets of metadata.

You become the signing party

Republishing a snapshot means re-signing it. Your fleet now trusts your key for those packages, and the chain of trust runs through your mirror host rather than through Debian or Red Hat.

That is a legitimate design, and it carries the obligations from the repository security part:

• The signing key lives on the publishing host, not in the configuration management repository, and ideally not on a host that also serves the packages.
• Hosts reference it with Signed-By: scoped to your source.
• Verify upstream signatures at mirror time. aptly mirror update verifies the upstream release signature; if you assemble a mirror with rsync or wget and re-sign the result, you have converted “Debian signed this” into “our build host signed this” and thrown the original assurance away.

A snapshot pipeline that skips upstream verification is the supply-chain equivalent of pinning a repository without checking its key: the mechanism is in place and it is authenticating the wrong thing.

Emergency updates need a bypass

A frozen production snapshot means a critical CVE fix does not reach production until the next promotion. That is unacceptable, and the answer is not to unfreeze.

Build the bypass in advance:

1. Take a fresh snapshot from upstream security.
2. Create a hotfix snapshot containing the previous production set plus only the fixed packages. aptly merges and filters snapshots for exactly this; on the RHEL side it is a small repo carrying the fixed RPMs at higher priority.
3. Publish it to the production distribution.
4. Run it through the emergency patching procedure from the vulnerability part - it is still a change, and it still wants a canary, just a compressed one.
5. Fold the fix into the normal promotion at the next cycle so the hotfix repo does not become permanent.

The property worth protecting is that production received one extra package and nothing else. A full unfreeze under time pressure delivers a hundred untested upgrades alongside the fix, during an incident, which is how a security fix becomes an outage.

Knowledge check