What to back up before an upgrade, and what a backup cannot do

Why this matters in production

The word “backup” in an upgrade plan often means two very different things, and conflating them produces a plan that does not work.

Two distinct things

Configuration backup is small, fast, and always worth taking.
Data backup is large, slow, and is the application's own concern.
An upgrade plan needs to be clear about which it has.

Configuration and metadata

BK=/backups/ceph-pre-upgrade
mkdir -p "$BK"

ceph auth export                 > "$BK/auth-export.txt"
ceph osd getcrushmap -o "$BK/crush.bin"
crushtool -d "$BK/crush.bin"     -o "$BK/crush.txt"
ceph config dump                 > "$BK/config.txt"
ceph osd pool ls detail          > "$BK/pools.txt"
ceph fs dump                     > "$BK/fs.txt"
ceph mon dump                    > "$BK/monmap.txt"
ceph orch ls --export            > "$BK/services.yaml"
ceph orch host ls --format yaml  > "$BK/hosts.yaml"

chmod 600 "$BK"/*

The auth export contains every key in the cluster and is the single most sensitive file a Ceph operator handles.

What each enables

# the service spec is directly re-appliable
ceph orch apply -i "$BK/services.yaml"

What backup cannot recover

None of the above restores data.
A restored configuration on empty OSDs is an empty cluster with the
right shape.

This is why the data protection question is answered separately, by
application-level backups, mirroring, or a second cluster — not by the
upgrade plan.

Verifying the backups are usable

# the CRUSH map recompiles
crushtool -c "$BK/crush.txt" -o /tmp/crush-verify.bin && echo "crush OK"

# the service spec parses
python3 -c 'import yaml,sys; yaml.safe_load(open(sys.argv[1])); print("yaml OK")' \
  "$BK/services.yaml"

# the auth export is complete
grep -c '^\[' "$BK/auth-export.txt"
ceph auth ls --format json | python3 -c '
import sys,json; print(len(json.load(sys.stdin)["auth_dump"]))'

The two counts should agree. A truncated auth export is a backup that
looks fine until it is needed.

Quiz

Production discipline

Be explicit in the upgrade plan about which kind of backup exists. Configuration backup restores the cluster’s shape and is always worth taking; data protection is a separate concern answered by application backups or mirroring. Verify the CRUSH map recompiles and the auth export count matches before relying on either.

Cross-course references

• Kubernetes: etcd backup restores cluster state, not the PersistentVolume contents
• Linux: configuration backup and data backup are distinct disciplines