Establishing the pre-upgrade baseline

Why this matters in production

Without a baseline, “is this normal?” after an upgrade has no answer, and every observation becomes a debate.

What to capture

BASE=/var/lib/ceph-upgrade-baseline
mkdir -p "$BASE"

ceph -s                          > "$BASE/status.txt"
ceph health detail               > "$BASE/health.txt"
ceph versions                    > "$BASE/versions.txt"
ceph df detail                   > "$BASE/df.txt"
ceph osd df tree                 > "$BASE/osd-df.txt"
ceph osd perf                    > "$BASE/osd-perf.txt"
ceph osd pool ls detail          > "$BASE/pools.txt"
ceph config dump                 > "$BASE/config.txt"
ceph orch ls                     > "$BASE/services.txt"
ceph orch ps                     > "$BASE/daemons.txt"
ceph features                    > "$BASE/features.txt"
ceph osd getcrushmap -o "$BASE/crush.bin"
crushtool -d "$BASE/crush.bin" -o "$BASE/crush.txt"
ceph auth ls                     > "$BASE/auth.txt"

The last is sensitive — it contains keys. Store the baseline where the
cluster's other secrets are stored, not in a ticket attachment.

Capturing performance

# a short workload sample, so post-upgrade numbers have a comparison
for i in $(seq 1 20); do
  ceph -s --format json | python3 -c '
import sys,json
d=json.load(sys.stdin)["pgmap"]
print(d.get("read_bytes_sec",0), d.get("write_bytes_sec",0),
      d.get("read_op_per_sec",0), d.get("write_op_per_sec",0))'
  sleep 15
done > "$BASE/throughput.txt"

ceph osd perf --format json > "$BASE/latency.json"

Where to store it

Not on a cluster host that the upgrade touches.
Not only in a ticket, which may not survive.
On a management host, in the same place other operational state lives,
with the timestamp in the path.

tar czf "/backups/ceph-baseline-$(hostname -s)-pre-upgrade.tar.gz" -C "$BASE" .

Using it afterwards

diff <(ceph osd pool ls detail) "$BASE/pools.txt"
diff <(ceph config dump) "$BASE/config.txt"
diff <(crushtool -d <(ceph osd getcrushmap -o /dev/stdout) -o /dev/stdout) "$BASE/crush.txt"

Differences in the pool and config dumps after an upgrade are worth
explaining. Some are expected — a new option with a default that now
appears — and some are not.

Quiz

Production discipline

Capture the performance baseline under the cluster’s normal workload, not during a quiet window — it is what makes a post-upgrade “it feels slower” report answerable. Store the archive off the cluster hosts, and treat the ceph auth ls output as the secret it is.

Cross-course references

• Kubernetes: pre-upgrade metric snapshots serve the identical comparison role
• Linux: baselines exist to make subjective reports measurable