Major upgrades: PVE version upgrades and Ceph upgrades

Why this matters in production

A botched major upgrade can take down the entire cluster. Planned, tested upgrades are routine.

Major PVE upgrade overview

A major PVE upgrade changes the underlying Debian release (e.g. Bookworm to Trixie) and may include schema migrations.

Process:

1. Read the official upgrade guide.
2. Verify the cluster is healthy.
3. Test in a non-production cluster.
4. Schedule a maintenance window.
5. Upgrade one node at a time.
6. Validate after each node.

pveversion -v

apt update && apt list --upgradable | grep pve

Ceph upgrade

Ceph upgrades are also major operations. Ceph uses a phased upgrade process:

1. Set the require_osd_release flag to prevent OSDs from joining with mismatched versions.
2. Upgrade MONs (one at a time).
3. Upgrade MGRs.
4. Upgrade OSDs in batches.
5. Optionally upgrade MDSs.
6. Set require_osd_release to the new version.

ceph versions

ceph osd require-osd-release nautilus

Rolling upgrade of Ceph

sequenceDiagram
  participant Ops as Operator
  participant MON1 as MON 1
  participant MON2 as MON 2
  participant MON3 as MON 3
  participant OSD as OSD

  Ops->>MON1: upgrade
  Note over MON1: restarted with new version
  Ops->>MON2: upgrade
  Note over MON2: restarted
  Ops->>MON3: upgrade
  Note over MON3: restarted
  Ops->>OSD: upgrade batch 1
  Ops->>OSD: upgrade batch 2

Each step validates the cluster remains healthy (ceph -s).

Testing before upgrading

A non-production cluster must:

• Match the production cluster’s hardware and configuration.
• Run a representative workload.
• Be upgrade-tested before production.

If you don’t have such a cluster, build one.

Rollback

Major upgrades cannot be cleanly rolled back. The standard mitigation is:

• Snapshot VM disks before upgrade (snapshot mode).
• Backup to PBS before upgrade.
• Have an off-site PBS that can restore from.

If the upgrade fails, the recovery is:

• Reinstall the previous version.
• Restore from PBS.
• Investigate the failure.

This is why testing in a non-production cluster is critical.

Production considerations

Common mistakes

• Upgrading all nodes simultaneously.
• Skipping the non-production test.
• Not having a documented rollback path.

Key takeaways

• Test before upgrading.
• Upgrade one node at a time.
• Backup before upgrading.
• Follow the official guide exactly.

Knowledge check