Upgrading Ceph: Squid to Tentacle · Proxmox VE

Ceph major upgrades are unusual among Proxmox maintenance tasks in two ways. They are online — the cluster keeps serving I/O throughout, and no guest needs to be shut down. And they are irreversible — there is no supported downgrade at any stage, and after one specific command there is not even an unsupported one.

Those two facts together produce the characteristic mistake. Because nothing has to be shut down, the upgrade feels low-risk, so it gets scheduled loosely and run quickly. Because it cannot be undone, a mistake made quickly is a mistake you now operate around.

pveceph install defaults to --version tentacle in PVE 9.2, so Tentacle is what a new cluster gets and Squid is the explicit choice. Existing Squid clusters keep running Squid until somebody upgrades them, which is what this lesson covers.

Before you start

The Proxmox wiki states the entry conditions, and they are conditions rather than recommendations:

Every node on Proxmox VE 9.1 or newer, with pve-manager at 9.1.4 or newer.
Ceph on Squid, at 19.2.3-pve3 or newer.
“The cluster must be healthy and working!”

Read-only / Safethe pre-flight, run on every node— Read-only. All four must be satisfied before the first repository line changes. ceph versions is the one people skip, and it is the one that reveals a node somebody forgot in the last upgrade.

pveversion -v | head -3
ceph -s
ceph versions
ceph health detail

Read-only / Safe— Illustrative. A single entry under overall is the state you want before starting. Two entries means an earlier upgrade did not finish, and that has to be resolved first.

$ ceph versions

{
  "mon": { "ceph version 19.2.3 (squid) stable": 3 },
  "mgr": { "ceph version 19.2.3 (squid) stable": 3 },
  "osd": { "ceph version 19.2.3 (squid) stable": 12 },
  "mds": { "ceph version 19.2.3 (squid) stable": 2 },
  "overall": { "ceph version 19.2.3 (squid) stable": 20 }
}

Illustrative output

Set noout, and know exactly what it buys

Cluster-wide riskstop the cluster reacting to planned restarts— Cluster-wide flag. Prevents OSDs that go down from being marked out, which is what would otherwise trigger a full data rebalance. It does not stop them being marked down, and it does not stop I/O being served from surviving copies.

ceph osd set noout
ceph -s | grep -A2 'health:'

The distinction between down and out is the whole point of the flag, and it is worth being precise about:

down means the monitors have stopped hearing from an OSD. It happens the moment you restart the daemon, noout or not. Placement groups on it become degraded and are served from their other copies.

out means the OSD is removed from the CRUSH data placement calculation, so every placement group it held is re-created elsewhere. That is a full rebalance — potentially terabytes of movement across the cluster network.

Without noout, the trigger is time: mon_osd_down_out_interval defaults to “10 minutes”. An OSD restart that completes in ninety seconds never reaches it. An OSD restart on a node that hangs on a firmware prompt, or a node that reboots into a broken initramfs, does — and now you are debugging a boot problem while the cluster moves several terabytes it did not need to move.

noout costs nothing and removes that entire class of outcome. It also means that if a disk genuinely dies mid-upgrade, Ceph will not self-heal, so leaving the flag set after the window is its own hazard — which is why unsetting it is a step in the procedure rather than an afterthought.

The upgrade, in order

1. Repository

PVE 9 is Debian 13 (Trixie) and uses deb822 .sources files. Older .list files still work if a node has them.

Configuration changepoint the Ceph repository at Tentacle, on every node— Rewrites the suite name in the Ceph repository definition. Changing the repository does not install anything; the next apt full-upgrade does. Run it on every node before upgrading any of them.

if [ -f /etc/apt/sources.list.d/ceph.sources ]; then
cp /etc/apt/sources.list.d/ceph.sources /root/ceph.sources.bak
sed -i 's/squid/tentacle/' /etc/apt/sources.list.d/ceph.sources
elif [ -f /etc/apt/sources.list.d/ceph.list ]; then
cp /etc/apt/sources.list.d/ceph.list /root/ceph.list.bak
sed -i 's/squid/tentacle/' /etc/apt/sources.list.d/ceph.list
fi

grep -r tentacle /etc/apt/sources.list.d/

The resulting deb822 stanza is:

Types: deb
URIs: https://enterprise.proxmox.com/debian/ceph-tentacle
Suites: trixie
Components: enterprise
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg

or, in the legacy format:

deb https://enterprise.proxmox.com/debian/ceph-tentacle trixie enterprise

Substitute no-subscription for enterprise in both the URI component and the Components: line if that is the repository you are entitled to.

2. Install the packages, everywhere

Service impact possibleon every node— Installs the new binaries. Running daemons keep executing the old code they already loaded, so this step alone changes nothing about cluster behaviour - which is exactly why it is safe to do on all nodes before restarting anything.

apt update
apt full-upgrade

3. Monitors, one node at a time

Service impact possiblerestart monitors and confirm the release before moving on— Restarts the monitor on this node only. Wait for quorum to re-form and for min_mon_release to advance before touching the next node. With three monitors, restarting two at once loses quorum and the cluster stops.

systemctl restart ceph-mon.target

sleep 15
ceph -s | grep -E 'mon:|quorum'
ceph mon dump | grep min_mon_release

The wiki names the expected result: min_mon_release 20 (tentacle). That value advances only once every monitor has been restarted, so it is the clean signal that this stage is complete.

4. Managers

Service impact possiblerestart managers— The manager is active-standby, so a restart briefly moves the active role. Metrics and the GUI Ceph panel gap for a few seconds; no client I/O is affected.

systemctl restart ceph-mgr.target

sleep 10
ceph -s | grep -A1 'mgr:'

5. OSDs, one node at a time, with a wait between nodes

This is the long stage and the one where haste costs you.

Service impact possiblerestart every OSD on one node, then wait for clean— Restarts all OSDs on this node. Placement groups go degraded while they are down and re-peer when they return. Do not proceed to the next node until every PG is active+clean again.

systemctl restart ceph-osd.target

# wait until no PG is in a recovering or degraded state before the next node
while ceph pg stat | grep -qE 'degraded|undersized|peering|recovering|backfill'; do
sleep 10
done

ceph -s
ceph pg dump_stuck unclean

Read-only / Safe— Illustrative. One state, and it is active+clean. Any pg count in a degraded, undersized or peering state means the previous node has not finished.

$ ceph pg stat

417 pgs: 417 active+clean; 4.1 TiB data, 12 TiB used, 21 TiB / 33 TiB avail

Illustrative output

The waiting is not superstition. Each node’s OSDs coming back triggers peering and a short recovery of writes that happened while they were down. Restarting the next node during that recovery stacks two degraded windows, and on a size 3 pool the second one can take placement groups to one copy.

6. Metadata servers, if you run CephFS

MDS upgrades have their own choreography because standby-replay daemons follow the active one’s journal, and a version mismatch there is not tolerated. The wiki procedure is:

Disable standby-replay and reduce the filesystem to a single rank.
Stop the standby MDS daemons: systemctl stop ceph-mds.target.
Restart the active MDS: systemctl restart ceph-mds.target.
Start the standbys again: systemctl start ceph-mds.target.
Restore the original max_mds and standby-replay settings.

CephFS is briefly single-ranked through this, which for a Proxmox cluster using CephFS to hold ISOs and backups is unremarkable and for a CephFS serving an application is a capacity reduction to plan for.

7. The point of no return

Cluster-wide riskenable Tentacle-only features— Cluster-wide and one-way. After this, an OSD running Squid cannot rejoin the cluster. Run it only when ceph versions reports every daemon on Tentacle and every PG is active+clean.

ceph versions
ceph -s

ceph osd require-osd-release tentacle

This command tells the cluster it may use on-disk and on-wire features that older OSDs do not implement. It is the last step for a reason: an OSD that has not been upgraded, or one you later restore from an old node image, will be refused.

Everything before it is reversible in principle — you could reinstall Squid packages on a node and restart its daemons, and the cluster would tolerate the mixed versions because it is designed to during an upgrade window. Everything after it is not.

8. Clear the flag

Cluster-wide riskre-enable automatic recovery— Removes noout, restoring Ceph's ability to mark a failed OSD out and rebuild its data. Do this as soon as the upgrade is verified - a cluster left with noout set does not self-heal from a genuine disk failure.

ceph osd unset noout
ceph -s
ceph osd stat

Under the hood

Why the order is MON, then MGR, then OSD, and why the reverse breaks.

The monitors own the cluster maps — the OSD map, the monitor map, the CRUSH map, the PG map. Every other daemon and every client obtains its view of the cluster from them. Those maps are versioned, and each version can encode features that older code cannot parse.

That asymmetry sets the direction. A newer monitor can produce a map that an older OSD understands, because it will not encode new features until told to. An older monitor cannot produce a map a newer OSD needs, and more importantly, cannot correctly interpret what a newer OSD reports about itself. So monitors go first, always. This is the same reasoning that puts require-osd-release last: it is the switch that authorises the monitors to start emitting the new encodings, and it must not be thrown until every consumer can read them.

Managers sit between. They consume the maps and produce metrics, autoscaling decisions and the dashboard, and nothing in the data path depends on them — so they upgrade after the monitors and their brief absence is a monitoring gap rather than an outage.

OSDs come last among the data daemons because they are the most numerous and the only ones whose restart moves the cluster’s health needle.

The mixed-version window this creates is deliberate and bounded. ceph versions exists precisely to show it, and Ceph supports running one major version apart during an upgrade. It does not support it as a steady state: feature negotiation across the mixed set means the cluster runs at the capability of its oldest member, and the health check ceph health detail will eventually complain about daemons that have not been restarted. Finishing the upgrade is part of the upgrade.

One more mechanism explains why step 2 is safe to run everywhere at once. apt full-upgrade replaces the files on disk. A running daemon holds its executable mapped and keeps executing the code it loaded at start. Installing Tentacle binaries under a running Squid daemon changes nothing about that daemon’s behaviour — the version only changes at systemctl restart. That decoupling is what lets you stage the packages across the estate in one pass and then control the actual upgrade, precisely, with restarts.

What happens if you need to roll back?

Mostly, you do not get to. It is worth being exact about where the door closes, because “we can roll back” is a claim that appears in change records for this work and is usually false.

Before any restart. The new packages are installed and no daemon is running them. Reverting the repository and running apt install ceph=<old-version> is untested but mechanically plausible, and nothing about the cluster has changed. This is the only stage that is genuinely safe.

After some daemons are restarted, before require-osd-release. The cluster is running mixed versions, which it supports. You can stop, and you can leave it mixed for hours while you investigate. Downgrading a restarted daemon back to Squid is not supported by upstream and not tested by Proxmox; people have done it and people have also corrupted OSD metadata doing it. Treat it as a last resort, not a plan.

After require-osd-release tentacle. There is no rollback. Not unsupported — absent. Squid OSDs are refused by the cluster. The only route back is rebuilding the cluster on Squid and restoring every guest from backup, which for a cluster holding tens of terabytes is a multi-day operation and a real RPO loss.

So the rollback plan is not a rollback plan. It is three other things:

A change window with slack. The stage that takes unpredictable time is the OSD restarts, because each node’s recovery depends on how much was written while it was down. Size the window for the slowest node times the node count, not the fastest.

Restorable backups, verified before you start. Not because you expect to use them, but because the honest failure mode here is “restore from PBS”, and a restore path you have not tested is not a path. The PBS verification job is the evidence.

A stopping point. You can pause indefinitely between any two stages. Deciding in advance that “if ceph -s is not clean within 20 minutes of a node’s OSDs restarting, we stop for the night with the flag set and investigate” is worth more than any rollback command, because it is the decision people fail to make at 2 AM.

Verifying the upgrade actually finished

Read-only / Safethe closing checks— Read-only. Each answers a different question: did every daemon move, is the cluster clean, did the release marker advance, and is the flag cleared.

ceph versions
ceph -s
ceph mon dump | grep min_mon_release
ceph osd dump | grep require_osd_release
ceph osd stat

The one that catches the common miss is ceph versions. A node that was powered off during the window, or an OSD whose systemctl restart was issued while the unit was masked, will still be on Squid — and it will work, because mixed versions are tolerated, right up until require-osd-release is set and it is refused on its next restart.

Check ceph versions before step 7, not after.

Common mistakes

Starting on HEALTH_WARN. Every OSD restart drops replication by one; starting from degraded can take a pool below min_size.
Not setting noout. A node that takes longer than ten minutes to come back triggers a full rebalance you did not need.
Leaving noout set afterwards. The cluster stops self-healing from real disk failures, silently.
Restarting two monitors at once. With three monitors that is quorum loss, and the cluster stops serving I/O.
Moving to the next OSD node before active+clean. Two overlapping degraded windows on a size 3 pool can reach one copy.
Running require-osd-release before checking ceph versions. A missed node becomes an OSD that can never rejoin.
Accepting apt full-upgrade without reading the package list. It upgrades PVE and the kernel too.
Writing “roll back if issues” in the change record. After step 7 there is no rollback, and the change record should say so.

Key takeaways

Entry conditions are conditions: PVE 9.1+, pve-manager 9.1.4+, Ceph 19.2.3-pve3+, and a healthy cluster.
noout prevents out, not down. It stops a slow restart from triggering a full rebalance, and it must be cleared afterwards.
Order is MON → MGR → OSD → MDS, because the monitors own the cluster maps that everything else reads.
Install packages everywhere first; nothing changes until a restart.
Wait for active+clean between OSD nodes. min_mon_release 20 (tentacle) confirms the monitor stage.
ceph osd require-osd-release tentacle is the point of no return. Verify ceph versions first.
There is no supported downgrade at any stage, and none at all after step 7. Plan a stopping point instead of a rollback.

Knowledge check

Why this matters

Most of what makes this upgrade risky is not technical. The procedure is eight steps, they are documented, and each one has a check that either passes or does not.

What makes it risky is that it feels routine. No guest is shut down. No storage is unmounted. ceph -s says HEALTH_OK at the start and, if you do it right, at the end. Everything about the experience resembles applying updates, and applying updates is something people do quickly and in parallel because that is how you get through fifty nodes before lunch.

Ceph punishes exactly that instinct. Restarting two monitors in parallel loses quorum. Restarting two OSD nodes in parallel stacks degraded windows. Running require-osd-release before checking ceph versions strands a node permanently. Each of those is a five-second action whose consequence is measured in hours, and each of them is the faster way to do the step.

The habit worth building is the one this lesson repeats between every stage: after you change something, run the command that would tell you if it went wrong, and read the answer before continuing. ceph -s after the monitors. active+clean after each OSD node. ceph versions before the irreversible step. Four or five extra minutes across the whole window.

And write the change record honestly. “Rollback: restore from PBS, multi-day, RPO loss” is an uncomfortable line to put in front of a change board, and it is the true one after step 7. A board that knows that will give you the window you need. A board told “roll back if issues” will give you two hours on a Tuesday.