Proxmox-integrated Ceph: install and configure

Why this matters in production

A misconfigured Ceph cluster can appear to work for months, then fail catastrophically during a recovery event. The install and initial configuration is when most mistakes are made; this lesson prevents them.

Pre-install checklist

Before running the Ceph wizard:

• At least 3 (preferably identical) nodes with enterprise SSDs.
• Dedicated 10 GbE+ network for Ceph.
• HBA (not RAID) attached to OSD disks.
• intel_iommu=on or amd_iommu=on (if also doing PCI passthrough).
• DNS forward and reverse for every node.
• NTP working on every node.
• All nodes joined to the cluster.

Step 1: Install Ceph packages

GUI: Datacenter → Ceph → Install. CLI on each node:

pveceph install

This installs the Ceph daemons and the ceph CLI. In PVE 9.2 the defaults are --version tentacle and --repository enterprise, so a node without an enterprise subscription needs the repository stated explicitly:

pveceph install --version tentacle --repository no-subscription

Step 2: Initialise the cluster

On one node:

pveceph init \
--network 192.0.2.0/24 \
--cluster-network 198.51.100.0/24 \
--size 3 \
--min_size 2

The documentation describes what this produces: “This creates an initial configuration at /etc/pve/ceph.conf with a dedicated network for Ceph. This file is automatically distributed to all Proxmox VE nodes, using pmxcfs. The command also creates a symbolic link at /etc/ceph/ceph.conf, which points to that file.”

The defaults are --size 3, --min_size 2 and --pg_bits 6. They are correct, and passing them explicitly means the values are in your build record rather than implied by the version you happened to install on.

Step 3: Create MONs and MGRs

On each of the other two nodes:

pveceph mon create

pveceph mgr create

Now you have 3 MONs and 3 MGRs across 3 nodes — quorum survives one node loss.

ceph -s

Step 4: Create OSDs

OSDs are the storage workhorses. Create one OSD per physical disk:

pveceph osd create /dev/sdb

For high-performance OSDs, separate DB/WAL on a faster device:

pveceph osd create /dev/sdb -db_dev /dev/nvme0n1 -wal_dev /dev/nvme1n1

Step 5: Create pools

Default pool configuration is conservative:

pveceph pool create vm-storage --size 3 --min_size 2 --pg_num 128 --add_storages

Note that pveceph pool create defaults --pg_autoscale_mode to warn, not on. In warn mode the autoscaler tells you the PG count is wrong and does nothing about it, which is a defensible default — autoscaling a production pool moves data — and it means a cluster built with defaults has an autoscaler that will only ever produce health warnings. Choose deliberately:

POOL=vm-storage

ceph osd pool autoscale-status
ceph osd pool set "$POOL" pg_autoscale_mode on
ceph osd pool get "$POOL" pg_autoscale_mode

ceph osd pool autoscale-status is the command worth knowing here: it shows, per pool, the current PG count and what the autoscaler believes it should be. A large gap on a pool in warn mode is a pending data movement you have not scheduled yet.

Step 6: Configure CRUSH and device classes

To use different device classes for different pools (e.g. NVMe for hot data, HDD for cold):

ceph osd crush rule create-replicated nvme-rule default host nvme

ceph osd pool set vm-storage crush_rule nvme-rule

Step 7: Add as Proxmox storage

The --add_storages flag in pveceph pool create does this automatically. Verify:

cat /etc/pve/storage.cfg | grep -A5 rbd

Step 8: Verify health

ceph status && ceph osd tree && ceph df && ceph pg stat

You should see:

• HEALTH_OK
• All MONs up, all MGRs up
• All OSDs up and in
• All PGs active+clean

The four checks that are not in HEALTH_OK

A fresh cluster reports HEALTH_OK before any of these has been verified, and each of them is expensive to correct later.

# 1. Does the CRUSH rule separate copies by host, not by OSD?
ceph osd crush rule dump | grep -E '"rule_name"|"type"'

# 2. Do the pools actually have size 3 and min_size 2?
ceph osd pool ls detail

# 3. Is the cluster network really separate from the public network?
ceph config get mon cluster_network
ceph config get mon public_network

# 4. Are the OSDs distributed evenly across hosts?
ceph osd tree

The third one catches a specific and common build error: pveceph init without --cluster-network puts replication traffic on the public network. Everything works, the cluster is healthy, and the first time a node fails the recovery traffic competes with client I/O on the same link. Fixing it later means editing ceph.conf and restarting every OSD.

GUI walkthrough

The Ceph panel in the GUI (Datacenter → Ceph, then per-node Ceph) shows:

• Cluster status, mon, mgr, osd, mds tabs.
• Pool creation, CRUSH rules.
• Per-OSD metrics.

Most operations can be done via GUI; some advanced operations (CRUSH rule tuning, PG manual resizing) require CLI.

Production considerations

Common mistakes

• Creating OSDs on consumer SSDs.
• Skipping the device class separation.
• Enabling PG autoscaler on a freshly-created pool with default settings (can cause unnecessary PG migrations).
• Omitting --cluster-network at pveceph init. Replication then shares the public network, and the cost only appears during recovery.
• Assuming --pg_autoscale_mode defaults to on. It defaults to warn, which reports and does nothing.
• Reaching for --disable_cephx when authentication misbehaves. It is cluster-wide, not per client, and not cleanly reversible.
• Not verifying CRUSH placement produces replicas on different hosts.

Key takeaways

• Install via pveceph install on every node; the version default is tentacle and the repository default is enterprise.
• Initialise on one node with --network, --cluster-network, --size and --min_size stated explicitly; create MONs and MGRs on the others.
• Create OSDs from raw (HBA-backed) disks.
• Use PG autoscaler; verify HEALTH_OK — and then verify the four things HEALTH_OK does not cover: the CRUSH failure domain, the pool size/min_size, network separation, and OSD distribution.

Knowledge check