Set up a 3-node Ceph cluster

The goal is a working hyperconverged Ceph cluster providing RBD for VM disks and CephFS for shared file storage.

Steps

1. Install Ceph

On each node, run through the GUI: Datacenter → Ceph → Install. Choose Reef (the version Proxmox 9.x ships). Repeat on all three nodes.

2. Initialise the monitors

From one node:

pveceph init --cluster-network 10.0.2.0/24

This creates the first MON. Join the other two:

# On pve-01:
pveceph mon add pve-02 10.0.2.12
pveceph mon add pve-03 10.0.2.13

3. Create OSDs

For each unused disk on each node:

pveceph osd create /dev/sdb

You should have at least 3 OSDs total (one per node) for any meaningful test. 9 OSDs (3 per node) gives you room to fail one disk per node.

4. Create the RBD pool

pveceph pool create vm-storage --pg-autoscale-mode on

Add it to Proxmox as a storage backend: Datacenter → Storage → Add → RBD.

5. Create the CephFS

pveceph fs create --pg-pool-cephfs-data cephfs-data \
                 --pg-pool-cephfs-metadata cephfs-meta

6. Mount CephFS from a client

# Install ceph-common on a Linux client, copy /etc/pve/ceph/ceph.conf
# and the keyring from the cluster.
mkdir /mnt/cephfs
mount -t ceph pve-01:/ /mnt/cephfs \
  -o name=admin,secret=$(ceph-authtool -p ceph.client.admin.keyring)
echo "hello from ceph" > /mnt/cephfs/test.txt
cat /mnt/cephfs/test.txt

Verification

• ceph -s shows health: HEALTH_OK
• The RBD pool appears in Datacenter → Storage and is selectable when creating a VM disk
• ls /mnt/cephfs/test.txt succeeds from the client

Failure-mode test

Stop one OSD host (pvecm stop pve-02 or pull the power). The cluster should report HEALTH_WARN with osds down, but reads and writes continue. Bring the host back; ceph -s should return to HEALTH_OK.

Cleanup

# On a clean cluster, leave these to the operator — Ceph teardown is
# intentionally manual to avoid accidental data loss.