3-node Ceph cluster lab

This is the canonical Ceph install lab. It produces a working 3-node Ceph cluster with default CRUSH rules and a usable VM pool.

Environment

Steps

1. Install Ceph on each node

# On all three nodes
apt install -y ceph ceph-common

2. Initialise Ceph from pve-01

In the GUI: pve-01 → Ceph → Install. Choose:

• Version: Ceph Reef (default)
• Cluster network: 10.0.1.0/24
• Public network: 10.0.0.0/24

Or via CLI:

pveceph install --version reef
pveceph init --cluster-network 10.0.1.0/24 --public-network 10.0.0.0/24

3. Create monitors

# On each node (including pve-01)
pveceph mon create

Verify:

ceph -s
# Expected: 3 mons, health OK

4. Create managers (mgrs)

# On two nodes (for HA)
pveceph mgr create
pveceph mgr create

5. Add OSDs

On each node, identify the disk to use (e.g. /dev/sdb):

lsblk
pveceph osd create /dev/sdb

Repeat on all three nodes. Verify:

ceph osd tree
# Expected: 3 hosts, 3 OSDs, root default
ceph -s
# Expected: osd: 3 osds: 3 up, 3 in

6. Create the VM pool

In the GUI: Ceph → Pools → Create.

Name: ceph-vm
Size: 3 (replicas)
PG autocomplete: yes (default 128)

Or via CLI:

pveceph pool create ceph-vm --size 3 --pg_autoscale_mode on

7. Add the pool as PVE storage

pvesm add rbd ceph-vm --pool ceph-vm --content images,rootdir

8. Create a test VM on Ceph

qm create 902 --name ceph-test --memory 2048 --cores 2 \
  --scsi0 ceph-vm:32,iothread=1 --net0 virtio,bridge=vmbr0 \
  --ostype l26
qm start 902

Verify the VM is functional and the OSDs report active I/O:

ceph osd perf
# Expected: read/write throughput distributed across the 3 OSDs

Verification

• ceph -s reports health: HEALTH_OK, 3 mons, 3 mgrs, 3 OSDs
• A VM created on ceph-vm storage boots and runs
• Killing one OSD does not stop the VM (replicas 3/2 absorb it)

Cleanup

# Substitute the OSD id reported by `ceph osd tree` on the node:
OSD_ID=0

qm stop 902 && qm destroy 902
pveceph pool destroy ceph-vm
# On each node:
pveceph osd destroy "$OSD_ID"
# Then via GUI or CLI: uninstall Ceph from all nodes