Connecting Proxmox to an external Ceph cluster

Every Ceph lesson so far has assumed pveceph — Proxmox installs the daemons, Proxmox owns /etc/pve/ceph.conf, and the GUI shows you OSD health because the OSDs are on the nodes you are looking at.

In a large organisation that is often not the arrangement. A storage team runs Ceph for several consumers, of which your virtualisation platform is one. They will give you three things: a list of monitor addresses, a pool name, and a keyring. Everything else — CRUSH rules, min_size, scrub schedules, when the cluster is upgraded — belongs to them.

Proxmox supports this properly. What changes is not the storage configuration, which is four lines, but what you can see and who you call at 3 AM.

What “external” means to Proxmox

There is no external: yes flag. The RBD and CephFS storage plugins each have two options that are documented as “Optional, only needed if Ceph is not running on the Proxmox VE cluster” — monhost and username — and setting them is what makes the storage external.

The consequences follow from that:

The row that surprises people is the GUI. Proxmox will show the storage as available and report its capacity, because that comes from the RBD client. It will not show you cluster health, because it is not the cluster’s administrator. A HEALTH_ERR on the far side is invisible from the Proxmox GUI until it becomes an I/O problem.

Attaching an RBD pool

Ask the storage team for a user scoped to your pool rather than client.admin. The previous lesson has the command they run; what you receive should look like a keyring with one section in it.

STORAGE=ceph-external
POOL=pve-vms
MONS="192.0.2.20 192.0.2.21 192.0.2.22"
USER=pve-client

pvesm add rbd "$STORAGE" \
--monhost "$MONS" \
--pool "$POOL" \
--username "$USER" \
--content images,rootdir \
--keyring /root/ceph.client.pve-client.keyring

The resulting entry in /etc/pve/storage.cfg matches the documented example:

rbd: ceph-external
        monhost 192.0.2.20 192.0.2.21 192.0.2.22
        pool pve-vms
        content images,rootdir
        username pve-client

Note what is not in that file: the key. pvesm add ... --keyring reads the file you point at and writes it to /etc/pve/priv/ceph/<STORAGE_ID>.keyring — named after the storage ID, not the Ceph user. storage.cfg is world-readable and replicated; /etc/pve/priv is mode 0700. That split is deliberate and it is the reason you should not hand-edit storage.cfg and copy the keyring yourself under a name that seemed logical at the time.

If a ceph.conf from the far cluster is needed — non-default ports, an explicit public_network, a cluster FSID your client cannot infer — place it as /etc/pve/priv/ceph/<STORAGE_ID>.conf. It is optional and most deployments do not need it.

Attaching CephFS

CephFS is the right backend for the content types RBD cannot hold: backups, ISOs, container templates, snippets.

STORAGE=cephfs-external
MONS="192.0.2.20 192.0.2.21 192.0.2.22"
USER=pve-client
FSNAME=cephfs

ceph auth print-key "client.$USER" > /root/cephfs.secret

pvesm add cephfs "$STORAGE" \
--monhost "$MONS" \
--username "$USER" \
--fs-name "$FSNAME" \
--content backup,iso,vztmpl \
--keyring /root/cephfs.secret

Two options are worth knowing before you need them:

subdir mounts a subdirectory of the filesystem rather than its root, defaulting to /. When several consumers share one CephFS, this is how each gets its own tree without a separate filesystem.

fuse switches from the kernel client to ceph-fuse, defaulting to 0. The kernel client is faster and is the right default; FUSE is the answer when the node’s kernel client is older than the features the filesystem requires, which happens when the storage team upgrades ahead of you.

krbd, and the one case where it is not a choice

RBD images can be attached two ways. librbd is a userspace library that QEMU links against directly. krbd is the kernel RBD module, which maps the image to a /dev/rbd* block device that anything on the host can open.

Proxmox exposes this as a single krbd flag on the storage. The default is off, and for VM disks that is correct: QEMU with librbd avoids a kernel round trip and picks up new RBD features with the userspace libraries rather than with the kernel.

grep -A6 '^rbd:' /etc/pve/storage.cfg
rbd showmapped
lsmod | grep -w rbd

Verification that can fail

pvesm status showing active proves the monitors answered and the key was accepted. It does not prove you can write, and it does not prove the pool is the one you meant.

STORAGE=ceph-external
POOL=pve-vms
USER=pve-client
KEYRING="/etc/pve/priv/ceph/$STORAGE.keyring"

pvesm status --storage "$STORAGE"

rbd --id "$USER" --keyring "$KEYRING" \
  -m 192.0.2.20 ls "$POOL"

# capacity as the client sees it, not as the storage team reports it
rbd --id "$USER" --keyring "$KEYRING" \
  -m 192.0.2.20 du "$POOL" 2>/dev/null | tail -1

Then allocate a small disk through Proxmox and delete it again. Creating a 1 GiB scratch disk on the storage exercises the whole path — monitor lookup, OSD write, storage.cfg parsing, the keyring file name — in a way that a listing does not.

The version-skew question

The Proxmox nodes run whatever librbd and kernel RBD their PVE version ships. The far cluster runs whatever the storage team has deployed. These do not have to match, and Ceph’s client compatibility is deliberately wide, but the direction of the skew matters:

Older client, newer cluster is the normal and supported direction. A Squid-era client talks to a Tentacle cluster; it simply does not use features it does not know about.

Newer client, older cluster is usually fine and occasionally not, because a feature the client enables on an image may not be readable by older code — which becomes visible only if the far cluster later needs to read that image with its own tooling.

The case that genuinely bites is the kernel client. krbd implements a subset of RBD image features, and it refuses to map an image whose feature bits it does not understand. If the storage team creates images for you with all features enabled, krbd mapping fails with a feature mismatch and librbd does not — so the same pool works for VMs and not for containers. Ask for rbd_default_features on their side, or create your images through Proxmox, which uses defaults Proxmox can consume.

Common mistakes

• One monitor in monhost. It works until the reboot after that monitor is retired, and then nothing starts.
• Hand-editing storage.cfg and placing the keyring under the Ceph user’s name. Proxmox looks for the storage ID. Renaming a storage without renaming its keyring breaks it the same way.
• Feeding a full keyring to the CephFS backend. It wants the bare key. The error says permission denied.
• Enabling krbd on a shared storage to make containers work, and changing how every VM attaches its disk as a side effect. Use two storage entries.
• Accepting client.admin from the storage team. It is what they will offer if you do not ask. It also means their cluster can be destroyed from your hypervisor, which is a risk for both of you.
• No read-only health credential. Without one, every incident starts with a phone call.
• Assuming the GUI would have told you. For an external cluster it reports capacity, not health.

Key takeaways

• Setting monhost and username is what makes a Ceph storage external; there is no separate flag.
• Credentials live in /etc/pve/priv/ceph/ named after the storage ID: .keyring for RBD (with its [client.x] section), .secret for CephFS (the bare key).
• monhost is a bootstrap list. Stale entries are invisible until a cold start, so list every monitor.
• krbd is required for container root disks and is a per-storage setting; use two storage entries when VMs and containers share a pool.
• Proxmox shows capacity for an external cluster, not health. Negotiate a mon 'allow r' user so you can run ceph health detail yourself.
• Verify by allocating and deleting a disk, not by reading pvesm status.

Knowledge check