Ceph authentication: cephx, keyrings and capabilities · Proxmox VE

pveceph init turns cephx on. It has been on in every Proxmox cluster you have ever built, and unless you went looking, every one of those clusters authenticates every client as client.admin — a user whose capability string is allow * on the monitors, the OSDs and the managers.

That is not a misconfiguration. It is the default, and for a hyper-converged cluster where the only clients are the Proxmox nodes themselves it is a defensible one. It stops being defensible the moment something that is not a Proxmox node needs a pool: a Kubernetes CSI driver, a second cluster, a backup appliance, a colleague who wants to mount CephFS from a workstation. At that point the question “what exactly can this key do” has an answer, and the answer is usually “everything”.

This lesson makes that question answerable.

What cephx is, and the thing it is not

cephx is mutual authentication using shared secrets. Every entity — each daemon and each client — has a name (client.admin, osd.4, mgr.pve1) and a secret key. The monitors hold a copy of every key. A client proves it knows its key, receives a time-limited ticket, and presents that ticket to the OSDs it wants to talk to.

Three settings control whether any of this is required, and all three default to cephx:

Setting	Default	What it requires
`auth_cluster_required`	`cephx`	Daemons authenticate to each other
`auth_service_required`	`cephx`	Clients authenticate to the cluster
`auth_client_required`	`cephx`	The cluster authenticates to the client

The alternative value is none, and the third row is the one people forget: cephx is mutual, so a client can also detect that it is talking to an impostor cluster.

cephx does not encrypt your data on the wire

This is the single most commonly misunderstood thing about cephx, and it matters for how you design the Ceph network.

cephx authenticates and it can sign messages, but the Ceph documentation states it plainly: “Note that even when signatures are enabled data is not encrypted in flight.” An attacker on the storage network cannot impersonate a client, and cannot forge a message, but can read every object that crosses the wire.

Encryption is a separate feature: the messenger v2 protocol’s secure mode, controlled by ms_cluster_mode, ms_service_mode and ms_client_mode. All three default to crc secure, which means “prefer CRC, accept secure” — so a default cluster negotiates the unencrypted mode. Putting secure first is what actually turns on AES-GCM encryption of post-authentication traffic.

This is why the sizing lesson insists on a dedicated, physically separated Ceph network. On a default cluster that network carries your tenants’ data in the clear.

Reading a capability string

A cephx user is a name plus a set of capabilities, one per daemon type. The grammar is small enough to learn in one sitting, and learning it is what turns allow rwx pool=vmdata from noise into a sentence.

Token	Grants
`r`	Read
`w`	Write to objects
`x`	Execute class methods, and use auth operations
`*` (or `all`)	Read, write, execute, plus administrative commands
`class-read`	Read-only class methods (a subset of `x`)
`class-write`	Write-only class methods (a subset of `x`)

Two restrictions narrow the scope: pool=<name> limits a capability to one pool, and namespace=<name> limits it further to one namespace inside a pool. Written as shell, that is pool=vmdata — a literal, not a placeholder.

On top of the letters, Ceph ships profiles: named bundles that express an intent rather than a permission set. profile rbd is the right capability for an RBD client and is far harder to get subtly wrong than assembling class-read object_prefix rbd_children, rwx by hand. profile rbd-read-only is its read-only counterpart, and profile rbd-mirror is what the mirroring daemon needs.

Read-only / Saferead every user and capability in the cluster— Read-only. Prints every entity, its key and its caps. The keys are real secrets, so treat the output as one - do not paste it into a ticket.

ceph auth ls

Read-only / Safe— Illustrative. Four lines of allow * is what every Proxmox storage entry uses unless somebody changed it.

$ ceph auth get client.admin

[client.admin]
key = REDACTED
caps mds = "allow *"
caps mgr = "allow *"
caps mon = "allow *"
caps osd = "allow *"

Illustrative output

Read that as: this key can create and destroy pools, read and write every object in all of them, remove OSDs from the CRUSH map, and edit the authentication database — including its own capabilities. There is no Ceph operation it cannot perform.

The least-privilege client, end to end

Suppose an application team wants RBD images in a pool of their own, from hosts that are not Proxmox nodes. The whole exchange is four commands.

Configuration changecreate a pool-scoped RBD client— Creates the pool if absent and a user whose reach ends at that pool. get-or-create is idempotent: running it twice returns the existing key rather than rotating it, which is what you want in a script.

POOL=app-images
CLIENT=app-team

pveceph pool create "$POOL" --size 3 --min_size 2 --application rbd

ceph auth get-or-create "client.$CLIENT" \
mon "profile rbd" \
osd "profile rbd pool=$POOL" \
mgr "profile rbd pool=$POOL"

Read-only / Safe— Illustrative. The output is keyring format, which is why it can be redirected straight into a file on the client.

$ ceph auth get-or-create client.app-team mon 'profile rbd' osd 'profile rbd pool=app-images'

[client.app-team]
key = REDACTED

Illustrative output

Then hand it over — and hand over only the key, not ceph auth ls:

Read-only / Safeexport just this client's keyring— Read-only against the cluster. Writes a keyring file containing one user. Deliver it over a channel you would use for a password, because it is one.

CLIENT=app-team
ceph auth get "client.$CLIENT" -o "/root/ceph.client.$CLIENT.keyring"

# just the secret, for tooling that wants the bare string
ceph auth print-key "client.$CLIENT"

Verification that can fail

This is the part people skip, and it is the only part that proves anything. A capability string that looks restrictive and is not is worse than no restriction, because it is documented as a control.

Read-only / Safeprove the restriction holds— Read-only. The first command must succeed and the second must fail with a permission error. If the second succeeds, the capability is wider than you think and the review that accepted it was wrong.

POOL=app-images
OTHER=vmdata
CLIENT=app-team
KEYRING="/root/ceph.client.$CLIENT.keyring"

# must succeed
rbd --id "$CLIENT" --keyring "$KEYRING" ls "$POOL"

# must fail: operation not permitted
rbd --id "$CLIENT" --keyring "$KEYRING" ls "$OTHER"

# must fail: no administrative capability
ceph --id "$CLIENT" --keyring "$KEYRING" osd pool ls

Read-only / Safe— Illustrative. This failure is the pass condition. A listing here means the osd capability was not scoped.

$ rbd --id app-team --keyring /root/ceph.client.app-team.keyring ls vmdata

rbd: error opening pool 'vmdata': (1) Operation not permitted

Illustrative output

Note the shape of the error. Ceph reports a scope violation as (1) Operation not permitted, not as “pool not found” — so a capability that is too narrow and a pool name that is misspelled produce different errors. That distinction saves ten minutes every time.

Where Proxmox keeps the keys

pveceph init writes the cluster configuration to /etc/pve/ceph.conf and, in the documentation’s words, “also creates a symbolic link at /etc/ceph/ceph.conf, which points to that file”. The consequence is worth stating: the Ceph configuration is in pmxcfs, so it is cluster-replicated, and it stops being writable when the cluster loses quorum.

The keyrings follow the same pattern. On a pveceph-managed cluster the admin keyring lives under /etc/pve/priv/, and each storage entry that talks to a Ceph cluster Proxmox does not manage gets its own keyring at /etc/pve/priv/ceph/<STORAGE_ID>.keyring. That file name is the storage ID from /etc/pve/storage.cfg, not the Ceph user name — a detail that makes external-cluster storage fail in a confusing way when somebody renames the storage.

Read-only / Safefind every Ceph credential on a node— Read-only. /etc/pve/priv is mode 0700 and only readable by root; that is the correct permission and a finding if it is anything else.

ls -l /etc/ceph/
ls -l /etc/pve/priv/ 2>/dev/null | grep -i ceph
ls -l /etc/pve/priv/ceph/ 2>/dev/null

Under the hood

Why cephx breaks when the clocks drift, and why the error blames the keyring.

The exchange has three steps. The client asks a monitor for a session key, encrypted under the client’s own secret — which proves the monitor knows that secret too. The client then asks for a ticket for the services it wants. The monitor issues one, signed with a key the OSDs share with the monitors. The client presents the ticket to any OSD, and the OSD validates it without ever contacting a monitor. That last property is what lets Ceph scale: authentication does not funnel through the monitors on every I/O.

Tickets carry an expiry. That means every participant has to agree on what time it is. Ceph’s own health check for this is MON_CLOCK_SKEW, and the default tolerance is small — a fraction of a second between monitors. When NTP fails on one node, what you observe is not “clock skew” first. You observe clients on that node failing to authenticate, with messages about the monitor session and the keyring, because an expired-looking ticket and a wrong key are indistinguishable from the client’s side.

The diagnostic order that follows from the mechanism:

1. ceph health detail          -> is MON_CLOCK_SKEW present?
2. chronyc tracking            -> on the node whose clients are failing
3. ceph auth get client.NAME   -> only after time is ruled out

Two more mechanical details fall out of the design.

The monitors are the key database. There is no separate identity store. ceph auth ls reads from the monitor’s key-value store, which is part of the Paxos-replicated cluster state — so a user you create is present cluster-wide immediately, and cannot be created at all while quorum is lost.

Keys never expire; tickets do. Rotating a cephx key is therefore an explicit act, and there is no built-in expiry to force it. ceph auth get-or-create will not rotate an existing key — that is the “get” half of its name. Rotation means deleting the user and recreating it, which is a brief outage for anything holding the old key.

What happens if you run ceph auth caps against the wrong user?

You remove capabilities, immediately and cluster-wide, and there is no undo.

ceph auth caps overwrites. It does not merge, append or extend. The command below does not add a monitor read capability to client.admin; it replaces all four capability lines with one, and the key that could do everything can now do almost nothing:

ceph auth caps client.admin mon 'allow r'

The symptom arrives within seconds and it is total: ceph -s refuses, the GUI Ceph panel errors, and every RBD storage entry that uses the admin key goes inaccessible. VMs with open images keep running on already-established connections for a while, which makes the blast radius look smaller than it is for the first few minutes.

What still works, in the order you would reach for them:

Another cephx user with mon 'allow *'. If you created a break-glass admin — and this is the reason to — you use it to restore the capabilities:

ceph --id backup-admin auth caps client.admin \
  mon 'allow *' osd 'allow *' mgr 'allow *' mds 'allow *'

The monitor’s own bootstrap keyring on disk. Each monitor holds mon. credentials locally in its data directory, independent of the auth database. A monitor’s local keyring can authenticate as mon. and repair the entry. This is a recovery procedure, not a routine one, and it wants the upstream troubleshooting documentation open beside you.

Nothing else. There is no read-only fallback and no maintenance mode. The cluster keeps serving I/O on established sessions and you cannot administer it.

The way not to be here costs one command. Before any ceph auth caps, capture what you are about to overwrite:

ceph auth get client.admin -o /root/client.admin.caps.bak

and create a second full-capability user on the day you build the cluster, stored somewhere that is not the cluster:

ceph auth get-or-create client.breakglass \
  mon 'allow *' osd 'allow *' mgr 'allow *' mds 'allow *'

Then restore is ceph auth import -i /root/client.admin.caps.bak.

Turning cephx off is not a troubleshooting step

pveceph init accepts --disable_cephx, and it exists for closed lab networks. It is the wrong response to an authentication problem, for three reasons worth being specific about.

It is not a per-client switch. Authentication is a cluster property. Turning it off removes authentication for every client, every daemon and every pool at once.

Everything on the storage network becomes trusted. Without cephx, any host that can route to the monitors can read and write every pool. There is no second control behind it — Ceph has no pool-level ACL that survives the loss of authentication.

It is not a clean toggle on a running cluster. Flipping the auth_* settings needs a restart of every daemon, and the window in between has daemons on both sides of the change failing to talk to each other. A cluster that was struggling is now down.

If the authentication is genuinely broken, the fix is in the keyring or the clock. Both are in the diagnostic order above.

Common mistakes

Handing out client.admin because it is what is already in /etc/pve/priv. It is the easiest key to copy and the worst one to give away. Five seconds of ceph auth get-or-create produces a scoped one.
Assuming cephx encrypts. It authenticates. Encryption is msgr2 secure mode, and the default negotiation prefers the unencrypted crc mode.
Using ceph auth caps to add a capability. It replaces the whole set. Read the current caps first, edit, then apply all of them.
Building the capability by hand when a profile exists. profile rbd is maintained upstream and changes when RBD needs new object prefixes. A hand-built equivalent silently stops working after an upgrade.
Scoping the osd capability to a pool and leaving mon at allow *. The data is protected and the cluster is not. Both lines need scoping.
Never testing the restriction. A capability is a claim until a denied command proves it.
Leaving a delivered keyring in /etc/pve/priv. It replicates to every node and lands in every configuration backup.

Key takeaways

cephx is on by default in Proxmox and authenticates every client; the default client is client.admin with allow * everywhere.
cephx does not encrypt. msgr2 secure mode does, and the default crc secure ordering means a stock cluster is unencrypted on the wire.
Capabilities are per daemon type. Use profile rbd and scope it with pool=; scope the mon capability too, not just osd.
ceph auth caps overwrites. Back up the existing caps before you run it, and keep a break-glass admin off the cluster.
Proxmox keeps per-storage keyrings at /etc/pve/priv/ceph/<STORAGE_ID>.keyring, named after the storage ID.
Prove a restriction with a command that must be denied. Untested least privilege is a comment, not a control.

Knowledge check

Why this matters

The Ceph clusters that get compromised are almost never compromised through Ceph. They are compromised through a key that was copied once, for a good reason, and then reused.

client.admin is the path of least resistance for that. It is already on disk on every node, it works for everything, and pasting it into a Kubernetes secret or a monitoring container takes ten seconds and no approval. What arrives with it is a credential that can destroy every pool in the cluster, and there is no audit trail distinguishing that container’s use of it from a node’s.

The counter-argument you will hear is that the storage network is private. It is worth taking seriously and it does not survive contact with the second fact in this lesson: on a default cluster that private network carries object data unencrypted, so “private” is doing all of the work. A key scoped to one pool limits what a compromise reaches. An encrypted messenger mode limits what an observer learns. Neither is difficult and neither is on by default.

The operational point is smaller and sharper than the security one: capabilities are the part of Ceph that fails quietly. A pool fills and you get a health warning. An OSD dies and you get a health warning. A capability that grants more than intended produces no signal at all — until an incident review asks what that key could reach, and the honest answer is everything.

Which is why the last command in the least-privilege procedure is one that has to fail.