The client protocol and msgr2 — how clients talk to the cluster

Intermediate⏱ ~15 mincephss

What you'll learn

Describe what msgr2 provides over the legacy protocol
Read Ceph address notation and identify ports in use
Choose between crc and secure connection modes
Diagnose connectivity faults from client-side symptoms

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Everything a client does goes over msgr2, and a surprising share of “Ceph is broken” reports are connectivity faults with Ceph-shaped symptoms. Knowing which ports matter and how addresses are written makes those quick to resolve.

What msgr2 provides

msgr2 replaced the original protocol and is the default in current releases. It adds:

Authentication before data, so cephx negotiation happens on a connection that has not yet carried payload.
Optional encryption in transit through the secure mode.
Integrity checking with CRC32c in crc mode.
A distinct port, 3300, alongside legacy 6789.

Reading Ceph addresses

Addresses appear in a compound notation:

[v2:10.0.1.10:3300/0,v1:10.0.1.10:6789/0]

That is one monitor reachable on msgr2 at port 3300 and on the legacy protocol at 6789. OSDs use a range of ports, typically 6800-7300, with each OSD binding several — separate ports for public traffic, cluster traffic, and heartbeats.

ceph mon dump
ceph osd find 12
ss -tlnp | grep ceph

The firewall rule that catches everyone

Opening 3300 and 6789 to clients is obvious. Forgetting the OSD port range is the common mistake, and its symptom is characteristic: ceph status works, ceph -s looks healthy, and every attempt to read or write an object hangs.

That combination — control plane fine, data plane hanging — almost always means monitor ports are open and OSD ports are not.

firewall-cmd --permanent --add-port=3300/tcp
firewall-cmd --permanent --add-port=6789/tcp
firewall-cmd --permanent --add-port=6800-7300/tcp
firewall-cmd --reload

Clients need the OSD range too, because clients talk to OSDs directly. There is no proxy.

crc versus secure

ceph config set global ms_cluster_mode secure
ceph config set global ms_service_mode secure
ceph config set global ms_client_mode secure

Mode	Provides	Cost
`crc`	integrity against corruption	negligible
`secure`	integrity plus AES-128-GCM encryption	measurable CPU, some throughput

crc is the default and is right for a trusted network. secure is appropriate when traffic crosses untrusted segments, when compliance requires encryption in transit, or for multi-tenant environments. Measure before and after on a representative workload — the cost depends heavily on CPU and on whether AES-NI is available.

Why MTU mismatches present as Ceph problems

Ceph moves large messages: object writes, replication traffic, recovery. On a jumbo-frame network with MTU 9000 configured inconsistently — say the hosts are set to 9000 and one switch port is still 1500 — small packets pass and large ones are dropped.

The result is that monitors talk to each other fine (Paxos messages are small), clients connect fine, and then large writes hang. It looks like a storage problem and it is a network configuration problem.

The test is a do-not-fragment probe at the real payload size:

ping -M do -s 8972 10.0.1.10

8972 is 9000 minus 28 bytes of IP and ICMP headers. If that fails while a default-size ping succeeds, the MTU is inconsistent somewhere on the path. This test belongs in the build checklist for every cluster using jumbo frames, run between every pair of hosts.

Diagnosing from symptoms

Symptom	Likely cause
client hangs on connect	monitor ports blocked or monitors unreachable
`ceph status` fine, I/O hangs	OSD port range blocked
large writes hang, small ones work	MTU mismatch
intermittent, correlates with load	network congestion or a degrading link
auth errors on connect	cephx keyring or clock skew

Each row points at a different test, which is the value of the table: it converts a vague report into one command.

Quiz

Knowledge check · 4 questions

Q1. A new client can run ceph status successfully but every attempt to read or write an object hangs. What is the most likely cause?

The cephx keyring lacks the capabilities needed for the pool the client is addressing The client holds a stale cluster map and has no monitor from which to refresh it The OSD port range is blocked by a firewall while the monitor ports are open The pool has placement groups below min_size, so every request to them blocks
Q2. An MTU mismatch on a jumbo-frame network can leave monitors healthy and client connections working while large writes hang.

True False
Q3. A compliance requirement mandates encryption in transit for all storage traffic. Plan the change and its verification.

96-OSD cluster serving RBD for 400 VMs and RGW for backups. Currently ms modes are at the default crc. Hosts are recent Xeon with AES-NI. Cluster network is 25 GbE, currently peaking around 8 Gb/s during recovery. The compliance deadline is in six weeks and there is a staging cluster available.
Q4. List the ports a Ceph client needs reachable and explain why the OSD range is required.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Put the full port set in the build checklist — 3300, 6789, and the OSD range — because clients talk to OSDs directly and the partial rule produces a healthy-looking cluster that cannot serve I/O. On any jumbo-frame network, run a do-not-fragment probe at full payload size between every pair of hosts before declaring the build complete. Choose secure mode deliberately with measurements from a representative workload rather than from a datasheet, and enable it in stages so a regression can be attributed.

Cross-course references

Ceph: Part XXVIII (Ceph Networking) for the full network model.
Ceph: Part XXXI (Ceph Authentication) for cephx on top of msgr2.
Ceph: Part XXXIII (Encryption) for in-transit and at-rest options together.