msgr2 and encryption on the wire

Why this matters in production

msgr2 is the default on any modern cluster, but having msgr2 and encrypting your traffic are different things. The protocol supports encryption; whether it is used depends on a mode setting that most clusters leave at the default. Knowing which you have is the difference between a true and a false statement in a compliance questionnaire.

What msgr2 provides

The default port arrangement carries both:

ceph mon dump
# 0: [v2:10.20.0.10:3300/0,v1:10.20.0.10:6789/0] mon.ceph-mon-01

Port 3300 is v2, 6789 is v1. A monitor advertising both accepts either, which is what allows a gradual client migration.

Integrity versus confidentiality

This is the distinction that matters:

crc mode — messages carry a CRC32c checksum. This detects accidental corruption in transit. It is not a cryptographic protection; an attacker who can modify packets can recompute the CRC.

secure mode — messages are encrypted and authenticated with AES-128-GCM. This provides confidentiality and cryptographic integrity, so an observer cannot read the data and a tamperer cannot forge it undetected.

cephx signing applies in both, and protects the authentication exchange — but signing the handshake does not encrypt the data that follows.

Checking what you have

ceph config get mon ms_cluster_mode
ceph config get mon ms_service_mode
ceph config get mon ms_client_mode
# often: crc secure   ← a preference list, most-preferred first

The values are ordered preference lists. crc secure means “prefer crc, accept secure” — which on a cluster where both ends support both results in crc, not secure. If you want encryption you must say so:

ceph config set global ms_cluster_mode secure
ceph config set global ms_service_mode secure
ceph config set global ms_client_mode secure

Migrating to msgr2

On a cluster upgraded from an older release:

ceph mon enable-msgr2
ceph mon dump          # confirm v2 addresses appear
ceph versions          # confirm all daemons are new enough

Clients using older kernel or librados versions may only support v1, which is why both are advertised during transition. Verify client support before disabling v1.

Quiz

Production discipline

Check the three mode settings explicitly before making any claim about encryption in transit, and record the verified values with the date. When moving to secure, confirm client support first and expect a modest CPU cost — then remove crc from the lists entirely so the guarantee does not depend on negotiation order.

Cross-course references

• Kubernetes: mTLS available but not enforced is the identical capability-versus-guarantee gap
• Linux: TLS cipher preference ordering has the same silent-downgrade property