Verifying encryption is actually in effect

Why this matters in production

A cluster can be configured for secure mode and still be carrying unencrypted traffic, because negotiation falls back rather than failing.

Configured versus negotiated

ceph config get global ms_cluster_mode
ceph config get global ms_service_mode
ceph config get global ms_client_mode
ceph config get global ms_mon_cluster_mode
ceph config get global ms_mon_service_mode
ceph config get global ms_mon_client_mode

These express a preference list, commonly "secure crc". A peer that
cannot do secure negotiates crc, and the connection succeeds.

# the strict form, once every peer is known to support it
ceph config set global ms_cluster_mode secure
ceph config set global ms_service_mode secure
ceph config set global ms_client_mode secure

Setting the strict form is what turns a preference into a guarantee, and
it is also what will disconnect a client that cannot comply.

Verifying per path

# monitors: are they advertising v2 at all?
ceph mon dump | grep -E 'v2:|v1:'

0: [v2:10.20.0.11:3300/0,v1:10.20.0.11:6789/0] mon.mon-01

A v1 address in the monmap means msgr1 is still available, and any
client preferring it will use it — unencrypted, regardless of the mode
settings.

# what an OSD actually bound
ceph osd metadata 0 --format json | python3 -c '
import sys,json
d = json.load(sys.stdin)
for k in ("front_addr", "back_addr", "hb_front_addr", "hb_back_addr"):
    print("%-14s %s" % (k, d.get(k)))'

# and whether msgr1 ports are still listening
ss -ltn | awk '$4 ~ /:6789$/ {print "msgr1 listening: " $4}'

Removing the msgr1 fallback

ceph config get global ms_bind_msgr1
ceph config set global ms_bind_msgr1 false

This is the change that makes encryption verifiable: with msgr1 gone,
the only path is msgr2, and the mode setting governs it.

# but confirm no client needs msgr1 first
ceph features

# after the change, verify
ceph mon dump | grep -c 'v1:'
ceph -s

A restart of the monitors is required for the monmap to drop the v1
addresses, and cephadm handles that as a normal daemon redeploy.

Quiz

Production discipline

Set ms_bind_msgr1 false and confirm the monmap carries only v2: addresses before claiming encryption — msgr1 has no secure mode, so a v1 address means unencrypted connections remain possible whatever the ms_*_mode settings say. Check ceph features before the strict form.

Cross-course references

• Kubernetes: TLS configured with fallback is not TLS enforced
• Linux: protocol negotiation that falls back makes configuration and effect different statements