Skip to main content
RunBook Academy

CephXXXIII · EncryptionEncryption

Deploying secure mode across a cluster

Advanced⏱ ~17 minceph

What you'll learn

  • Configure the three mode settings correctly
  • Sequence a migration to secure mode safely
  • Measure the performance cost
  • Verify that connections actually negotiate secure

Prerequisites

  • A
  • l
  • l
  • d
  • a
  • e
  • m
  • o
  • n
  • s
  • o
  • n
  • a
  • r
  • e
  • l
  • e
  • a
  • s
  • e
  • s
  • u
  • p
  • p
  • o
  • r
  • t
  • i
  • n
  • g
  • m
  • s
  • g
  • r
  • 2
  • ,
  • a
  • n
  • d
  • a
  • v
  • e
  • r
  • i
  • f
  • i
  • e
  • d
  • i
  • n
  • v
  • e
  • n
  • t
  • o
  • r
  • y
  • o
  • f
  • c
  • l
  • i
  • e
  • n
  • t
  • v
  • e
  • r
  • s
  • i
  • o
  • n
  • s
  • .

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Not yet marked complete on this device.

Why this matters in production

Turning on encryption is a two-line change that can disconnect every client that cannot negotiate it. Doing it in the right order, with the client inventory checked first, is the difference between a quiet change and an outage.

The three settings

ceph config get global ms_cluster_mode    # daemon ↔ daemon
ceph config get global ms_service_mode    # daemon serving clients
ceph config get global ms_client_mode     # what clients request
SettingGoverns
ms_cluster_modeOSD-to-OSD, MON-to-MON — internal traffic
ms_service_modewhat daemons accept from clients
ms_client_modewhat clients offer when connecting

Each takes an ordered list. secure crc prefers encryption and falls back; secure alone requires it.

The migration order

Start with the internal traffic, which involves only daemons you control:

# 1. cluster-internal first — lowest risk
ceph config set global ms_cluster_mode secure

Restart or let daemons pick it up, and verify the cluster stays healthy.

# 2. offer secure to clients while still accepting crc
ceph config set global ms_service_mode 'secure crc'
ceph config set global ms_client_mode 'secure crc'

Now modern clients negotiate secure and older ones still connect.

# 3. only after verifying every client supports it
ceph config set global ms_service_mode secure
ceph config set global ms_client_mode secure

Step 3 is the one that disconnects clients that cannot comply. Do not take it on inventory alone — verify.

Verifying negotiation

# per-daemon connection state
ceph daemon osd.12 dump_ops_in_flight
ceph tell mon.* sessions

# what clients are connected and how
ceph daemon mon.$(hostname -s) sessions | jq -r '.[] | "\(.entity_name) \(.con_type)"'

Kernel clients are the usual constraint: an older kernel’s libceph may not support secure mode, and those clients are exactly the ones you cannot upgrade quickly.

# on a kernel client host
dmesg | grep -i 'libceph\|ceph:'

The performance cost

AES-GCM with AES-NI is fast, but the cost is real and lands on the OSD CPU:

WorkloadTypical impact
Large sequentiala few percent throughput
Small randomsmall latency increase
Recovery on wide ECmeasurable, adds to existing decode cost
CPU-constrained hoststhe constraint gets tighter

Measure on your hardware rather than assuming. On hosts already near CPU saturation during recovery, encryption is the change that pushes them over.

Quiz

Knowledge check · 4 questions

  1. Q1. Which mode setting should be changed to secure first when migrating a cluster?

  2. Q2. Changing the messenger mode immediately encrypts existing established connections.

  3. Q3. Plan a secure-mode migration with a compliance deadline.

    A compliance requirement mandates encrypted storage traffic within 60 days. The cluster serves 200 Kubernetes nodes using kernel RBD, 40 hypervisors using librbd, and an RGW deployment. Kernel versions across the Kubernetes fleet vary.

  4. Q4. Why does verifying a secure-mode migration require inspecting connections rather than configuration?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Enumerate kernel client versions before committing to a secure-mode deadline — they set the schedule and cannot be upgraded without reboots. Verify with monitor session data rather than configuration, and measure the CPU cost during a recovery rather than under normal load, where it is smallest.

Cross-course references

  • Kubernetes: enforcing mTLS in a service mesh follows the same permissive-then-strict sequence
  • Linux: disabling older TLS versions requires the same client inventory discipline