Segmenting and firewalling a Ceph cluster

Why this matters in production

Firewall rules written from a partial port inventory take the cluster down in a way that looks like a Ceph fault.

The port inventory

# what is actually listening
ss -ltnp | grep -E 'ceph|radosgw|3300|6789|68[0-9][0-9]|7[0-2][0-9][0-9]'

ceph mon dump | grep -E '^[0-9]+:'
ceph orch ps --format json | python3 -c '
import sys,json
for d in json.load(sys.stdin):
    p = d.get("ports")
    if p: print("%-24s %-14s %s" % (d["daemon_name"], d.get("hostname"), p))'

Which traffic crosses which boundary

Clients → monitors (3300) and OSDs (6800–7300) on the public network
OSDs ↔ OSDs on the cluster network, replication and recovery
Monitors ↔ monitors on the public network, Paxos
Management → dashboard, Prometheus, SSH, from admin networks only
Nothing from outside → the cluster network, ever

Writing the rules

# nftables, on a Ceph host — public network side
nft add rule inet filter input ip saddr 10.20.0.0/16 tcp dport 3300 accept
nft add rule inet filter input ip saddr 10.20.0.0/16 tcp dport 6800-7300 accept

# cluster network side, cluster members only
nft add rule inet filter input ip saddr 10.30.0.0/24 tcp dport 6800-7300 accept

# management, from the admin subnet only
nft add rule inet filter input ip saddr 10.10.5.0/24 tcp dport { 22, 8443, 9283, 9100 } accept

The 6800–7300 range appears on both networks because OSDs bind on both.
Restricting it on one and not the other is the mistake that produces
"recovery works but clients cannot read".

# verify before committing to a default-deny
ceph -s
ceph osd tree | grep -c up

Verifying segmentation

# from a client host: monitors reachable, cluster network not
nc -z -w2 mon-01 3300 && echo "mon reachable (expected)"
nc -z -w2 10.30.0.11 6800 && echo "CLUSTER NETWORK REACHABLE — misconfigured"

# from a Ceph host, confirm both bindings
ss -ltn | awk '$4 ~ /:(3300|68[0-9][0-9])$/ {print $4}' | sort -u

# and that the cluster is using the networks intended
ceph config get global public_network
ceph config get global cluster_network
ceph osd metadata 0 --format json | python3 -c '
import sys,json
d=json.load(sys.stdin)
print("front:", d.get("front_addr"))
print("back: ", d.get("back_addr"))'

Quiz

Production discipline

Write firewall rules from ceph osd metadata’s bound front_addr and back_addr, not from the configured CIDRs — the CIDRs are hints and a daemon may bind elsewhere. Permit the 6800–7300 range on both networks; restricting one side breaks either clients or recovery.

Cross-course references

• Kubernetes: NetworkPolicy written from intent rather than observed endpoints breaks the same way
• Linux: firewall rules must match bound addresses, not configured intent