Host networking configuration for Ceph

Intermediate⏱ ~17 minipnmcli

What you'll learn

Configure predictable interface naming
Set up bonds and VLANs correctly for Ceph
Assign addresses matching the Ceph network CIDRs
Verify the configuration before deployment

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Ceph daemons select their addresses by matching local interfaces against the configured CIDRs. Every networking mistake therefore surfaces as a daemon that will not start or binds to the wrong network, which is a confusing symptom for a straightforward cause.

Predictable interface names

ip -br link show

lo         UNKNOWN  00:00:00:00:00:00
enp1s0f0   UP       ac:1f:6b:2c:4d:8e
enp1s0f1   UP       ac:1f:6b:2c:4d:8f
bond0      UP       ac:1f:6b:2c:4d:8e
bond0.20   UP       ac:1f:6b:2c:4d:8e
bond0.30   UP       ac:1f:6b:2c:4d:8e

Predictable names — enp1s0f0 rather than eth0 — matter because eth0 can change across reboots depending on probe order. A bond built on names that moved comes up with the wrong members or not at all.

The typical stack

physical NICs → bond0 (LACP) → VLAN sub-interfaces → addresses

# bond
nmcli con add type bond ifname bond0 bond.options \
  "mode=802.3ad,miimon=100,lacp_rate=fast,xmit_hash_policy=layer3+4"
nmcli con add type ethernet ifname enp1s0f0 master bond0
nmcli con add type ethernet ifname enp1s0f1 master bond0

# VLANs
nmcli con add type vlan ifname bond0.20 dev bond0 id 20 ip4 10.20.0.21/24 gw4 10.20.0.1
nmcli con add type vlan ifname bond0.30 dev bond0 id 30 ip4 10.30.0.21/24

VLAN 20 is the public network, VLAN 30 the cluster network. Only the public network needs a gateway.

MTU across the stack

nmcli con mod bond0 802-3-ethernet.mtu 9000
nmcli con mod bond0.20 802-3-ethernet.mtu 9000
nmcli con mod bond0.30 802-3-ethernet.mtu 9000

Set it on the bond and on each VLAN sub-interface — sub-interfaces do not reliably inherit it, which is the most common jumbo-frame mistake.

Matching the Ceph CIDRs

ip -br addr show
ceph config get mon public_network
ceph config get mon cluster_network

Each host must have exactly one address matching each configured CIDR. Zero matches means the daemon cannot bind and fails to start; several matches means the selection is not deterministic.

Verifying

# bond health
cat /proc/net/bonding/bond0 | grep -E 'MII Status|Aggregator|Hash|LACP rate'

# addresses against the CIDRs
ip -br addr show | grep -E '10\.20\.|10\.30\.'

# MTU end to end on both networks
ping -M do -s 8972 -c 3 10.20.0.22
ping -M do -s 8972 -c 3 10.30.0.22

# throughput baseline
iperf3 -c 10.30.0.22 -P 8 -t 30

Run all four on every host before deployment and record the results.

Quiz

Knowledge check · 4 questions

Q1. A host has two addresses matching the configured public_network CIDR. What is the risk?

The daemon will not start, because binding requires exactly one address inside the configured CIDR Traffic is load balanced across both addresses, halving the throughput available to each daemon The cluster network is used instead, because an ambiguous public match makes the daemon fall back Address selection depends on enumeration order and can differ between daemons or across reboots
Q2. Setting MTU 9000 on a bond reliably propagates to VLAN sub-interfaces created on it.

True False
Q3. Prepare host networking for a new storage host.

A new host has two 25 Gb NICs and must join a cluster using VLAN 20 for the public network and VLAN 30 for the cluster network, with jumbo frames enabled throughout.
Q4. Why do predictable interface names matter for a bonded Ceph host?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Verify that exactly one address matches each Ceph CIDR before adding a host; zero produces a daemon that will not start and several produce non-deterministic binding. Set MTU explicitly on VLAN sub-interfaces — the parent bond’s setting does not reliably propagate and the resulting failure is partial and hard to attribute.

Cross-course references

Kubernetes: node network configuration errors surface as pods that cannot be scheduled or reached
Linux: bonding and VLAN configuration is standard practice with Ceph-specific verification