Pre-check: network configuration

Why this matters in production

A node whose network differs subtly from the cluster’s produces failures that appear later and elsewhere.

What must match

Testing both networks

# addresses present
ip -br addr show | grep -E '10\.0\.[23]\.'

# MTU on the interfaces
ip link show | grep -oE 'mtu [0-9]+'

# reachability and MTU to every existing node, on both networks
for n in 10.0.2.11 10.0.2.12 10.0.2.13; do
  printf 'public  %-14s ' "$n"
  ping -M do -s 8972 -c 2 -W 2 "$n" >/dev/null 2>&1 && echo OK || echo FAIL
done
for n in 10.0.3.11 10.0.3.12 10.0.3.13; do
  printf 'cluster %-14s ' "$n"
  ping -M do -s 8972 -c 2 -W 2 "$n" >/dev/null 2>&1 && echo OK || echo FAIL
done

# throughput, both directions
iperf3 -c 10.0.3.11 -t 20 -P 8
iperf3 -c 10.0.3.11 -t 20 -P 8 -R

A node reaching only half the throughput of its peers becomes the bottleneck for every PG it holds.

Bond configuration

cat /proc/net/bonding/bond0 | grep -E 'Bonding Mode|Hash Policy|MII Status'

Compare against an existing node:
  same mode
  same hash policy
  same member count
  both members up

A bond with a different hash policy distributes traffic differently, which produces uneven utilisation that is hard to attribute later.

Time and name resolution

chronyc tracking
chronyc sources

# forward and reverse must both work
getent hosts ceph-07
dig +short -x 10.0.2.27

# and from the cluster's perspective
ceph orch host ls

Reverse resolution matters because some Ceph components and much tooling use it, and a missing PTR record produces confusing output rather than a clear error.

Catching asymmetries

# a comparison script, run on the new node and an existing one
{
  ip -br addr show | grep -E '10\.0\.[23]\.'
  ip link show | grep -oE 'mtu [0-9]+' | sort -u
  cat /proc/net/bonding/bond0 2>/dev/null | grep -E 'Mode|Hash Policy'
  ethtool bond0 2>/dev/null | grep Speed
  chronyc tracking | grep -E 'Reference|Stratum'
} > /tmp/netcheck-$(hostname -s).txt

Diffing the output between the new node and an existing one surfaces every difference at once.

Quiz

Production discipline

Test MTU and reachability to every existing node on both networks separately — one misconfigured port affects only its paths, and a node correct publicly and wrong on the cluster network backfills nothing while appearing healthy. Diff a configuration summary against a known-good node to surface every difference at once.

Cross-course references

• Kubernetes: node configuration drift is caught the same way, by comparison
• Linux: verifying a new member matches the fleet is standard before adding it