Purpose
Ceph is a distributed system pretending to be a disk, and the pretence holds only as long as the network does. Recovery traffic saturating the same link as client traffic, an MTU mismatch that works for small packets and hangs on large ones, DNS that resolves forward but not reverse - each of these presents as storage being slow, which sends the wrong team to the incident. This checklist is run before the first OSD is deployed.
Order of work
The list runs in three passes. The fabric pass is public-cluster,
bond-mode and switch-redundancy: the physical and VLAN layout, settled
first because changing it later means an outage. The path pass is mtu and
firewall, which prove that a packet of the size Ceph will actually send can
cross every hop it needs to reach. The identity pass is dns and
time-sync - the two services Ceph depends on that are not Ceph, and the two
that most often make a healthy cluster look broken.
Before you start
Have the VLAN and subnet plan, the switch port map for each host, the MTU configured on every hop including the switch fabric, and the site NTP sources.
Verification
The network is ready when every critical item is checked and the MTU test has been run host to host with the do-not-fragment bit set, not assumed from the interface configuration. Mark an item N/A rather than leaving it unchecked when it does not apply to this cluster.
Escalation
If public and cluster traffic cannot be separated, record the accepted risk with the network owner before deployment rather than discovering it during recovery.