Path redundancy with RBD

Why this matters in production

Teams arriving from SAN storage look for multipath configuration and do not find it. The reason is architectural rather than an omission.

Why there is no single path

flowchart TD
  C[Client] --> O1[OSD 12]
  C --> O2[OSD 31]
  C --> O3[OSD 47]
  C --> O4[OSD 8]
  C --> O5[... every OSD holding its data]

A SAN client talks to one or two storage processors, so path redundancy means having a second route to them. An RBD client computes placement with CRUSH and connects directly to every OSD holding its data — there is no controller, no single endpoint, and therefore nothing to fail over between.

# a client's connections
ss -tn | grep -c ':68[0-9][0-9]'

If an OSD becomes unreachable, the client uses the replicas. That is not failover; it is the normal read path.

What RBD does require

# multiple monitors in the storage configuration
monhost 10.0.2.11,10.0.2.12,10.0.2.13

The monitor list is the closest analogue to multipath: the client tries each until one answers. Configuring a single monitor address is the actual single point of failure that teams should look for.

grep monhost /etc/pve/storage.cfg

SAN assumptions that do not transfer

# there is nothing to configure here
multipath -ll     # returns nothing for RBD, correctly

Configuring what matters

# bond with members on different switches
cat /proc/net/bonding/bond0 | grep -E 'Mode|Hash Policy|MII Status'

# all monitors listed
pvesm set ceph-vms --monhost '10.0.2.11,10.0.2.12,10.0.2.13'

# verify the client can reach all monitors
for m in 10.0.2.11 10.0.2.12 10.0.2.13; do
  timeout 3 bash -c "cat < /dev/null > /dev/tcp/$m/3300" && echo "$m OK" || echo "$m FAIL"
done

Quiz

Production discipline

List every monitor address in the client configuration — it is RBD’s only path-like construct and a single address is a genuine single point of failure. Direct SAN path-redundancy thinking toward network bonding and switch redundancy, which is where it actually applies.

Cross-course references

• Kubernetes: service discovery replaces path management in the same way
• Linux: distributed storage removes the controller that multipath exists to route around