Aligning Kubernetes and Ceph topology

Why this matters in production

Two systems each describe where things are, and neither validates the other. Misalignment produces placement decisions that look correct and are not.

The two descriptions

Kubernetes:  node labels
  topology.kubernetes.io/zone=zone-a
  topology.kubernetes.io/region=eu-west

Ceph: the CRUSH hierarchy
  root default
    rack zone-a
      host ceph-01

kubectl get nodes -L topology.kubernetes.io/zone
ceph osd tree

Neither system knows about the other’s description, so agreement is maintained by convention and verification.

Aligning them

# 1. establish the physical truth
#    which host is in which rack, verified against the rack

# 2. label Kubernetes nodes to match
kubectl label node k8s-01 topology.kubernetes.io/zone=zone-a

# 3. place Ceph hosts to match
ceph osd crush move ceph-01 rack=zone-a

# 4. verify both against the physical layout

The physical layout is the authority; both descriptions must match it rather than each other.

Verifying

# Kubernetes view
kubectl get nodes -o custom-columns=\
'NODE:.metadata.name,ZONE:.metadata.labels.topology\.kubernetes\.io/zone'

# Ceph view
ceph osd tree | awk '/rack/ {r=$NF} /host/ {print $NF, r}'

# and against the physical reality — the part nothing automates

The third check is the one that fails: a host moved between racks updates neither description automatically.

Failures from misalignment

The last is the dangerous one: both descriptions are internally consistent and both are wrong about reality, so every check passes and the protection does not exist.

Maintaining alignment

# a check worth scheduling
#!/bin/bash
echo "== Kubernetes"
kubectl get nodes -o json | python3 -c '
import sys,json
for n in json.load(sys.stdin)["items"]:
    print(n["metadata"]["name"],
          n["metadata"]["labels"].get("topology.kubernetes.io/zone","NONE"))'
echo "== Ceph"
ceph osd tree | awk '/rack/ {r=$NF} /host/ {print $NF, r}'

Quiz

Production discipline

Verify both topology descriptions against the physical layout rather than against each other — two internally consistent descriptions can both be wrong, and every health check passes while the protection does not exist. Add the alignment check to any procedure that moves hardware.

Cross-course references

• Kubernetes: node labels describing zones are equally unverified against reality
• Linux: any redundancy scheme depends on the topology description being true