Kubernetes workloads during a Ceph cluster problem

Why this matters in production

Different Ceph problems produce very different Kubernetes symptoms, and matching them correctly avoids treating a capacity problem as a driver problem.

The mapping

ceph -s
kubectl get pods --all-namespaces --field-selector status.phase!=Running

Running both alongside each other is what makes the correlation.

Prioritising the response

1. Is Ceph the cause? — check ceph -s before anything Kubernetes-side
2. What Ceph condition? — the mapping above
3. Fix the Ceph condition — Kubernetes recovers when it clears
4. Only then assess pods that need individual attention

Kubernetes-side actions during a Ceph problem rarely help and frequently compound it: deleting pods causes rescheduling that needs mounts that cannot happen, and scaling down loses the ability to observe.

Protecting workloads during a degraded period

# stop the scheduler making it worse
NAME=acme
CURRENT=current
kubectl scale deploy ${NAME} --replicas=${CURRENT}

Avoid during a Ceph incident:
  deleting pods hoping they recover
  draining nodes
  scaling deployments
  rolling out changes
  anything that requires a volume to be mounted

# a cluster-wide pause on new workload placement, if warranted
NODES=nodes
kubectl cordon ${NODES}

The reasoning is that every one of those actions creates a mount operation, which is the thing that cannot succeed.

Recovering afterwards

# once Ceph is healthy
ceph -s | grep HEALTH_OK

# pods that need attention
kubectl get pods --all-namespaces -o json | python3 -c '
import sys, json
for p in json.load(sys.stdin)["items"]:
    st = p.get("status", {})
    for c in st.get("containerStatuses", []):
        if not c.get("ready"):
            print(p["metadata"]["namespace"], p["metadata"]["name"],
                  c.get("state", {}))'

Quiz

Production discipline

Check ceph -s before any Kubernetes-side action when pods report storage problems — the Ceph condition determines everything and Kubernetes actions during the incident all require mounts that cannot succeed. Leave stalled pods alone; they resume with no mount operations when storage returns.

Cross-course references

• Kubernetes: backend outages should be fixed at the backend, not worked around at the workload layer
• Linux: restarting a process stuck on unavailable storage rarely helps