Troubleshoot Kubernetes / Ceph CSI

This runbook covers Troubleshoot Kubernetes / Ceph CSI on a Ceph Tentacle cluster.

When to use

• Active production scenario matching the impact (service-affecting)
• The prerequisites are met

Prerequisites

Before starting this runbook, verify each prerequisite has been met. If a prerequisite is not met, resolve it first or escalate.

• A Kubernetes Pod cannot mount a PVC
• The CSI driver reports errors

Pre-checks

Collect evidence before taking action. Evidence is the proof the runbook worked.

# Check the PVC: kubectl describe pvc
# Check the PV
# Check the StorageClass
# Check the CSI driver logs: kubectl logs
# Check Ceph: ceph -s
# Check the cephx credentials in the csi-rbd secret
# Check the network between Kubernetes and Ceph

Procedure

Step 1

Identify the failing layer: CSI driver, cephx, network, Ceph cluster

Step 2

If CSI driver: restart it; verify version compatibility

Step 3

If cephx: rotate the cephx key; update the Kubernetes secret

Step 4

If network: verify the storage network

Step 5

If Ceph cluster: see ceph-rb-investigate-mon-quorum and friends

Step 6

Verify: the Pod mounts the PVC and reads / writes correctly

Verification

After completing the procedure, verify the result:

# The PVC mounts
# The application is running
# The cluster is HEALTH_OK

Rollback

If the procedure does not produce the expected verification, roll back:

• If the PVC is permanently Pending: recreate the PVC; check the StorageClass
• If the CSI driver is broken: upgrade or downgrade

Escalation

If you cannot roll back or the rollback fails, escalate:

• Storage team
• Kubernetes team