Skip to main content
RunBook Academy

← All break/fix scenarios in Ceph

intermediatekubernetes-csi~22 min

Kubernetes PVC fails to bind - Ceph backend error

Reported symptoms

  • ●kubectl describe pvc shows Pending with provisioning failed
  • ●The csi-rbd / csi-cephfs pod logs show errors
  • ●Other PVCs in the cluster are fine

Evidence

  • · kubectl logs csi-rbd-provisioner-xxx shows the error
  • · ceph health is HEALTH_OK
  • · The StorageClass parameters are correct
Diagnosis and resolutionclick to reveal

Root cause

The PVC StorageClass or its parameters are incompatible with the backend (wrong pool, wrong fsType, missing features).

Remediation

Inspect the StorageClass; correct the parameters; verify a new PVC binds; investigate why the original PVC failed.

Verification

The new PVC is bound; the application schedules a Pod; the volume mounts; I/O flows.

Prevention

Validate StorageClass parameters at design time; rehearse PVC creation; monitor PVC bind time.

Architecture

The student examines the broken architecture diagram and identifies the responsible component. The cluster is running Ceph Tentacle (20.2.x); the incident is one a production operator must diagnose from the evidence presented.

Symptoms

  • kubectl describe pvc shows Pending with provisioning failed
  • The csi-rbd / csi-cephfs pod logs show errors
  • Other PVCs in the cluster are fine

Evidence

  • kubectl logs csi-rbd-provisioner-xxx shows the error
  • ceph health is HEALTH_OK
  • The StorageClass parameters are correct

Student investigation

The student follows the methodology: define the symptom, determine the impact, gather evidence, identify the component, form a hypothesis, test safely, restore, validate.

Progressive hints

  • Hint 1: check the cluster state with ceph -s first.
  • Hint 2: read ceph health detail and identify the affected PGs / OSDs / daemons.
  • Hint 3: use ceph osd tree, ceph pg dump, or ceph mds stat as the next-level diagnostic.
  • Hint 4: the root cause is documented in the frontmatter root_cause field.

Validation

The student runs the verification steps and confirms the symptom cleared.

Root cause

See the frontmatter root_cause field.

Remediation

Apply the fix from the frontmatter remediation field.

Prevention

Apply the prevention measures from the frontmatter prevention field.