Deploying and configuring Ceph-CSI

Why this matters in production

Most Ceph-CSI problems are deployment problems — a capability missing, a cluster ID mismatched, a secret in the wrong namespace — and they present as opaque errors.

Preparing the Ceph side

ceph osd pool create k8s-rbd 128 128 replicated
ceph osd pool application enable k8s-rbd rbd
rbd pool init k8s-rbd

ceph auth get-or-create client.k8s-rbd \
  mon 'profile rbd' \
  osd 'profile rbd pool=k8s-rbd' \
  mgr 'profile rbd pool=k8s-rbd'

ceph fsid

The mgr capability is required for image operations that go through the manager, and omitting it produces failures that appear unrelated.

Deploying

# the fsid printed by the previous step
FSID=$(ceph fsid)

helm repo add ceph-csi https://ceph.github.io/csi-charts
helm install ceph-csi-rbd ceph-csi/ceph-csi-rbd \
  --namespace ceph-csi --create-namespace \
  --set "csiConfig[0].clusterID=$FSID" \
  --set 'csiConfig[0].monitors={10.0.2.11:3300,10.0.2.12:3300,10.0.2.13:3300}'

kubectl -n ceph-csi get pods -w
kubectl -n ceph-csi get csidriver

The secret

apiVersion: v1
kind: Secret
metadata:
  name: csi-rbd-secret
  namespace: ceph-csi
stringData:
  userID: k8s-rbd
  userKey: AQD...

ceph auth get-key client.k8s-rbd
kubectl -n ceph-csi create secret generic csi-rbd-secret \
  --from-literal=userID=k8s-rbd \
  --from-literal=userKey="$(ceph auth get-key client.k8s-rbd)"

The secret must exist in the namespace the StorageClass references, and the StorageClass references it explicitly for provisioning, node staging, and node publishing — three separate references that must all be correct.

Verifying end to end

# 1. the driver is registered
kubectl get csidriver rbd.csi.ceph.com

# 2. a test PVC provisions
kubectl apply -f - <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: csi-test
spec:
  accessModes: [ReadWriteOnce]
  resources: { requests: { storage: 1Gi } }
  storageClassName: ceph-rbd
EOF

kubectl get pvc csi-test -w

# 3. the image exists in Ceph
rbd -p k8s-rbd ls

# 4. a pod mounts it
kubectl run csi-test-pod --image=busybox --restart=Never \
  --overrides='{"spec":{"containers":[{"name":"c","image":"busybox","command":["sleep","3600"],"volumeMounts":[{"name":"v","mountPath":"/data"}]}],"volumes":[{"name":"v","persistentVolumeClaim":{"claimName":"csi-test"}}]}}'

kubectl exec csi-test-pod -- sh -c 'echo ok > /data/test && cat /data/test'

Four steps, each confirming a different component.

Diagnosing deployment problems

kubectl -n ceph-csi logs deploy/csi-rbdplugin-provisioner -c csi-rbdplugin --tail=50

Quiz

Production discipline

Include mgr alongside mon and osd in the Ceph-CSI user’s capabilities — provisioning succeeds without it and snapshot and trash operations fail later with unrelated-looking errors. Verify the deployment with a test PVC and a pod that writes a file before any application depends on it.

Cross-course references

• Kubernetes: CSI driver deployment verification is a standard pre-production step
• Linux: service account permissions are the most common integration failure