Rook: Ceph as a Kubernetes operator

Why this matters in production

Rook runs Ceph inside Kubernetes and manages it declaratively. The Ceph underneath is the same Ceph; what differs is how it is operated.

What Rook manages

flowchart TD
  A[CephCluster CR] --> B[Rook operator]
  B --> C[MON pods]
  B --> D[MGR pods]
  B --> E[OSD pods]
  B --> F[MDS / RGW pods]
  G[CephBlockPool CR] --> B
  B --> H[Ceph pools]

The custom resources

apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: rook-ceph
  namespace: rook-ceph
spec:
  cephVersion:
    image: quay.io/ceph/ceph:v19
  dataDirHostPath: /var/lib/rook
  mon:
    count: 3
    allowMultiplePerNode: false
  storage:
    useAllNodes: true
    useAllDevices: false
    deviceFilter: "^sd[b-z]"

apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
  name: replicapool
  namespace: rook-ceph
spec:
  failureDomain: host
  replicated:
    size: 3

The reconciliation model

Declare desired state in a CR
  → the operator observes the difference
  → it acts to converge
  → it continues observing

This differs from imperative management in a consequential way: changes made directly to Ceph that contradict a CR are reverted by the operator.

# a pool's size changed directly
ceph osd pool set replicapool size 2
# the operator reconciles it back to the CR's value
kubectl -n rook-ceph get cephblockpool replicapool -o jsonpath='{.spec.replicated.size}'

Managing Ceph settings that Rook owns must be done through the CRs.

Locating Ceph within Rook

kubectl -n rook-ceph get pods
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph -s
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd tree

The toolbox pod provides the ordinary Ceph CLI against the cluster, and everything learned about Ceph applies unchanged from there.

kubectl -n rook-ceph get cephcluster rook-ceph -o jsonpath='{.status.ceph.health}'

Quiz

Production discipline

Change anything Rook expresses in a custom resource through that resource — the operator reconciles direct Ceph changes away, often minutes later, which makes the reversion hard to attribute. Diagnose through the toolbox pod with the ordinary Ceph CLI; the cluster underneath is standard Ceph.

Cross-course references

• Kubernetes: operators reconciling away manual changes is universal controller behaviour
• Linux: configuration management systems revert out-of-band changes the same way