Storage fundamentals — what storage is in Kubernetes and what it is not

Storage in Kubernetes is a layered model with several moving parts: the kubelet’s volume subsystem, the CSI driver, the storage backend, and the user’s request for storage (the PVC). This lesson walks the layers and establishes what Kubernetes abstracts and what it does not.

The layered model

flowchart TB
    A[User: PVC manifest] --> B[API server]
    B --> C[StorageClass]
    C --> D[CSI controller]
    D --> E[Storage backend: NFS, EBS, Ceph, etc.]
    E --> F[CSI node plugin on node-1]
    F --> G[kubelet on node-1]
    G --> H[Mount into Pod]
    H --> I[Application reads/writes]

Each layer has a responsibility:

• User: writes a PVC manifest that requests storage.
• API server: persists the PVC in etcd.
• StorageClass: a template for provisioning; defines the CSI driver, the parameters, the reclaim policy.
• CSI controller: a Pod that runs the CSI driver’s control-plane operations (create, delete, snapshot).
• Storage backend: the actual storage (NFS server, cloud volume, Ceph cluster, local disk).
• CSI node plugin: a Pod on each node that performs node-side operations (attach, mount, format).
• kubelet: coordinates with the CSI node plugin to attach and mount volumes.
• Application: reads and writes files.

What Kubernetes abstracts

Kubernetes abstracts the request for storage from the provisioning of storage. The user writes a PVC; the CSI driver provisions the storage. The user does not need to know whether the storage is EBS, NFS, Ceph, or local disk.

What Kubernetes does not abstract:

• Performance: a PVC does not include IOPS, latency, or throughput. The user picks a StorageClass; the StorageClass maps to a backend with performance characteristics.
• Durability: a PVC’s reclaim policy and the backend’s replication determine durability. The user must read the StorageClass and the backend’s documentation.
• Snapshots: a snapshot is a backend-specific concept. Kubernetes exposes snapshots as a CRD; the CSI driver implements the snapshot operation.
• Backup: snapshots are not backups. The user must implement application-consistent backup separately.

The kubelet’s role

The kubelet is the per-node agent that:

• Watches for Pods scheduled to its node.
• Calls the CSI node plugin to attach volumes.
• Calls the CSI node plugin to format and mount volumes.
• Mounts the volumes into the Pod’s containers.
• Reports volume status to the API server.

The kubelet’s volume subsystem is the bridge between the cluster-level abstraction (PVC) and the node-level reality (a mounted filesystem in a Pod). When the kubelet fails, the Pod’s volumes fail.

# Pod UID, from `kubectl get pod ... -o jsonpath={.metadata.uid}`:
POD_UID=6b3f1c0e-9a24-4d7e-bb51-0f2c8d47a913

# Kubelet's volume state
ls -la "/var/lib/kubelet/pods/$POD_UID/volumes/"
# kubernetes.io~csi/
#   pvc-2f8ad51c-77b0-4e39-9c14-6ae0b3d95f82/
#     mount (the mounted filesystem)

The kubelet writes volume metadata under /var/lib/kubelet/pods/<pod-uid>/volumes/. This is the filesystem view of the kubelet’s volume subsystem; it is useful for debugging volume attach and mount failures.

The CSI driver’s role

The CSI (Container Storage Interface) driver is a containerized storage driver that runs in the cluster. It has two components:

• Controller plugin: a Deployment (or StatefulSet) that runs the control-plane operations (create, delete, snapshot, expand). It runs anywhere in the cluster.
• Node plugin: a DaemonSet that runs the node-side operations (attach, mount, format). It runs on every node.

The controller plugin and the node plugin are separate Pods because the control-plane operations are cluster-wide and the node operations are per-node. A failure in one does not affect the other.

# The app= label your CSI driver's Pods carry:
CSI_DRIVER_APP=ebs-csi-node

# CSI driver Pods
kubectl -n kube-system get pods -l app="$CSI_DRIVER_APP"

# CSI driver CRDs
kubectl get crd | grep csi

# StorageClasses backed by the CSI driver
kubectl get storageclass

The storage backend’s role

The storage backend is the actual storage. The CSI driver is the interface; the backend is the implementation. Common backends:

• Cloud-provider volumes: AWS EBS, GCP Persistent Disk, Azure Disk. Block storage; one volume per node.
• Network filesystems: NFS, CephFS, GlusterFS. Filesystem storage; many volumes per node.
• Distributed block storage: Ceph RBD, Longhorn, OpenEBS, Rook. Block storage replicated across nodes.
• Local storage: the node’s local disk. Fast, but not replicated.

The backend determines the storage’s properties: IOPS, latency, throughput, replication, snapshots, encryption.

What the user sees

The user writes a PVC:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: ssd
  resources:
    requests:
      storage: 100Gi

The PVC requests 100 GB of storage from the ssd StorageClass. The user does not know (and does not need to know) whether ssd is backed by EBS, Ceph, or local disk. The StorageClass encodes the choice.

Quiz

Production discipline

• PVCs are requests, not guarantees. The backend determines the performance, durability, and replication; the user must read the StorageClass and the backend’s documentation.
• Snapshots are not backups. A snapshot is a crash-consistent point-in-time; an application-consistent backup requires the application to quiesce.
• The kubelet is the bridge. Volume attach and mount failures are kubelet issues, not CSI issues. The diagnostic starts at the kubelet’s volume state.
• CSI driver components are separate. Controller plugin failures affect provisioning; node plugin failures affect attach and mount. The diagnostic distinguishes them.