Block vs file vs object storage — and what Kubernetes uses

Storage comes in three categories — block, file, and object — and each has a Kubernetes use case. This lesson walks the categories, the trade-offs, and the production patterns.

Block storage

Block storage exposes a raw block device to the consumer. The consumer formats the block device with a filesystem and uses it like a local disk. Block storage is:

• Fast: low latency, high IOPS (especially on SSDs).
• Exclusive: a block device is mounted by one node at a time (RWO) or shared across nodes with a cluster filesystem (rare in Kubernetes).
• Not network-accessible: the block device is attached via a SAN protocol (iSCSI, FC) or a hypervisor API (EBS, GCE PD, Azure Disk).
• Snapshot-friendly: most block backends support crash-consistent snapshots.

flowchart LR
    A[Pod] --> B[kubelet mount]
    B --> C[CSI node plugin]
    C --> D[Block device on the node]
    D --> E[Backend: EBS, Ceph RBD, local disk]

Kubernetes workloads that use block storage:

• Databases (PostgreSQL, MySQL, MongoDB) — fast, exclusive, predictable latency.
• Message queues (Kafka, RabbitMQ) — fast, exclusive, sequential write performance.
• Stateful applications with strict latency requirements — anything that benchmarks its tail latency.

The trade-off: block storage is single-node. A Pod that needs to mount the same volume on multiple nodes cannot use block storage unless the backend supports multi-attach (EBS multi-attach, GCE PD shared disks, Ceph RBD).

File storage

File storage exposes a filesystem over the network. The consumer mounts the filesystem; multiple consumers can mount the same filesystem simultaneously. File storage is:

• Slower than block: network filesystem overhead (NFS, CephFS, GlusterFS).
• Shared: multiple nodes mount the same filesystem (RWX — ReadWriteMany).
• Network-accessible: mounted via NFS, SMB, CephFS, or similar.
• Snapshot-friendly: most file backends support snapshots.

flowchart LR
    A[Pod 1] --> B[kubelet mount]
    C[Pod 2] --> B
    B --> D[CSI node plugin]
    D --> E[NFS / CephFS / GlusterFS]

Kubernetes workloads that use file storage:

• Web content — shared HTML, images, uploads.
• Machine learning — shared datasets across training Pods.
• CMS / shared uploads — multiple Pods need to read and write the same files.
• Stateless apps that share configuration — though ConfigMaps usually suffice.

The trade-off: file storage has higher latency than block storage because of the network filesystem overhead. For latency-sensitive workloads, block storage is preferable.

Object storage

Object storage exposes a REST API for storing and retrieving objects (files, blobs). The consumer does not mount object storage; the application calls the REST API (S3, GCS, Azure Blob, MinIO).

Object storage is:

• Scalable: petabytes and beyond.
• Eventually consistent: most object stores are eventually consistent; strongly consistent stores are available but more expensive.
• REST-accessible: the application uses an HTTP client to PUT, GET, LIST, DELETE.
• Cheap per GB: object storage is the cheapest storage per byte.

Kubernetes workloads that use object storage:

• Backups: application backups go to S3, GCS, etc.
• Logs: aggregated logs are stored in object storage.
• Static content: large media files, machine learning datasets.
• Artifacts: CI/CD artifacts, container images (though the registry is typically a separate service).

Object storage is not Pod-attached. The application interacts with it via the REST API, not via a filesystem mount.

flowchart LR
    A[Pod] -->|HTTPS PUT/GET| B[S3 / GCS / MinIO]
    B --> C[Backend: distributed storage cluster]

Comparison

Choosing the right category

The decision tree:

• Latency-sensitive single-node workload: block.
• Multi-node shared workload: file.
• Backup, log, archive, large media: object.
• Hybrid: databases on block, shared content on file, backups on object.

A production cluster typically has all three: a database on EBS (block), shared uploads on NFS or CephFS (file), and backups on S3 (object).

The S3 CSI driver

The S3 CSI driver (e.g., csi-s3 by yandex-cloud, or the alternative implementations by various vendors) exposes S3-compatible object storage as a filesystem-like interface. The Pod mounts an S3 bucket; reads and writes are translated to S3 PUT and GET operations.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: s3-data
spec:
  storageClassName: s3
  accessModes: ["ReadWriteMany"]
  resources:
    requests:
      storage: 1Pi

The S3 CSI driver is not a substitute for object storage in the REST sense — it is a workaround for workloads that expect a filesystem mount. Production workloads should use the S3 REST API directly via a client library, not via the CSI driver.

Quiz

Production discipline

• Choose the storage category by the workload. Block for databases, file for shared content, object for backups and logs.
• Block storage is single-node. Workloads that need multi-node access must use file or object storage.
• Object storage is REST-accessible, not Pod-mounted. Use the AWS SDK or equivalent, not the S3 CSI driver, for production workloads.
• Document the storage choices in the cluster bootstrap. Each workload class has a storage category; the choices must be auditable.