The I/O path — from application syscall to storage backend

A Pod write traverses several layers before reaching the storage backend. Each layer adds latency; production storage debugging is the exercise of identifying which layer is slow. This lesson walks the I/O path and the diagnostic signals at each layer.

The I/O path

flowchart LR
    A[Application write] --> B[Container filesystem]
    B --> C[Kernel VFS]
    C --> D[Block layer]
    D --> E[Bind-mount: Pod path -> staging path]
    E --> F[CSI driver: device path]
    F --> G[Storage backend: EBS / Ceph / NFS]

Each layer:

1. Application write: the application calls write() on a file descriptor. This is the user-space syscall.
2. Container filesystem: the container’s overlay filesystem (overlay2 by default) merges the upper directory with the lower image layers. Writes go to the upper directory.
3. Kernel VFS: the Virtual File System routes the write to the underlying filesystem (ext4, xfs).
4. Block layer: the filesystem submits I/O to the block layer, which queues it to the device driver.
5. Bind-mount: the kubelet’s bind-mount makes the staging path visible to the Pod. The actual device is on the node’s device path.
6. CSI driver: the CSI driver exposes the device path to the kubelet. For cloud-provider volumes, the CSI driver calls the cloud API to attach the volume.
7. Storage backend: the actual storage. EBS sends the write to the AWS backend; Ceph sends it to the OSD; NFS sends it to the NFS server.

Latency budget

A typical latency budget for a Pod write:

The storage backend is by far the largest contributor to I/O latency. A local SSD backend is ~150 µs; a cloud EBS backend is 1-10 ms; a network filesystem (NFS over WAN) is 1-50 ms.

Synchronous vs asynchronous I/O

Kubernetes I/O is typically synchronous from the application’s perspective: the application calls write() and waits for the kernel to acknowledge. The kernel acknowledges when the write is in the page cache, not when it reaches the backend. The actual write to the backend is asynchronous.

The distinction matters for:

• Durability: a synchronous write() does not guarantee the write reached the backend. A fsync() is required for durability.
• Latency: a synchronous write() returns when the page cache is updated; the actual backend write happens later.
• Throughput: the page cache batches writes, so asynchronous backend I/O can be higher throughput than synchronous.

Production databases (PostgreSQL, MySQL) use fsync() or O_DIRECT to bypass the page cache and write directly to the backend. This is the durability-vs-latency trade-off that storage-aware applications manage.

I/O observability

Each layer exposes observability:

# Pod cgroup directory name, from `ls /sys/fs/cgroup/system.slice/kubelet.slice`:
CGROUP=kubepods-burstable-pod9f2c1a34_5b7e_4d21_8a6f_0c3d5e7b1a29.slice

# Container filesystem: overlay2 metrics
# Per-container I/O via cgroups
cat "/sys/fs/cgroup/system.slice/kubelet.slice/$CGROUP/io.stat"

# Block layer: device I/O via iostat
iostat -x 1 /dev/xvdba

# Per-process I/O via iotop
iotop

# CSI driver: gRPC metrics
# The CSI driver exposes metrics in Prometheus format
# Volume operations, attach/detach latency, etc.

# Storage backend: cloud-provider metrics
# CloudWatch for EBS (VolumeQueueLength, BurstBalance)
# Prometheus for Ceph (OSD latency, PG state)

A storage problem manifests as high latency at one or more layers:

• High container FS latency: overlay2 issue (rare).
• High block layer latency: device is busy or saturated.
• High CSI driver latency: cloud API call is slow (EBS API throttle).
• High backend latency: the storage is congested or the network to the backend is slow.

The I/O path failure modes

• Container filesystem full: the overlay2 upper directory is full. The application cannot write. Diagnostic: df -h inside the container.
• Block device full: the underlying volume is full. Diagnostic: df -h on the staging path.
• CSI driver hang: the cloud API call is hanging. Diagnostic: CSI driver logs, cloud-provider API metrics.
• Backend unreachable: the network to the backend is broken. Diagnostic: ping the backend, check network policies.
• I/O timeout: the application has set an I/O timeout that is shorter than the backend latency. Diagnostic: application logs, I/O timeout configuration.

Quiz

Production discipline

• The storage backend dominates the I/O budget. Choose the backend for the latency requirement, not the cost.
• fsync() or O_DIRECT for durability. A write() is not durable until the backend has acknowledged.
• Observe at every layer. Container FS, block layer, CSI driver, backend. The slow layer is the diagnostic target.
• Tune the workload’s I/O pattern. Database fsync frequency, write combining, batch size. The application can be the bottleneck.
• Monitor the backend’s health. EBS BurstBalance, Ceph OSD latency, NFS server load. The backend is the most likely source of latency spikes.