Performance IOPS and throttling — the storage bottleneck

A throttled volume does not report an error. The device shows ordinary throughput and a high await because it is queueing rather than saturated, and the application simply gets slower until its own timeouts start firing. The cap can sit in three different places — the backend’s provisioned IOPS, the Pod’s cgroup io.max, or the application’s own I/O pattern — and this lesson covers telling them apart from inside the Pod.

The storage performance

The storage performance is governed by:

• IOPS. The Input/Output Operations Per Second. The backend’s IOPS limit.
• Throughput. The data transfer rate. The backend’s throughput limit.
• Latency. The time to complete an I/O operation. The backend’s latency.
• Throttling. The backend’s mechanism to limit the IOPS when the limit is exceeded.

flowchart LR
    A[Pod] --> B[cgroup]
    B --> C[CSI driver]
    C --> D[Storage backend]
    D --> E{IOPS limit?}
    E -->|Yes| F[Throttling]
    E -->|No| G[Performance]

The performance is the storage’s bottleneck.

The diagnostic

The canonical diagnostic:

# Substitute your own values before running:
POD=postgres-0
NS=production
PVC=postgres-data-postgres-0

# 1. Check the disk I/O
kubectl exec -it "$POD" -n "$NS" -- iostat -x 1 5

# 2. Check the backend's metrics
# (backend-specific)

# 3. Check the cgroup's limits
kubectl exec -it "$POD" -n "$NS" -- cat /sys/fs/cgroup/io.max

# 4. Check the application's logs
kubectl logs -n "$NS" "$POD" --tail=200

# 5. Check the PVC
kubectl get pvc -n "$NS"
kubectl describe pvc "$PVC" -n "$NS"

The diagnostic is the disk I/O, the backend’s metrics, the cgroup’s limits, and the PVC.

Common failures

• Backend IOPS limit reached. The backend’s IOPS limit is reached. The remediation is to upgrade the backend.
• Throughput limit reached. The backend’s throughput limit is reached. The remediation is to upgrade the backend.
• cgroup IOPS limit reached. The cgroup’s IOPS limit is reached. The remediation is to increase the cgroup’s limit.
• Application is I/O intensive. The application is causing high I/O. The remediation is to optimise the application.

flowchart TD
    A[Storage slow] --> B{Backend limit?}
    B -->|Yes| C[Upgrade the backend]
    B -->|No| D{cgroup limit?}
    D -->|Yes| E[Increase the cgroup limit]
    D---|No| F{Application I/O?}
    F -->|Yes| G[Optimise the application]
    F -->|No| H[Unknown]

The remediation

The remediation depends on the cause:

# Option 1: Upgrade the backend
# Move to a higher-IOPS backend

# Option 2: Increase the cgroup's limit
# Edit /sys/fs/cgroup/io.max

# Option 3: Optimise the application
# (application-specific)

# Option 4: Add caching
# Add a read cache (e.g., Redis) to reduce the backend load

The remediation is the storage performance recovery.

Production discipline

A storage performance incident is the cluster’s hypothesis. The discipline is to walk the 11-step methodology applied to the storage layer, identify the cause, apply the remediation. The storage is the cluster’s data; the remediation is the storage performance recovery.

• Check the disk I/O. The disk I/O is the storage’s bottleneck.
• Check the backend’s metrics. The backend’s metrics are the storage’s source.
• Check the cgroup’s limits. The cgroup’s limits are the workload’s budget.

Quiz