Storage IOPS and tail latency — the storage performance

Averages hide the failure users actually feel: a request that fans out to ten backends has close to a one-in-ten chance of touching at least one p99 response, so a healthy p50 beside an ugly p99 is the normal shape of a slow service rather than an anomaly. Storage produces that shape more often than anything else, because a volume with a burst credit balance serves beautifully until the credits are spent and then falls off a cliff with no change in the workload. This lesson covers reading the latency distribution instead of its mean, telling an IOPS ceiling apart from a slow query, and what each of those is fixed with.

The tail latency

The tail latency is the 99th-percentile (p99) latency. The tail latency is the latency at the slowest 1% of requests. The tail latency is the application’s user experience for the unlucky users.

flowchart LR
    A[Latencies] --> B{p50}
    A --> C{p95}
    A --> D{p99}
    A --> E{p99.9}

The tail latency is the application’s worst case.

The diagnostic

The canonical diagnostic:

# Substitute your own values before running:
NS=production
POD=orders-api-6d4f8c9b7d-h4t2v

# 1. Check the application's metrics
kubectl port-forward -n "$NS" "$POD" 8080:8080
# Browse to http://localhost:8080/metrics

# 2. Check the disk I/O
kubectl exec -it "$POD" -n "$NS" -- iostat -x 1 5

# 3. Check the database's metrics
# (database-specific)

# 4. Check the storage backend's metrics
# (backend-specific)

# 5. Check the application's logs
kubectl logs -n "$NS" "$POD" --tail=200

The diagnostic is the application’s metrics, the disk I/O, the database’s metrics, the backend’s metrics, and the logs.

Common failures

• Tail latency high. The p99 latency is high. The remediation is to investigate the slowest requests.
• IOPS limit reached. The IOPS limit is reached. The remediation is to upgrade the backend.
• Database query slow. The database query is slow. The remediation is to optimise the query.
• No caching. The application is not caching. The remediation is to add caching.

flowchart TD
    A[Tail latency high] --> B{IOPS limit?}
    B -->|Yes| C[Upgrade the backend]
    B -->|No| D{Database query slow?}
    D -->|Yes| E[Optimise the query]
    D---|No| F{No caching?}
    F -->|Yes| G[Add caching]
    F -->|No| H[Unknown]

The percentile

The percentile is the application’s latency distribution. The p50 is the median; the p99 is the slowest 1%.

# Substitute your own values before running:
SVC=orders-api
NS=production

# Check the percentiles
curl -w "time_total: %{time_total}\n" -o /dev/null -s "http://$SVC.$NS.svc.cluster.local"

The percentile is the application’s latency distribution.

The remediation

The remediation depends on the cause:

# Option 1: Upgrade the backend
# Move to a higher-IOPS backend

# Option 2: Optimise the database query
# Add an index, optimise the query

# Option 3: Add caching
# Add a read cache (e.g., Redis)

# Option 4: Add retries with exponential backoff
# (client-specific)

The remediation is the storage performance recovery.

Production discipline

A storage performance issue is the cluster’s hypothesis. The discipline is to walk the 11-step methodology applied to the storage layer, identify the cause, apply the remediation. The storage is the cluster’s data; the remediation is the storage performance recovery.

• Check the percentile. The percentile is the application’s latency distribution.
• Check the IOPS. The IOPS is the storage’s bottleneck.
• Check the database. The database is the application’s source of truth.

Quiz