Storage Bottleneck

A Grafana panel that evaluated in two hundred milliseconds last month now evaluates in twelve seconds. The query has not changed. The metric has not changed. The series count has not changed. The dashboard owner is sure they did not change anything; the platform team is sure they did not change anything. Someone mutters “the disk is slow.”

The disk is, in fact, slow. The Prometheus TSDB on this host was provisioned on a network-attached volume two months ago because the local SSD filled up. The NFS share is on a storage array that serves four hundred other workloads. Per-block read latency has crept from one millisecond to twenty milliseconds. The query still scans the same blocks; the blocks now take twenty times longer to deliver.

This lesson is about the slow-query shape that lives underneath the other four: when the storage path is the slow path. The query is correct. The metric is correct. The disk is wrong.

What a storage bottleneck is

A storage bottleneck is a query whose evaluation cost is dominated by the block read phase, not by the index, the scan, or the pipeline. The block read cost is paid on disk I/O to deliver the compressed sample blocks from the storage medium to the page cache.

  cost  =  blocks_to_read  x  per_block_latency  x  cache_miss_rate

       =  B                x  L                  x  (1 - hit_rate)

The components:

• Blocks to read. The number of two-hour TSDB blocks the query touches. A 30-day range scan touches 360 blocks.
• Per-block latency. The wall-clock time for the storage layer to deliver one block. Latency is dominated by disk seek + transfer for local disks, by network round-trip for remote disks.
• Cache miss rate. The fraction of blocks that are not in the page cache. A cold block must be read from disk; a hot block is served from RAM.

The classic shape of a storage bottleneck is a TSDB whose working set exceeds the page cache and falls back to the disk for every block read. The same query returns in milliseconds on a hot block and seconds on a cold one.

A query that takes 200 ms on a local NVMe SSD takes 4 seconds on NFS and 40 seconds on a remote object store for the same window. The cost scales linearly with the per-block latency.

Why a sysadmin cares

A storage bottleneck is the slow-query shape that is hardest to attribute. The query has not changed; the metric has not changed; the user did not change anything. The platform team did not change anything. The disk changed because the disk got busier. Or the disk changed because someone moved the TSDB to a different volume. Or the disk changed because the workload grew and the page cache no longer fits the working set.

The user-visible symptom is a panel that takes seconds to load when it took hundreds of milliseconds before. The platform-visible symptom is the same engine metrics: high prometheus_engine_query_duration_seconds. The metric does not distinguish storage cost from query cost. The distinction is in the storage metrics.

How to detect a storage bottleneck

Two layers of metrics. The first is the Prometheus engine’s own storage metrics.

# READ-ONLY. Per-block read latency from the TSDB.
promql='histogram_quantile(0.99,
  rate(prometheus_tsdb_compaction_chunk_range_seconds_bucket[5m])
)'
# Note: this exposes compaction, not query. The query-side
# signal is the engine histogram and the kernel I/O stats.
curl -s --data-urlencode "query=${promql}" http://prometheus:9090/api/v1/query

{ "data": { "result": [{ "metric": {}, "value": [1735000000.000, "8.4" }] } }

Eight-second compaction chunk range. The compaction is spending most of its budget waiting on disk. The query path will see the same wait.

The second layer is the kernel’s I/O statistics.

# READ-ONLY. Per-device read latency from the kernel.
# The Prometheus TSDB volume is /dev/nvme0n1 in this example.
iostat -dx /dev/nvme0n1 5 2

Device   r/s    w/s    rkB/s    wkB/s   await  %util
nvme0n1  142    38     18432    1024    18.4   42.1

Eighteen milliseconds average wait, forty-two percent utilisation. The disk is responding, but slowly. Compare to the same metric before the symptom appeared: the read wait was three milliseconds and utilisation was twelve percent.

The third layer is the page cache hit rate.

# READ-ONLY. Page cache hit rate for the Prometheus data
# directory. A hit rate below 90 percent for a hot metric is
# a problem.
pcstat /var/lib/prometheus/data

|  cache hit rate  | total pages | cached pages |
|               68 |     8200000 |      5576000 |

Sixty-eight percent cache hit rate. The working set exceeds RAM. Every cold block must be read from disk.

For Loki and Tempo, the storage path is different but the shape is the same. Loki reads chunks from the chunk store (S3-compatible object store by default; Cassandra or filesystem for self-hosted). Tempo reads blocks from the block store. The metrics are different; the diagnostic is the same.

# READ-ONLY. Loki query throughput and latency.
curl -s http://loki:3100/metrics | grep -E 'loki_request_duration_seconds_count'

How to fix it

Five options, in increasing cost of change.

Option 1. Move the TSDB to local NVMe. The cheapest fix in cost-per-block-latency. Local NVMe SSD delivers sub-millisecond read latency for the working set. The TSDB stays on the host; the storage layer is the host’s disk.

# /etc/default/prometheus -- relevant fragment.
# The TSDB lives at the path given by --storage.tsdb.path.
# The Prometheus default is data/ relative to the working
# directory; distribution packages override it in the unit
# file or the defaults file. Place this directory on a local
# NVMe volume, not on NFS or shared block storage.
ARGS="--storage.tsdb.path=/var/lib/prometheus/data"

The compaction window is set by --storage.tsdb.min-block-duration and --storage.tsdb.max-block-duration, both hidden flags that Prometheus documents “for use in testing”. Leave them at their defaults of 2 hours and 10% of the retention period; the fix for a slow TSDB is the device, not the block layout.

Option 2. Move to object storage for cold blocks. When the deployment is sized for long retention and the local disk is not large enough, push the cold blocks to object storage and keep the head block on local NVMe. Thanos and Mimir are the reference architectures.

# /etc/prometheus/prometheus.yml -- relevant fragment.
# Thanos sidecar pushes the compacted blocks to an S3 bucket.
# Queries read the head block locally and the cold blocks from
# the bucket.
sidecar:
  type: s3
  config:
    bucket: thanos-cold-prod-eu-1
    endpoint: s3.eu-west-1.amazonaws.com
    region: eu-west-1
    access_key: ${THANOS_S3_ACCESS_KEY}
    secret_key: ${THANOS_S3_SECRET_KEY}

The query engine reads the head block from local NVMe and the cold blocks from object storage. The head block is small (three hours of samples) and fits in RAM. The cold blocks are served from S3 with twenty to one hundred milliseconds of per-block latency, which is acceptable for the cold-block workload (the long-range scan that runs occasionally).

Option 3. Resize the disk. When the working set has grown because the platform has scaled, a larger local disk is the right answer. A larger disk does not help when the working set exceeds RAM; a larger RAM is the answer in that case.

# READ-ONLY. Working set versus RAM.
promql='prometheus_tsdb_head_series * 3072'
# Rough approximation: 3 KB per series.
# Compare to node_memory_MemAvailable_bytes.
curl -s --data-urlencode "query=${promql}" http://prometheus:9090/api/v1/query

{ "data": { "result": [{ "metric": {}, "value": [1735000000.000, "30688231424" }] } }

Twenty-eight gigabytes. The host has sixteen. The head block cannot fit in RAM. The right answer is more RAM, not more disk.

Option 4. Move to a remote storage backend. For very large deployments, push the TSDB to a remote storage backend that is designed for the workload. Cortex, Mimir, and Thanos are the canonical choices. Each provides horizontal scale-out for both ingestion and query.

# mimir-distributed: /etc/mimir/runtimeconfig.yaml
# Mimir uses object storage for both hot and cold blocks.
# The query path is distributed across queriers.
blocks_storage:
  backend: s3
  s3:
    bucket_name: mimir-blocks-prod-eu-1
    endpoint: s3.eu-west-1.amazonaws.com
    region: eu-west-1

Option 5. Use the right storage class. Object stores have storage classes with different latency and cost trade-offs. Hot storage (frequent access) costs more per GB and delivers lower latency. Cold storage (infrequent access) costs less per GB and delivers higher latency. The hot tier is the right choice for the TSDB blocks that are queried often; the cold tier is the right choice for the long-tail archive.

How to validate it

Three steps.

Step 1. Confirm the per-block latency.

# READ-ONLY. Read latency from the kernel for the TSDB
# volume. Expect < 1 ms on local NVMe, < 5 ms on a
# well-provisioned Ceph RBD, < 10 ms on NFS.
iostat -dx /var/lib/prometheus 5 2

Device   r/s   w/s   rkB/s   wkB/s   await  %util
nvme0n1  86    12    9216    512     0.6    4.1

Point-six milliseconds average wait, four percent utilisation. The disk is no longer the bottleneck.

Step 2. Confirm the page cache hit rate.

# READ-ONLY. Page cache hit rate for the TSDB directory.
pcstat /var/lib/prometheus/data

|  cache hit rate  | total pages | cached pages |
|               94 |     8200000 |      7708000 |

Ninety-four percent cache hit rate. The working set fits in RAM. The cold reads are the long tail.

Step 3. Confirm the query cost is reduced.

# READ-ONLY. Engine evaluation cost for the same query that
# was the offender before the change.
promql='topk(5, sum by (query) (rate(prometheus_engine_query_duration_seconds_sum[5m])))'
curl -s --data-urlencode "query=${promql}" http://prometheus:9090/api/v1/query \
  | jq '.data.result[] | {query: .metric.query, rate: .value[1]}'

{
  "query": "sum by (status) (rate(http_requests_total[5m]))",
  "rate": "0.18"
}

Eighteen percent of the previous rate. The fix has landed.

How it can fail

Five failure shapes.

1. TSDB placed on NFS during a capacity crisis. The local SSD filled; someone moved the TSDB to NFS to recover space. The latency went up. The queries went slow. The capacity problem was solved; the latency problem was created.
2. Working set exceeds RAM after a scale-out. A platform sized for ten million series grew to twenty million. The head block memory grew from thirty to sixty gigabytes. The host has thirty-two gigabytes of RAM. The head block thrashes the page cache.
3. Compaction pressure from a high-cardinality metric. A label explosion produces millions of new series per hour. The compaction runs constantly. The disk is busy compacting. The query waits.
4. Cross-region S3 for hot blocks. A bucket in a different region was chosen for cost. The per-block latency is two hundred milliseconds. The query takes thirty seconds for a 30-day range. The cost saving is paid back as lost engineer time.
5. Disk shared with another workload. A host runs Prometheus and a log shipper on the same disk. The log shipper saturates the disk during peak hours. Prometheus queries time out during peak hours.

How to troubleshoot it

1. Find the offender. topk(5, sum by (query) (rate(prometheus_engine_query_duration_seconds_sum[5m]))). Note the queries.
2. Check the engine histogram. If the engine cost is high and the per-block latency is high, the storage is the suspect. If the engine cost is high and the per-block latency is normal, the storage is innocent; the query is the problem.
3. Check the kernel I/O stats. iostat -dx on the Prometheus volume. Read wait above 5 ms on local SSD, above 15 ms on local HDD, above 10 ms on NFS is a bottleneck.
4. Check the page cache hit rate. Below 90 percent for the hot metric is a working-set problem. Resize RAM or resize the metric.
5. Apply the cheapest fix. Move the disk to local NVMe, then resize RAM, then move to object storage for cold blocks.

Security implications

A storage backend exposes the data over its own protocol. Prometheus’s local TSDB is a directory; access is governed by the host’s file permissions. Object storage is a remote endpoint; access is governed by credentials and bucket policies.

• Restrict the credentials. A leaked Thanos sidecar credential can read or delete every block in the bucket.
• Use bucket policies. A bucket that holds Prometheus blocks should not also hold application data; the blast radius of a leak is the whole observability stack.
• Encrypt in transit. Object storage over TLS. Self-hosted Ceph or NFS over a private network.
• Audit writes. Object storage access logs are the right place to detect a leaked credential.

Performance implications

• CPU. Block decompression is the dominant CPU cost of a storage-bound query. The decompression is single-threaded per query. The CPU is rarely the bottleneck; the I/O is.
• Memory. The head block index must fit in RAM. A working set that exceeds RAM falls back to disk on every read.
• Disk. The disk must deliver the blocks at the latency the query budget allows. Local NVMe is the right default for the head block; object storage is the right default for the cold blocks.
• Network. Object storage traffic is the right amount of bandwidth for the workload. A bucket in a different region doubles the bandwidth cost.

Production guidance

• Place the TSDB on local NVMe. The head block is hot; the cold blocks are archival. Different storage layers.
• Size the RAM to fit the head block index. The rule of thumb is three kilobytes per active series; a ten-million- series deployment needs roughly thirty gigabytes.
• Use object storage for the cold blocks. S3-compatible storage is the default; Ceph RGW and MinIO are common alternatives for on-prem.
• Monitor node_disk_read_time_seconds_total and node_disk_writes_completed_total against the disk capacity. A read wait above five milliseconds on local SSD is a bottleneck.
• Back up the storage configuration. A TSDB whose bucket credentials are lost is unrecoverable. The configuration is the durable state; the blocks can be re-ingested from the exporters.

Verification

You should now be able to answer:

• What three numbers determine the cost of a storage-bound query?
• Why is NFS the wrong choice for the TSDB?
• What is the right storage layer for the head block versus the cold blocks?
• How do you distinguish a storage bottleneck from a query bottleneck using the engine metrics?

Quiz