Local Disk for TSDB

A team moves Prometheus from a SATA SSD to a gp3 EBS volume on the same instance type. They expected similar performance. Within an hour the head block reports storage_tsdb_out_of_order samples; alerts evaluate as stale; the on-call engineer gets paged. The investigation reveals that the previous host had local NVMe with roughly 200 K IOPS, and the new EBS volume is delivering roughly 3 K IOPS at the configured size. The compactor fell hours behind within the first shift.

Local disk for Prometheus is not interchangeable with any volume you can attach. The right choice depends on the ingest profile and the WAL behaviour.

What local disk for TSDB is

Local disk for the Prometheus TSDB is the storage that underwrites every dashboard, alert, and recording rule in the stack. It is the disk that holds the head block (the most recent two hours of samples), the compacted on-disk blocks (everything older), the write-ahead log that absorbs ingestion spikes, and the mmap’d files the querier reads on every query.

   +---------------------------------------------+
   |            Prometheus process               |
   |                                             |
   |  +----------+         +-----------------+   |
   |  |  head    |  write  |  mmap'd blocks  |   |
   |  |  block   |-------->|  (immutable)    |   |
   |  |  (mmap)  |         |  /var/lib/      |   |
   |  +----------+         |  prometheus/    |   |
   |        |              |  data/01/...    |   |
   |        v              +-----------------+   |
   |  +----------+                  |            |
   |  |   WAL    |                  | read       |
   |  |  (fsync) |                  v            |
   |  +----------+           querier / rule       |
   |        |                evaluator            |
   +--------|-----------------------------------+
            v
   /var/lib/prometheus (local disk)

The two I/O shapes that matter:

• Write path — head block appends and WAL fsync. Random 4 KB writes with fsync latency in the millisecond range. This is IOPS-bound.
• Read path — querier iterates the mmap’d blocks. Mostly sequential scans through chunk files. This is throughput-bound.

A disk that is fast on one path is not necessarily fast on the other. The right choice depends on the workload.

Why a sysadmin cares

The local disk choice is made once and rarely revisited. Three failure shapes appear when the choice was made without measurement.

1. The slow head block. A team puts Prometheus on EBS gp3 with no provisioned IOPS. At 50 K samples per second the head block stalls; the WAL fsync takes longer than the scrape interval; samples are rejected as out-of-order. Symptom: alerts that depend on rate() over a 5-minute window show flat lines; the prometheus_tsdb_head_series metric stays constant while the wall clock advances.
2. The compaction backlog. A team runs Prometheus on a SATA SSD that delivers 30 K IOPS sustained. The head block streams in samples; the compactor falls behind because the compaction writes are random 4 KB writes at high concurrency. Symptom: the prometheus_tsdb_compactions_failed_total counter rises; the head block grows past its 2-hour window; the mmap’d block set on disk becomes too large to fit in the host page cache.
3. The volume that fills. A team runs Prometheus with 30-day retention on a 200 GB gp3 volume. The ingest rate doubles after a new exporter ships; the volume fills on day 18. Symptom: prometheus_tsdb_storage_blocks_bytes hits the volume cap; the head block returns storage_tsdb_out_of_bounds_time; the dashboards go blank.

How it works

The write path

Every scrape produces a batch of samples. The samples are appended to the in-memory head block. The head block is memory-mapped to a file on local disk. The write is durable once the WAL is fsync’d.

The WAL is a sequence of segment files. Each segment is typically 128 MB. The WAL is replayed on startup to recover the head block state after a crash. The WAL is also the file that absorbs ingestion spikes: if the head block is compacting and cannot accept new samples, the WAL queues them until the compaction completes.

The write path is IOPS-bound because:

• The WAL fsync is a synchronous write that must complete before the next batch can be accepted.
• The head block appends are 4 KB page faults that hit the page cache first and the disk on flush.

The compaction path

Every two hours the head block is compacted into an immutable on-disk block. The compaction reads the existing blocks for the same time window, merges them with the head block, and writes a new block that replaces them.

The compaction path is IOPS-bound because:

• It reads many small chunk files from disk.
• It writes many small chunk files to disk.
• The writes are not sequential; they are random within the output block.

The compaction cost scales with the number of series and the number of blocks in the compaction window. A Prometheus with 10 M series and a 25-hour default compaction window writes roughly 80 MB per compaction per million series.

The read path

Every query iterates the mmap’d blocks. The query engine reads chunks; each chunk is a compressed time-series. The read path is throughput-bound because:

• The block files are large (hundreds of MB).
• The chunks within a block are read sequentially.
• The compression (Gorilla) is decompressed in memory.

The read path benefits from large sequential reads. A disk that delivers 1 GB/s sequential read outperforms a disk that delivers 200 K IOPS random read for the same query volume.

The interaction

The three paths contend for the same disk. A compaction running while a query is iterating the same blocks steals IOPS from the query. A WAL fsync storm (caused by a remote receiver outage) steals throughput from the compactor. The right hardware has spare capacity on both axes.

Under the hood

The Prometheus TSDB is implemented as a custom on-disk format optimised for the access pattern of time-series data. The relevant kernel and userspace layers:

• mmap — the head block and the on-disk blocks are memory-mapped. Reads do not copy; the page cache is the read cache.
• fsync — the WAL fsync is the durability boundary. A WAL segment that has been fsync’d is durable across a crash.
• page cache — Linux’s page cache is the hot read cache. A block file that fits in RAM is read at memory speed.
• io_uring (kernel 5.6+) — modern kernels can submit reads through io_uring, which is more efficient than the legacy read() syscall for the high-concurrency query pattern. Prometheus benefits when the kernel and the storage device both support it.

The local disk the TSDB lives on is a single point of failure for the Prometheus instance. A disk that fails loses every block that has not yet been compacted and every sample in the WAL that has not yet been fsync’d.

How to configure it

Prometheus local TSDB configuration is small, and almost none of it is in prometheus.yml. The path and the retention are command-line flags read once at startup. The storage: section of the configuration file carries only tsdb.out_of_order_time_window and exemplars.max_exemplars; Prometheus parses the file strictly, so an invented key under storage: stops the process before it opens the TSDB.

# /etc/default/prometheus  -- flags read by the packaged unit
ARGS="--storage.tsdb.path=/var/lib/prometheus \
      --storage.tsdb.retention.time=30d \
      --storage.tsdb.retention.size=200GB"

The configuration file carries the reloadable half:

# /etc/prometheus/prometheus.yml  -- reloadable configuration
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    cluster: eu-west-1-prod

The flags, annotated:

• --storage.tsdb.path — the directory the TSDB uses. Defaults to data/ relative to the working directory. Mount the filesystem with noatime and nodiratime; the TSDB reads the same files many times and the access time is never useful.
• --storage.tsdb.retention.time — the maximum age of a block. Older blocks are deleted. Defaults to 15 days when no retention flag is set.
• --storage.tsdb.retention.size — the maximum total size of all blocks. Oldest blocks are deleted first when the cap is reached. Set this to roughly 80% of the disk capacity to leave room for the WAL and the head block. A unit is required: B, KB, MB, GB, TB, PB or EB.

Two more flags turn up in tuning advice and are worth naming so they are recognised rather than copied. --storage.tsdb.min-block-duration and --storage.tsdb.max-block-duration set the compaction window; both are hidden flags that Prometheus documents “for use in testing”, the minimum defaulting to 2 hours and the maximum to 10% of the retention period. Leave them alone on a production host. --storage.tsdb.wal-compression compresses the WAL and has defaulted to on since Prometheus 2.20, so it does not need to be set either.

The WAL is colocated with the blocks by default. Some operators separate them onto different devices so the WAL fsync does not contend with compaction I/O:

# /etc/fstab  -- WAL on a separate NVMe device
UUID=...  /var/lib/prometheus/wal   ext4  noatime,nodiratime,defaults  0 2
UUID=...  /var/lib/prometheus/data  ext4  noatime,nodiratime,defaults  0 2

This is a useful pattern on hosts with multiple NVMe devices and is not necessary on hosts with a single high-end NVMe.

How to validate it

# READ-ONLY: TSDB retention is configured as expected.
curl -fsS http://prometheus:9090/api/v1/status/runtimeinfo | jq .data
# {"data":{"GOGC":"100","GOMAXPROCS":"4","storageRetention":"30d,200GB"}}

# READ-ONLY: TSDB head is healthy.
curl -fsS http://prometheus:9090/api/v1/status/tsdb | jq '.data.headStats'
# {"numSeries": 1234567, "chunkCount": 89, ...}

# READ-ONLY: head block is not stalled on the WAL.
curl -fsS http://prometheus:9090/api/v1/status/tsdb | \
  jq '.data.headStats.chunkCount'

# READ-ONLY: WAL segments are not piling up.
find /var/lib/prometheus/wal -name "*.wal" | wc -l

# READ-ONLY: storage is within capacity.
node_filesystem_avail_bytes{mountpoint="/var/lib/prometheus"}

# READ-ONLY: disk IOPS are within budget.
rate(node_disk_io_now{mountpoint="/var/lib/prometheus"}[5m])

A clean validation: headStats.chunkCount is in the low double digits (one per scrape interval in the head window); the disk has more than 20% free; the WAL segment count is under 10.

How it can fail

The most expensive local-disk failures, in order of how often they appear in incident reviews.

1. Volume fills; head block returns storage_tsdb_out_of_bounds_time. The disk has no free space; the TSDB refuses to write the head block. Symptom: dashboards show gaps for the duration of the outage; alerts evaluate over staleness; the WAL grows without bound until the disk fills further.
2. WAL fsync storm on a slow volume. A remote receiver outage causes Prometheus to batch WAL writes; the WAL fsync latency exceeds the scrape interval. Symptom: prometheus_tsdb_head_series is flat for several scrape intervals; the WAL segment count climbs past 10.
3. Compaction backlog on a low-IOPS volume. A gp3 volume without provisioned IOPS delivers roughly 3 K IOPS; the compactor cannot drain the head block in 2 hours. Symptom: prometheus_tsdb_compactions_failed_total rises; the head block grows past 2 hours; queries take longer because they must scan more chunks.
4. Read latency spikes under contention. A backup job runs at the same time as a query burst. The page cache is evicted by the backup reads; the queries hit the disk. Symptom: query latency rises from 200 ms to 5 s; the node_disk_io_now metric shows high read IOPS.
5. Disk failure; blocks lost. The local NVMe fails; the blocks that were on the failed disk are gone. The WAL segments that were not fsync’d are also gone. Symptom: the TSDB refuses to start because the head block is corrupt; the operator must restore from a backup.
6. Filesystem full of write-ahead logs. The WAL is on the same volume as the blocks and the operator forgot to exclude /var/lib/prometheus/wal from a quota. Symptom: the WAL segment count is zero but the head block is not flushing; the volume is at 100% but the blocks are only 80%.

How to troubleshoot it

The diagnostic order is “is the disk healthy?”, “is the head block draining?”, “is the compaction keeping up?”.

1. Start at the disk. node_disk_io_now and node_filesystem_avail_bytes. If the disk is at 100% utilisation and 0% free, the volume is the problem.
2. Check the head block. prometheus_tsdb_head_series and prometheus_tsdb_head_chunks. A head block that is not draining is the compactor or the disk.
3. Check the WAL. The WAL segment count should be under 10. A count that climbs without bound is a sign the head block is not flushing.
4. Check the compactor. prometheus_tsdb_compactions_total and prometheus_tsdb_compactions_failed_total. A failure counter that rises is a sign of a write fault or a permission fault.
5. Check the queries. The prometheus_engine_query_duration_seconds histogram. A p99 that spikes under load is a sign of read contention.

Security implications

• Local disk is a single point of failure. A disk that fails loses every block that has not been compacted and every sample in the WAL that has not been fsync’d. Backups of the TSDB path are not optional.
• Filesystem permissions are the access boundary. The TSDB path should be owned by the Prometheus user and not readable by other users. The on-disk blocks contain metric names, label values, and the timestamp series; depending on the workload, that data may include user IDs or business KPIs.
• Disk encryption protects data at rest. LUKS on Linux with a key managed by the cloud KMS is the baseline. An unencrypted local disk that is replaced or decommissioned without being wiped is a data-leak vector.
• Multi-tenant isolation requires separate backends. Two teams sharing a Prometheus share all blocks. Use a separate Prometheus per tenant, or run Mimir / Cortex with tenant IDs in the label.

Performance implications

• NVMe outperforms SATA SSD by roughly 10x on the relevant axes. A 2026 NVMe device delivers 1 M IOPS and 7 GB/s throughput. A SATA SSD delivers 100 K IOPS and 600 MB/s throughput. The difference is visible in the head-block and compaction paths.
• EBS gp3 without provisioned IOPS delivers 3 K IOPS. A Prometheus at 30 K samples per second needs roughly 30 K IOPS for the WAL alone. gp3 requires provisioned IOPS for any workload above 30 K samples per second.
• EBS io2 Block Express delivers 1 K IOPS per GB provisioned, up to 256 K. io2 is the right choice for high-ingest Prometheus instances. The cost per GB-month is roughly 10x gp3.
• Instance store is fast and ephemeral. AWS instance store and equivalent offerings on other clouds deliver high IOPS but lose data on instance stop. Useful for short-retention Prometheus where the long retention lives on Mimir via remote_write.

Production guidance

• Local NVMe for the hot tier. A 2026 production Prometheus instance with 30-day retention should run on local NVMe or io2 Block Express with at least 50 K IOPS provisioned.
• Co-locate the WAL with the blocks by default. Splitting the WAL onto a separate device is an optimisation for high-IOPS hosts with spare devices.
• Set retention.size to 80% of the disk. Leave 20% for the WAL and the head block.
• Alert on the storage metric, not just the disk. The TSDB-specific metric (prometheus_tsdb_storage_blocks_bytes) is the right signal; the disk-level metric (node_filesystem_avail_bytes) is the floor.
• Back up the TSDB path. A snapshot tool or a filesystem-level snapshot is the minimum. The promtool tsdb create-snapshot command creates a consistent snapshot of the on-disk blocks.

Verification

You should now be able to answer:

• What I/O path is the WAL on, and what latency does it require?
• What I/O path is the compactor on, and what IOPS does it require?
• What is the right ratio between retention.size and the disk capacity?
• What is the metric that warns the head block is stalling?
• When is EBS gp3 sufficient for Prometheus, and when is io2 required?

Quiz