TSDB Compaction

A restart of Prometheus takes twelve minutes. Grafana is stuck on “loading” for every dashboard. The system is healthy. CPU is at 5 per cent. Disk has 200 GB free. The cause is not a network failure and it is not a missing file. The compactor goroutine has fallen behind and the on-disk block directory now holds hundreds of 2h blocks that should already have been merged into 6h and 24h blocks.

This lesson is about the TSDB compaction cycle: what runs, when it runs, what slows it down and what the operator is expected to monitor.

What it is

Prometheus stores recent samples in an in-memory head block and persists them to immutable on-disk blocks when each 2h window closes. The compactor goroutine merges adjacent small blocks into progressively larger ones (2h → 6h → 24h → larger), reducing the number of files the query engine must open and the per-block index overhead.

Compaction runs in the background. It does not affect the ingestion path. It does, however, affect query latency (more blocks means more files to open per query), disk usage (uncompacted blocks cost more bytes per sample), and restart time (more blocks to replay after a crash).

Why a sysadmin cares

Compaction pressure produces three production pains:

1. Slow restart after a crash. A Prometheus that has not compacted for 24 hours must reopen thousands of 2h blocks to rebuild the head-block index. A healthy platform restarts in 30 seconds; an uncompacted one can take 15 minutes or longer.
2. Disk usage growth. Each 2h block carries an index header and footer. A compactor that lags by 24 hours doubles the on-disk footprint compared with a fully compacted platform.
3. Query latency drift. Queries that span a 6h window must merge results from three 2h blocks in an uncompacted system, where a compacted system reads a single 6h block. The difference is invisible at 100 series per second and very visible at 10 million.

The lesson is that compaction is invisible until it is not. By the time the operator notices the symptom, the backlog is hours deep.

How it works

TSDB writes to the head block during each scrape. When 2h of samples have accumulated, the head is rotated into a new on-disk block and a fresh head block begins. The compactor then merges adjacent blocks into larger ones on a fixed schedule:

  time --->

  +----------+----------+----------+----------+----------+
  |  2h blk  |  2h blk  |  2h blk  |  2h blk  |   head   |
  +----------+----------+----------+----------+----------+
       \           \          |          /
        +-----------+----- 6h blk -----+
                       \         |
                        +--- 24h blk ---+
                                 |
                          (eventually merged
                           into larger blocks)

Each merge is a read of all source blocks, a sort-and-merge of series, and a write of the new block. The compactor cycles through three sizes by default: 2h → 6h after six hours, 6h → 24h after twenty-four hours, and 24h → larger after the retention window. The thresholds are configurable through storage.tsdb.min-block-duration and storage.tsdb.max-block-duration.

How to configure it

Compaction is configured through four flags on the Prometheus command line. The defaults are sensible for a modest platform; high-cardinality environments must override them.

# Severity: CONFIGURATION
# Command-line flags for the Prometheus binary
--storage.tsdb.path=/var/lib/prometheus
--storage.tsdb.min-block-duration=2h
--storage.tsdb.max-block-duration=24h
--storage.tsdb.retention.time=30d
--storage.tsdb.retention.size=200GB
--storage.tsdb.wal-compression

For the most common tuning case — high cardinality that cannot be reduced — raise max-block-duration to 48h or 72h. Larger blocks reduce the number of files the query engine opens, but they also lengthen the compaction window and increase the cost of a single compaction cycle.

For predictable disk usage, set both retention flags and treat whichever is smaller as the effective cap.

How to validate it

Validation is read-only. Confirm the on-disk layout, the head-block metrics, and the compactor metrics together:

# Severity: READ-ONLY
ls /var/lib/prometheus/ | sort
promtool tsdb list /var/lib/prometheus/

The first command lists the top-level block directories. The second prints each block’s min-time, max-time, duration and compaction source. A healthy platform shows blocks at three sizes (2h, 6h, 24h) and no 2h blocks older than the configured retention.

The platform metrics tell the same story from the running process:

# Severity: READ-ONLY
prometheus_tsdb_head_series
prometheus_tsdb_compactions_total
prometheus_tsdb_compactions_failed_total
prometheus_tsdb_compaction_duration_seconds

prometheus_tsdb_compaction_duration_seconds should move upward steadily. A flat line for more than an hour means the compactor has stalled.

How it can fail

Six failure shapes account for nearly every compaction incident in production:

1. Compactor falls behind under high series count. CPU or IO is too small for the number of active series. Symptom: prometheus_tsdb_compactions_total advances slowly; the number of 2h blocks on disk grows without bound.
2. Disk full. The compactor cannot write the merged block because the volume has run out of inodes or bytes. Symptom: the compactor log shows compaction failed: no space left on device on every cycle.
3. Block corruption after a crash. A torn write leaves one block with an inconsistent index. The compactor skips the corrupted block. Symptom: a 2h block remains on disk indefinitely while neighbours compact normally.
4. WAL replay takes hours. A long-out-of-restart platform must replay the WAL into the head block before scraping resumes. Symptom: Prometheus serves traffic but up lags by tens of minutes after a restart.
5. Retention smaller than the compaction cycle. Setting retention.time to a value that is shorter than max-block-duration produces blocks that are deleted before they finish compacting. Symptom: disk usage is low, but the compactor cannot make progress.
6. Disk write saturation during the merge. The compaction read and the scrape WAL append compete for IO. Symptom: scrape duration rises precisely when compaction runs.

How to troubleshoot it

The diagnostic order is consistent across all six failure shapes:

1. Confirm compaction is actually running. Inspect prometheus_tsdb_compaction_duration_seconds. If the counter has not advanced in over an hour, the goroutine is blocked.
2. Inspect the compactor log. Grep for compaction failed, no space left on device and block is corrupted. The message names the failure.
3. Confirm the head is not the bottleneck. Inspect prometheus_tsdb_head_series. A head with tens of millions of active series will starve the compactor of CPU regardless of any tuning.
4. Inspect disk usage. df -h on the data volume. The compactor needs roughly twice the on-disk footprint during a merge — one block for read, one for write.
5. Inspect the WAL size. A WAL that has grown past a few GB will lengthen restart time considerably. Reduce retention.time if the WAL is unexpectedly large.
6. Inspect the runtime. runtime_go_goroutines should show the compactor goroutine alive and progressing.

Security implications

The TSDB data directory contains every sample Prometheus has collected for the duration of retention. Three disciplines matter in production:

1. Restrict the directory to the Prometheus user. Mode 0700 on /var/lib/prometheus and an explicit owner.
2. Encrypt at rest when the data is sensitive. TSDB files are not encrypted by the binary. Use a volume or block device that encrypts at rest.
3. Restrict the admin API. The --web.enable-admin-api flag enables block deletion. Without it the operator cannot remove a corrupted block by hand — a useful safety guard.

A leaked TSDB directory is a full disclosure of the platform’s recent history. Treat it with the same care as a database dump.

Performance implications

Compaction cost is dominated by three variables: active series count, retention window, and max-block-duration. The arithmetic is:

compaction_cpu_seconds_per_cycle
    = series * blocks_per_cycle * per_series_merge_cost

A platform with 10 million active series and a 24h max block spends roughly three times the compaction CPU of a platform with the same retention and 3 million active series. The cost is largely linear in series count, which is why cardinality control is the most powerful lever.

Verification

You should now be able to answer:

• What on-disk structure does Prometheus use to store samples, and what does the compactor merge?
• How would you confirm the compactor is making progress from the running process?
• What is the difference between min-block-duration and max-block-duration, and when does each matter?
• Why does a high active series count slow the compactor even when CPU is plentiful?
• What does the WAL do during a restart, and how long should it take?

Quiz