Capacity Planning for Storage

A team’s storage bill doubles in month six. They had budgeted for 5 TB of Prometheus and 20 TB of Loki at the start of the year. By month six they are using 14 TB of Prometheus and 58 TB of Loki. The CFO wants to know why. The platform team looks at the data and finds three contributors: a new exporter that added 2 M series to Prometheus, a verbose debug log stream from a new service that doubled Loki ingest, and a Tempo deployment that spans at 100% sampling and was never re-tuned. None of these were in the original capacity plan.

Capacity planning is the discipline that makes the storage budget predictable. Without it, the storage line item grows in proportion to whatever the engineering team ships.

What capacity planning for storage is

Capacity planning is the practice of forecasting the storage budget for each observability backend, validating the forecast against the actual usage, and adjusting the budget before the storage runs out.

   +-----------------+     +-------------------+     +-----------+
   |  Ingest rate    |     |  Retention time   |     |  Bytes    |
   |  (samples/s,    | --> |  (hours, days,    | --> |  per unit |
   |  bytes/s,       |     |   weeks, months)  |     |  (sample, |
   |  spans/s)       |     |                   |     |  byte,    |
   +-----------------+     +-------------------+     |  span)    |
                                                     +-----+-----+
                                                           |
                                                           v
                                                  Storage budget
                                                  + headroom + replicas

The inputs to the calculation:

• Ingest rate — the rate at which the backend accepts data. Prometheus samples per second. Loki bytes per second. Tempo spans per second.
• Retention time — how long the data is kept. Hours for some signals, days for others, months for the rare case.
• Bytes per unit — the storage cost per ingest unit. Bytes per sample for Prometheus. Bytes per log byte (after compression) for Loki. Bytes per span for Tempo.

The output is the storage budget per tier plus the headroom.

Why a sysadmin cares

Storage is the line item that grows fastest and is hardest to predict. Three failure shapes appear when the capacity plan is missing.

1. The disk that fills at 02:00. A team has no monitoring on the storage growth metric. The first time they hear about it is when dashboards go blank and an on-call engineer gets paged. The fix is monitoring on the storage metric of the relevant backend with an alert at 70% of the tier capacity.
2. The bill that doubles overnight. A new release adds a high-cardinality label or a verbose debug log stream. There is no per-stream ingest cap and no cardinality limit. The bill arrives at the end of the month and surprises the finance team. The fix is per-stream limits on the ingester and an alert on the rate of ingest.
3. The retention that compliance needs but the platform cannot answer. A regulator asks for 90 days of audit logs. The platform keeps 30. The data exists in the ingesters but was evicted from the hot tier before the export ran. The fix is per-stream retention overrides and a documented mapping between regulated streams and their retention values.

How it works

The Prometheus formula

The Prometheus storage budget is:

bytes = samples_per_second * bytes_per_sample
      * retention_seconds

The bytes-per-sample number is the average size of a sample on disk after compression. For a typical workload:

samples_per_second    = 50_000
bytes_per_sample      = 1.3 (Gorilla-compressed average)
retention_seconds     = 30 * 24 * 3600 = 2_592_000
bytes                 = 50_000 * 1.3 * 2_592_000
                      = 168 GB

The number is illustrative. The actual bytes-per-sample depends on label cardinality (more labels mean more bytes) and on the number of histogram buckets (a histogram with 50 buckets is 50 samples per scrape).

For a Prometheus that includes many histograms:

samples_per_second    = 200_000 (counter gauges histograms)
bytes_per_sample      = 2.0 (with histogram overhead)
retention_seconds     = 30 * 24 * 3600
bytes                 = 200_000 * 2.0 * 2_592_000
                      = 1.04 TB

The hot-tier budget is the local TSDB. The warm-tier budget is the remote_write destination (Mimir, Thanos). The warm-tier budget includes the replication factor (3x by default).

The Loki formula

The Loki storage budget is:

bytes = ingest_bytes_per_second * retention_seconds
      * (1 + replicas * 1/compression_ratio)
      * 1/chunk_efficiency

Loki compresses chunks with gzip; the compression ratio is typically 5-10x. The replication factor is typically 3 in production. The chunk efficiency is the ratio of useful log data to chunk overhead; small streams have low efficiency.

For a typical workload:

ingest_bytes_per_second  = 10_000_000 (10 MB/s)
retention_seconds        = 30 * 24 * 3600
compression_ratio        = 6
replicas                 = 3
chunk_efficiency         = 0.7 (70% useful data)
bytes                    = 10_000_000 * 2_592_000 * (1 + 3/6) * (1/0.7)
                         = 49.4 TB

The Loki budget is dominated by the ingest rate and the retention. The compression ratio is the lever that controls the budget; a team that ships verbose logs that do not compress well (already-encoded JSON, base64) sees the budget inflate.

The Tempo formula

The Tempo storage budget is:

bytes = spans_per_second * span_size_bytes
      * retention_seconds
      / compression_ratio
      * replicas

For a typical workload at 1% sampling:

spans_per_second       = 5_000_000 (1% of 500 M spans per day)
span_size_bytes        = 1024 (1 KB per span, including attributes)
retention_seconds      = 30 * 24 * 3600
compression_ratio      = 3
replicas               = 1 (Tempo does not replicate blocks by default)
bytes                  = 5_000_000 * 1024 * 2_592_000 / 3 * 1
                       = 4.4 TB

Tempo budgets scale linearly with sampling rate. A team that bumps sampling from 1% to 10% sees a 10x increase in the Tempo budget.

Headroom

The budget calculation is the steady-state budget. The production budget includes headroom for:

• Growth. Forecast at 2x the current rate for the next 18 months; provision for the forecast, not the current.
• Spikes. A deployment that ships a verbose log stream for two weeks; a Prometheus cardinality burst from a faulty label.
• Compaction overhead. The TSDB compactor needs 20% of the disk in addition to the blocks for temporary files.

The standard headroom rule is 20-30% above current usage plus the 18-month growth forecast. A team that provisions exactly to current usage is budgeting for a 02:00 incident.

Under the hood

The relevant inputs and their sources:

• Ingest rate. Prometheus: rate(prometheus_tsdb_head_series[5m]) is the rate at which new series are created. Loki: rate(loki_distributor_bytes_received_total[5m]). Tempo: rate(tempo_ingester_spans_received_total[5m]).
• Bytes per unit. Prometheus: derive from prometheus_tsdb_storage_blocks_bytes / total_samples. Loki: derive from sog_objctl_bucket_bytes / total_bytes. Tempo: derive from the block size divided by the number of spans.
• Retention. The retention is a configuration value. The actual retention may be lower if the compactor falls behind or the disk fills.

The capacity plan is validated by comparing the calculated budget against the observed usage at 30, 60, and 90 days. A discrepancy of more than 20% is a sign that the inputs are wrong or the workload has changed.

How to configure it

Prometheus local TSDB path and retention are command-line flags, not prometheus.yml keys:

# /etc/default/prometheus
ARGS="--storage.tsdb.path=/var/lib/prometheus \
      --storage.tsdb.retention.time=30d \
      --storage.tsdb.retention.size=300GB"

The --storage.tsdb.retention.size flag is the cap that the TSDB enforces when the disk fills. Set it to 80% of the disk capacity to leave room for the WAL and the head block.

Loki retention configuration:

limits_config:
  retention_period: 744h
  retention_stream:
    - selector: '{job="compliance-audit"}'
      priority: 1
      period: 2160h

The retention_stream block overrides the global retention for specific streams. The compliance-audit stream keeps 90 days; everything else keeps 31 days.

Tempo retention configuration:

compactor:
  compaction:
    block_retention: 744h
    compaction_window: 6h

Tempo’s block_retention is the period after which blocks are removed by the compactor. The compaction_window is the window size for compaction; smaller windows mean more frequent compactions but faster queries on recent data.

How to validate it

# READ-ONLY: Prometheus storage usage.
curl -fsS http://prometheus:9090/api/v1/status/tsdb | \
  jq '.data.headStats'
# {"numSeries": 1234567, "chunkCount": 89, ...}
curl -fsS http://prometheus:9090/api/v1/status/tsdb | \
  jq '.data.'
# (sum of all on-disk blocks)

# READ-ONLY: Prometheus storage blocks size.
du -sh /var/lib/prometheus/data
# 280G  /var/lib/prometheus/data

# READ-ONLY: Loki ingester rate.
curl -fsS http://loki:3100/metrics | \
  grep loki_distributor_bytes_received_total
# loki_distributor_bytes_received_total{...} 1.234e+10

# READ-ONLY: Loki bucket size.
aws s3api list-objects --bucket loki-prod \
  --query "sum(Contents[].Size)" --output text
# 58473829123

# READ-ONLY: Tempo ingester spans.
curl -fsS http://tempo:3200/metrics | \
  grep tempo_ingester_spans_received_total
# tempo_ingester_spans_received_total{...} 4.5e+08

A clean validation: the on-disk usage is below 70% of the provisioned capacity; the ingest rate is steady; the forecast matches the actual usage at the 30/60/90 day marks.

How it can fail

The most expensive capacity-planning failures, in order of how often they appear in incident reviews.

1. No monitoring on storage growth. The team never measures prometheus_tsdb_storage_blocks_bytes or sog_objctl_bucket_bytes. The first time they hear about the disk filling is at 02:00 when the dashboards go blank. Symptom: an on-call engineer gets paged; the only fix is to grow the disk or shorten the retention.
2. Single-label cardinality explosion. A team adds a request_id label to a high-traffic service. The series count explodes by 10x; the Prometheus budget explodes by 10x; the disk fills in days. Symptom: the Prometheus memory and disk usage climb without bound.
3. Verbose log stream not capped. A new release sets log level to debug for one service. The Loki ingest doubles overnight. There is no per-stream cap. Symptom: the Loki bill doubles; the bucket size doubles; the compactor falls behind.
4. Sampling rate never re-tuned. A team deploys Tempo at 100% sampling for a single service. The Tempo budget grows linearly with the service traffic. Symptom: the bucket size grows without bound; the bill surprises the finance team.
5. Retention set without a per-stream override. A team sets Loki retention to 30 days globally. A regulated stream needs 90 days. The override is not configured. Symptom: audit data is unqueryable after 30 days; a regulatory finding follows.
6. Forecast never re-run. The capacity plan was right in January. The workload changed in March. The plan was not re-run. By July the disk is at 90%. Symptom: the team scrambles to grow the disk; the on-call engineer gets paged at 02:00.

How to troubleshoot it

The diagnostic order is “is the disk filling?”, “is the ingest rate growing?”, “is the per-unit size growing?”, “is the retention correct?”.

1. Check the disk. prometheus_tsdb_storage_blocks_bytes for Prometheus. sog_objctl_bucket_bytes for Loki. tempo_ingester_* for Tempo. A metric that is approaching the cap is the failure.
2. Check the ingest rate. rate(prometheus_tsdb_head_series[5m]) for Prometheus. rate(loki_distributor_bytes_received_total[5m]) for Loki. rate(tempo_ingester_spans_received_total[5m]) for Tempo. A rate that is growing without bound is a sign of a cardinality or ingest burst.
3. Check the per-unit size. Compare the on-disk bytes to the total ingest. A per-unit size that is growing without bound is a sign of an encoding or compression change.
4. Check the retention. The retention is a configuration value; verify it is the value the team thinks it is.
5. Re-run the forecast. The capacity plan is wrong if the inputs have changed. Re-run with the current numbers.

Security implications

• Cardinality budgets are an access-control concern. A label that contains user IDs, session IDs, or other per-request identifiers is a privacy risk. The cardinality budget prevents accidental exposure; the label hygiene prevents deliberate exposure.
• Retention policies are a compliance concern. A retention that is shorter than the regulatory requirement is a compliance failure. A retention that is longer than the regulatory requirement is a cost failure. The retention policy must match the regulatory requirement, not the team’s preference.
• Forecast accuracy is a security concern. A team that cannot forecast storage growth cannot budget for the controls (encryption, access logs) that the storage requires. The capacity plan is the foundation for the security plan.

Performance implications

• Storage budget growth and query latency are coupled. A 2x storage budget means 2x more data to scan on every query. A team that grows the storage without growing the query budget sees query latency rise linearly.
• Compaction cost scales with storage size. The TSDB compactor must read every block to compact it. A 2x storage budget means 2x the compactor work. The compactor host must be sized for the worst-day compaction, not the average.
• Cardinality growth and query cost are coupled. A 2x series count means 2x the time-series the query must iterate. A team that grows cardinality without growing the query budget sees query latency rise linearly.

Production guidance

• Forecast at 2x current. Provision for the 18-month forecast, not the current rate.
• Re-run the forecast every quarter. The workload changes; the plan must change with it.
• Alert at 70% of capacity. The disk-fills-at-02:00 failure mode is preventable with monitoring and an alert.
• Set retention per-stream, not just globally. Default retention applies to the bulk of streams; per-stream overrides cover compliance and audit needs.
• Validate the inputs. The bytes-per-unit and ingest rate inputs must be measured, not assumed.

Verification

You should now be able to answer:

• What are the three inputs to the Prometheus storage budget calculation?
• What is the difference between volume growth and cardinality growth?
• How much headroom should be added to the steady-state budget?
• Why must the capacity plan be re-run quarterly?
• Which metric warns that the Prometheus disk is approaching capacity?

Quiz