Capacity Headroom

It is 02:14 on a Saturday. A retailer has just launched a flash sale. The monitoring Prometheus was sized for the forecast peak — exactly, with no margin. The first ten minutes of the sale push active series 20 percent above forecast. The head block OOMs at 02:18. The WAL replay takes 18 minutes. Rule evaluation is silent through the window. The retailer is blind during the highest-stakes ten minutes of the quarter.

The arithmetic was correct. The margin was zero. The margin is what this lesson is about.

What headroom is

Headroom is the working space between the steady-state workload and the platform’s budget on each tier. It is the buffer that absorbs the difference between the forecast and the actual — peaks, deploys, traffic anomalies, retention extensions, cardinality drift. The platform has four tiers of capacity, and each tier has its own headroom:

• Memory headroom. Resident bytes available between the Prometheus working set and the container limit. Sized to absorb a 30-50 percent series spike without OOM.
• Disk headroom. Bytes available between the on-disk data set and the data-volume cap. Sized to absorb a retention extension or a compaction backlog.
• Ingest headroom. Spans per second, MB per second, or samples per second available between the steady-state rate and the rate limit. Sized to absorb a 2-5x peak burst.
• Bucket headroom. Bytes available between the steady- state Loki/Tempo bucket and the next price-tier breakpoint or volume cap.

The four are linked. A platform that has ingest headroom but no memory headroom will OOM on the spike the ingest headroom allowed in. A platform that has memory headroom but no disk headroom will start deleting blocks to enforce retention.size regardless of age.

Why a sysadmin cares

Because the cost of zero headroom falls on operations, not on the spreadsheet.

• The OOM at peak. The platform is sized to forecast; the peak is 30 percent above forecast; the head block dies.
• The retention cliff. The bucket is sized to forecast; the ingest rate is 20 percent above forecast; the bucket crosses a price-tier breakpoint; the month-end invoice jumps 12 percent.
• The orphan stream. A misconfigured agent emits a stream with no retention; the bucket fills the headroom before anyone notices; the compactor starts deleting legitimate chunks to enforce the size cap.

The discipline is to set the headroom target before the platform is built, alert at 80 percent of the target, and re-derive the target every quarter as the workload shifts.

How it works: the four headroom bands

Each tier has one equation.

  headroom_fraction
    =  (budget  -  steady_state)  /  budget

  working_space
    =  budget  *  headroom_fraction

  tripwire
    =  budget  -  working_space
       (the value that fires the 80% alert)

For a 64 GiB Prometheus host with 5 million steady-state series at 4 KiB per series:

  budget          =  40 GiB (container limit)
  steady_state    =  20 GiB (5M series * 4 KiB)
  headroom_frac   =  (40 - 20) / 40       =  50%
  working_space   =  20 GiB
  tripwire        =  budget  -  0.5 * working_space
                  =  40 GiB  -  10 GiB
                  =  30 GiB  (alert when RSS above 30 GiB)

The 50% headroom target absorbs a 100 percent series spike (5M to 10M) without OOM. The 30 GiB tripwire is the alert threshold; RSS above 30 GiB is the leading indicator.

The same shape applies on the disk and bucket tiers:

  disk (data volume 200 GB):
    budget          =  200 GB
    steady_state    =  140 GB (70% target)
    headroom_frac   =  30%
    tripwire        =  200 GB  -  0.5 * 60 GB
                    =  170 GB  (alert when blocks above 170 GB)

  bucket (S3 standard):
    budget          =  3.6 TB  (just above the 3.5 TB tier
                                breakpoint)
    steady_state    =  2.0 TB  (under the breakpoint)
    headroom_frac   =  44%
    tripwire        =  3.2 TB  (alert when bucket above 3.2 TB)

  ingest rate:
    budget          =  20 MB / s (per-tenant limit)
    steady_state    =  10 MB / s
    headroom_frac   =  50%
    tripwire        =  16 MB / s (alert when 5-min rate
                                  above 16 MB / s)

The four are written into one file with one owner.

How to configure it

A headroom plan is one alert per tier and one target per tier. The targets belong in version control next to the capacity file.

1. The memory tripwire. A scrape-time alert on RSS as a fraction of the container limit:

groups:
  - name: capacity-headroom
    rules:
      - alert: HostMemoryHeadroomSmall
        expr: |
          (
            node_memory_MemAvailable_bytes
            - container_memory_working_set_bytes{container="prometheus"}
          )  /  node_memory_MemTotal_bytes  <  0.20
        for: 15m
        labels: {severity: critical, team: observability}
        annotations:
          summary: 'Prometheus has less than 20 percent
                     memory headroom'
          description: |
            MemAvailable minus the container working set is
            below 20 percent of total memory. The OOM
            tripwire is within minutes.

2. The series headroom. A scrape-time alert on active series as a fraction of the platform cap:

      - alert: SeriesHeadroomSmall
        expr: |
          prometheus_tsdb_head_series
            /  on() group_left()  vector(7_000_000)
          >  0.80
        for: 30m
        labels: {severity: warning, team: observability}
        annotations:
          summary: 'Active series above 80 percent of cap'

3. The disk headroom. A scrape-time alert on the data directory size as a fraction of the volume cap:

      - alert: DiskHeadroomSmall
        expr: |
          prometheus_tsdb_storage_blocks_bytes
            /  on() group_left()  vector(200 * 1024 * 1024 * 1024)
          >  0.85
        for: 30m
        labels: {severity: critical, team: observability}

4. The ingest headroom. A scrape-time alert on Loki ingest as a fraction of the per-tenant limit:

      - alert: LokiIngestHeadroomSmall
        expr: |
          sum by (tenant) (
            rate(loki_distributor_bytes_received_total[5m])
          )  /  on(tenant) group_left()  vector(20 * 1024 * 1024)
          >  0.80
        for: 10m
        labels: {severity: warning, team: observability}

5. The bucket headroom. A daily scrape of the bucket size compared to the next tier breakpoint:

      - alert: S3BucketHeadroomSmall
        expr: |
          s3_bucket_size_bytes{bucket="loki-chunks-eu-west-1"}
            /  on() group_left()  vector(3.5 * 1024 * 1024 * 1024 * 1024)
          >  0.90
        for: 1h
        labels: {severity: warning, team: observability}

How to validate it

The tripwires are correct when the steady state sits below the threshold and a synthetic spike crosses it.

# READ-ONLY: where are the four tripwires relative to
# their budgets, right now?
curl -s 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=prometheus_tsdb_head_series' | jq .
# 5.6 million / 7 million cap = 80 percent, sitting on
# the tripwire. A 10 percent growth crosses it.

# READ-ONLY: the data directory size.
du -sh /var/lib/prometheus/data/
# Expected: below 170 GB; alert fires above.

# READ-ONLY: the Loki bucket size.
aws s3 ls --recursive s3://loki-chunks-eu-west-1 --summarize \
  --human-readable | grep 'Total Size'
# Expected: below 3.2 TB; alert fires above.

# READ-ONLY: the Loki ingest rate, summed.
sum(rate(loki_distributor_bytes_received_total[5m]))
# Expected: below 16 MB / s; alert fires above.

A simulated spike — a single job temporarily raising its sample_limit — should cross each tripwire. If a tripwire does not fire on a synthetic spike, the threshold is wrong.

How it can fail

1. The zero-margin forecast. The platform is sized for the forecast peak with no buffer. The actual peak is 20 percent above forecast; the platform OOMs. Symptom: prometheus_tsdb_head_series crossing the cap during the peak; node_memory_pressure_total rising through the same window.
2. The price-tier breakpoint. The bucket sits within 5 percent of a tier breakpoint; a small spike crosses the line; the next invoice is 12 percent above the prior month. Symptom: month-end invoice jumps despite a forecast that was only modestly above the line.
3. The orphan-stream stealth growth. A misconfigured agent emits a stream that ages but is never deleted; the bucket fills the headroom before anyone notices; the compactor starts deleting legitimate chunks to enforce the size cap. Symptom: s3_bytes_used climbing while loki_distributor_bytes_received_total is flat.
4. The headroom target that does not match the workload. A platform with daily 5x peaks is sized with a 25 percent headroom; the platform OOMs at every peak. Symptom: recurring critical alerts at the same hour every day.
5. The tripwire on the wrong axis. A headroom alert on RSS without an alert on series count misses the series-driven OOM. Symptom: the platform dies while the alert is green.
6. The headroom target that is not re-derived. A workload that grew 30 percent in a quarter still uses the same headroom target as the prior quarter; the target is no longer a buffer. Symptom: the headroom alert fires earlier every month as the workload approaches the cap.

How to troubleshoot it

Cheap diagnostic first.

1. Which tier is the failure on? Read prometheus_tsdb_head_series, process_resident_memory_bytes, prometheus_tsdb_storage_blocks_bytes, and loki_distributor_bytes_received_total together. The tier closest to its cap names the failure.
2. Read the peak-to-average ratio for that tier. A memory tier with a peak-to-average of 5x needs a 50 percent headroom; a tier with a 1.2x ratio can run on 25 percent. The headroom target is a function of the ratio.
3. Read the trend over the last 90 days. A tier that has grown 10 percent over the quarter is consuming the headroom by design; re-derive the target from next quarter’s forecast.
4. Confirm the price-tier layout. If the platform sits close to a tier breakpoint, the headroom target must be sized to absorb the spike without crossing the breakpoint. A platform that crosses a breakpoint by 2 percent pays the same price as a platform that crosses it by 30 percent.

Security implications

The headroom dashboard exposes ingest rate, series count and bucket size — none of which is sensitive in itself, but each can include a breakdown by X-Scope-OrgID or service. Treat the dashboard like every other read-only dashboard for access control: same RBAC path, same audit log.

The credentials used to verify bucket size (aws s3 ls, gcloud storage, az storage) should be scoped to read-only. A capacity-planning run that needs more than read is the wrong run.

Performance implications

• The headroom alert itself is cheap. The PromQL expressions in this lesson are aggregate queries against a handful of gauges; they evaluate in milliseconds on a Prometheus host with normal load.
• Re-derivation has a cost. The quarterly re-derivation runs the live query against the live metrics; for a 90-day window with 5-second scrape resolution, the query is well within Prometheus’s range-vector budget. The discipline is to constrain the dashboard time range to the window the re-derivation needs.
• The headroom is unused capacity. A 50 percent memory headroom on a 64 GiB host is 32 GiB that is not allocated to Prometheus. The cost is real but small relative to the failure it absorbs.

Production guidance

• Pick the headroom target from the peak-to-average ratio. A 2x peak-to-average (budget at 2x the steady state) earns a 50 percent headroom target. A 5x peak-to- average earns an 80 percent target. The fraction is (peak - 1) / peak of the budget.
• Write the headroom target into the capacity file next to the budget. A headroom target that is not written down is a rumour.
• Alert at 80 percent of the working space, not at 100 percent of the budget. The 80 percent line is the leading indicator; the OOM is the lagging indicator.
• Re-derive the target every quarter. A workload that grows consumes its headroom by design; the target is never a constant for long.
• Pair the headroom alert with the capacity alert. The headroom alert fires when the working space is small; the capacity alert fires when the budget is reached. Together they tell you when to scale.

Verification

You should now be able to answer:

• What is headroom, and how does it differ from budget?
• What is the peak-to-average ratio that earns a 50 percent headroom target?
• Why is the tripwire at 80 percent of the working space, not at 100 percent of the budget?
• What is the failure shape of zero headroom on a tier with hard price breakpoints?
• How do you re-derive the headroom target when the workload grows?

Quiz