Capacity Planning

The end-of-month invoice arrives. The observability line item is 2.2x the forecast. The forecast was a number a previous engineer wrote in a spreadsheet eight months ago and never re-checked. The on-call team has spent the last week turning off verbose debug streams to recover the budget. The same pattern will repeat in three months because nobody is re-computing the forecast from the live ingest rate.

This lesson is the formula and the habit behind that forecast.

What capacity planning means for Loki

Capacity planning in the retention context is the prediction of two values forward in time:

1. Bucket size — the steady-state bytes stored in object storage at any given moment, given current ingest and current retention.
2. Ingest growth — how fast (1) is rising or falling per month, so the forecast for next month is honest.

The first is a function of the runtime; the second is a function of the workload. Both are needed; neither is hard once the metrics exist.

Why a sysadmin cares

Because the cost of being wrong falls on one of three budget lines:

• Storage over-spend. Bucket is twice the forecast; finance asks why.
• Storage under-spend plus outage. Bucket is twice the forecast and the ingest path is now throttling at the distributor; user-visible alerts start firing.
• Wasted engineering time. Bucket grows by 20% a month forever; nobody notices until the bucket-prefix fill alarm fires at 03:00 and the runbook now takes 30 minutes to run.

A monthly forecast written to a dashboard prevents all three.

The capacity equation

  bucket_size  =  ingest_rate  *  retention_seconds
                 /  compression_ratio

The four terms, in production units:

A worked example:

  r =  10 MB / s                    (sum of ingest)
  t =  30 days = 2,592,000 s
  c =  8                            (typical Loki chunk gzip ratio)
  b =  (10e6 * 2.592e6) / 8
    =  ~3.24 TB

That is the steady-state bucket size for that tenant at that retention. The cost (S3 Standard at $23 / TB / month) is:

  3.24 TB * $23 / TB / month  =  ~$75 / month

Doubling ingest doubles cost. Doubling retention doubles cost. The relationship is linear in both, which is what makes the forecast manageable.

How to plan it

The forecast loop runs monthly. The inputs are metrics, not spreadsheets.

# Byte ingest rate, averaged over the last 7 days.
# This is the "r" in the formula, summed across all
# distributors. Excludes traffic that did not make it past
# the distributor (rate-limited).
sum(
  rate(loki_distributor_bytes_received_total[5m])
)
# expected units: bytes / sec

# 30-day average. Use this for the steady-state calculation,
# not the instantaneous rate, to absorb business-day spikes.
sum(
  avg_over_time(
    rate(loki_distributor_bytes_received_total[5m])[30d:5m]
  )
)

# Linear regression for next month's forecast.
# Grafana renders this as a thin line on the same panel as
# the live rate, showing where next month is likely to be.
predict_linear(
  sum(rate(loki_distributor_bytes_received_total[1h]))[30d],
  30d * 24 * 60 * 60
)

The forecast expressed in storage:

# forecast_bucket_size.sh
INGEST_BPS=$(curl -s 'http://prometheus/api/v1/query?query='\
'sum(rate(loki_distributor_bytes_received_total[5m]))' \
  | jq '.data.result[0].value[1] | tonumber')
RETENTION_S=$((30 * 24 * 60 * 60))
COMPRESSION=8
AWS_COST_PER_TB=23

BYTES=$(echo "$INGEST_BPS * $RETENTION_S / $COMPRESSION" | bc -l)
TB=$(echo "$BYTES / 1024 / 1024 / 1024 / 1024" | bc -l)
COST=$(echo "$TB * $AWS_COST_PER_TB" | bc -l)

echo "Forecast bucket: ${TB} TB"
echo "Forecast cost:   \$${COST} / month"

How to validate it

Three checks to confirm the forecast is honest:

# 1. The metric is fresh and the sum covers all distributors.
curl -s 'http://prometheus/api/v1/query?query='\
'sum(rate(loki_distributor_bytes_received_total[5m]))' \
  | jq '.data.result[0].value[1]'
# expected:
#   "10485760"    # 10 MB / s, matches plan

# 2. The actual bucket size from S3 matches the formula.
aws s3 ls --recursive s3://loki-chunks --summarize \
  --human-readable | tail -5
# expected:
#   Total Objects: 184,322
#   Total Size: 3.21 TiB

# 3. The forecast is on the same panel as the live rate,
#    with the predicted-linear overlay visible.
# (manual: open the dashboard; the next-month line is shown.)

When (1), (2), and (3) all line up, the forecast and the reality agree within a few percent. When (2) drifts above (1), something in the chunk store is accumulating extras — usually orphan streams, covered in lesson 6.

How it can fail

Security implications

The capacity-forecast dashboard exposes ingest rate, which is not itself sensitive, but it can include breakdown by X-Scope-OrgID. Treat the dashboard like every other read-only dashboard for access control: same RBAC path, same audit log.

The aws cli / gcloud cli / az cli credentials used to verify bucket size should be scoped to read-only (e.g. AmazonS3ReadOnlyAccess). A capacity-planning run that needs more than read is the wrong run.

Performance implications

• predict_linear is a Prometheus function that runs on the query path. At 30 days of rate() over distributors, the cost is small. At multi-year ranges over high-cardinality breakdowns, the same query can dominate the Prometheus server. Constrain the dashboard queries to the time range the forecast actually needs.
• avg_over_time over long windows is similarly cheap but should be evaluated on rules / recording rules if the dashboard refreshes every second. A one-minute cache is the simplest fix.
• Grafana panel rendering itself can hit the bucket API for older lookback windows. Verify the dashboard with --no-cache once and then enable caching at the source.

Production guidance

• Keep one recording rule that captures the steady-state ingest rate, in bytes per second, with a “this value vs last month’s” annotation.
• Run the forecast script as a weekly cron, write the output to a markdown file in the same repo as the retention map, and review the file in the same weekly change-control meeting.
• Show the headroom band on the same dashboard as the ingest rate. The line above is “what we used”; the band is “what we plan for”; the difference is the working space.
• Re-measure the compression ratio on real data every quarter, not on the docstring.

Verification

You should now be able to answer:

• What four terms feed into the bucket-size formula, and which one changes most under workload?
• Why is avg_over_time a better input than the live rate for a steady-state forecast?
• What is the headroom band for, and what is a sensible default value?
• Which Loki metric sums correctly across all ingesters for a fleet-wide byte rate?
• Why does the forecast break if the compression ratio is assumed constant?

Quiz