Growth Modelling

The end-of-month invoice arrives. The observability line item is 2.2x the forecast. The forecast was a number a previous engineer wrote in a spreadsheet eight months ago and never re-checked. The on-call team has spent the last week turning off verbose debug streams to recover the budget. The same pattern will repeat in three months because nobody is re-computing the forecast from the live ingest rate.

This lesson is the formula and the habit behind that forecast. The arithmetic is small. The discipline of running it from the live metric, on a cadence, is what separates a capacity review that prevents outages from one that records them.

What growth modelling is

Growth modelling is the prediction of two values forward in time from the live metrics:

1. Next-period ingest. The expected ingest rate at the end of the forecast window, derived from a regression on the live metric.
2. The growth shape. Whether the workload is growing linearly, sublinearly (maturing) or superlinearly (still in adoption). The shape determines which regression and which cadence to use.

The two together describe what the platform will look like at the end of next month, next quarter, and next year. A healthy platform re-forecasts every quarter; an unhealthy platform re-forecasts when finance asks.

Why a sysadmin cares

Because the cost of an un-reforecasted forecast falls on one of three budget lines:

• Storage over-spend. Bucket is twice the forecast; finance asks why.
• Storage under-spend plus outage. Bucket is twice the forecast and the ingest path is now throttling at the distributor; user-visible alerts start firing.
• Wasted engineering time. Bucket grows by 20 percent a month forever; nobody notices until the bucket-prefix fill alarm fires at 03:00 and the runbook now takes 30 minutes to run.

A monthly re-forecast written to a dashboard prevents all three.

How it works: the forecast equation

The forecast for next month is a linear regression on the 30-day window of the live metric.

  next_period_ingest
    =  predict_linear(metric, window)
       evaluated at the forecast horizon

PromQL has the function built in. For Loki:

# Forecast the next 30 days of Loki byte ingest.
predict_linear(
  sum(rate(loki_distributor_bytes_received_total[1h]))[30d],
  30d * 24 * 60 * 60
)
# Expected units: bytes per second at the forecast horizon.

For Prometheus active series:

# Forecast the next 30 days of active series.
predict_linear(
  prometheus_tsdb_head_series[30d],
  30d * 24 * 60 * 60
)
# Expected units: active series at the forecast horizon.

The function fits a least-squares line to the points in the range vector and extrapolates forward by the given number of seconds. The result is a single number — the predicted value at the horizon — which the dashboard shows as a thin line extending past the live data.

For a 30-day window showing a steady 2 MB/s/day growth, the forecast for next month is 30 * 2 = 60 MB/s above the last data point. The forecast grows linearly with the trend; the trend is what the function reads.

How to configure it

A growth forecast lives on the same dashboard as the live ingest rate. The forecast line is the panel’s prediction; the headroom band is the panel’s “is this OK” reference.

1. The forecast recording rule. Materialise the forecast so the dashboard does not recompute it on every panel refresh:

groups:
  - name: capacity-forecast
    interval: 5m
    rules:
      - record: loki:ingest:forecast_30d
        expr: |
          predict_linear(
            sum(rate(loki_distributor_bytes_received_total[1h]))[30d],
            30d * 24 * 60 * 60
          )

      - record: prometheus:active_series:forecast_30d
        expr: |
          predict_linear(
            prometheus_tsdb_head_series[30d],
            30d * 24 * 60 * 60
          )

A five-minute evaluation interval is the right cadence for a forecast that changes on a weekly timescale.

2. The dashboard panel. Plot the live value, the forecast, and the headroom band on the same panel:

# Live ingest rate.
sum(rate(loki_distributor_bytes_received_total[5m]))

# Forecast for next 30 days.
loki:ingest:forecast_30d

# Headroom band, as a constant.
vector(16 * 1024 * 1024)   # tripwire at 80 percent of 20 MB/s

The dashboard renders the live value as a thick line, the forecast as a thin line that extends past the live data, and the tripwire as a horizontal threshold. When the forecast crosses the tripwire, the conversation about scaling is due.

3. The cadence rule. Re-derive the forecast on the cadence that matches the workload’s volatility:

# forecast-cadence.yaml — owned by the observability team
workloads:
  consumer-facing:
    re_forecast_cadence: monthly
    rationale: weekly cycle dominates the noise floor
  backend-integration:
    re_forecast_cadence: quarterly
    rationale: smooth 24-hour cycle; long windows are honest
  spike-prone:
    re_forecast_cadence: weekly
    rationale: peaks can shift the trend; long windows
                mask the shift

A consumer-facing workload with a weekly cycle earns a monthly re-forecast; a backend integration with a smooth 24-hour cycle earns a quarterly re-forecast; a workload with structural volatility (deploys, feature flags, launches) earns a weekly re-forecast.

How to validate it

The forecast is honest when the live value tracks the predicted line within sampling noise.

# READ-ONLY: the forecast recording rule's value.
loki:ingest:forecast_30d
# Expected: a single number close to the current live
# rate plus the expected 30-day drift.

# READ-ONLY: the live rate, summed.
curl -s 'http://prometheus/api/v1/query?query='\
'sum(rate(loki_distributor_bytes_received_total[5m]))' \
  | jq '.data.result[0].value[1]'
# Expected: within 10 percent of the forecast line at the
# same point in time. A 30 percent divergence means the
# trend has shifted.

# READ-ONLY: the per-tenant breakdown. A single tenant
# dominating the trend is the first place to look for
# adoption, debug-stream leak, or a structural change.
sum by (tenant) (rate(loki_distributor_bytes_received_total[1h]))

A simple shell capture for the weekly review:

# weekly_forecast.sh
LIVE=$(curl -s 'http://prometheus/api/v1/query?query='\
'sum(rate(loki_distributor_bytes_received_total[5m]))' \
  | jq '.data.result[0].value[1] | tonumber')
FORECAST=$(curl -s 'http://prometheus/api/v1/query?query='\
'loki:ingest:forecast_30d' \
  | jq '.data.result[0].value[1] | tonumber')

DRIFT=$(echo "scale=2; ($FORECAST - $LIVE) / $LIVE * 100" | bc)
echo "Live:    $(echo "$LIVE / 1024 / 1024" | bc -l) MB/s"
echo "Forecast:$(echo "$FORECAST / 1024 / 1024" | bc -l) MB/s"
echo "Drift:   ${DRIFT} percent over 30 days"

How it can fail

1. Forecast based on the rate from one month ago, not the live metric. Bucket is 30 percent over plan; the dashboard’s “next month” line points at a number that already happened. Symptom: the live line crosses the forecast line by mid-month; the gap is the un-reforecasted drift.
2. Window too short for the workload’s cycle. A 7-day window on a consumer-facing platform catches a single weekly peak and treats it as the trend. Symptom: forecast spikes every Monday; bucket oscillates between over- and under-forecast.
3. Window too long for the workload’s volatility. A 90-day window on a platform that shipped a major change last month still weights the prior regime. Symptom: forecast predicts the prior quarter’s trajectory; the new regime is ignored.
4. The structural change. A deploy introduces a new metric; a tenant onboard doubles a fleet’s traffic; an acquisition brings a platform with three times the current load. Symptom: the forecast jumps after the change; the prior 30-day window is invalidated; the forecast from the new 30 days is the only honest one.
5. The exponential shape that the linear regression misses. A consumer-facing workload in adoption grows at 3 percent per week, which is 36 percent per year compounded; a 30-day linear fit under-predicts the next quarter. Symptom: the forecast is honest for next month and a lie for next quarter; the next review finds the discrepancy.
6. Forecast lives only in a spreadsheet. Updated only when someone remembers. The dashboard and the spreadsheet drift; the spreadsheet wins the room but the dashboard wins the month. Symptom: every quarterly capacity review starts with “let me check the spreadsheet.”

How to troubleshoot it

Cheap diagnostic first.

1. Is the forecast close to the live? Read both. A divergence of more than 20 percent means either the forecast is stale or the trend has shifted. Confirm which.
2. Is the window the right length? For a consumer- facing workload, a 30-day window is the minimum; for a backend integration, 90 days is fine. A window that is too short oscillates; a window that is too long weights obsolete regimes.
3. Has the trend shifted in the last week? Compute the rate over the last 7 days and compare to the rate over the prior 7 days. A 30 percent week-over-week change is a structural event; the next 30 days will look different from the prior 30.
4. Is the forecast on the dashboard? A forecast that nobody reads is not a forecast. Confirm the panel is on the observability team’s own dashboard, not on a separate page that nobody opens.

Security implications

The forecast dashboard exposes ingest rate and series count — none of which is sensitive in itself, but each can include a breakdown by X-Scope-OrgID or service. Treat the dashboard like every other read-only dashboard for access control: same RBAC path, same audit log.

The credentials used to verify bucket size (aws s3 ls, gcloud storage, az storage) should be scoped to read-only. A re-forecast run that needs more than read is the wrong run.

Performance implications

• predict_linear is a Prometheus function that runs on the query path. At 30 days of rate() over distributors, the cost is small. At multi-year ranges over high-cardinality breakdowns, the same query can dominate the Prometheus server. Constrain the dashboard queries to the time range the forecast actually needs.
• avg_over_time over long windows is similarly cheap but should be evaluated on rules / recording rules if the dashboard refreshes every second. A five-minute cache is the simplest fix.
• The recording rule interval controls how often the forecast is re-evaluated. A five-minute interval is fine for a weekly-cadence workload; a one-minute interval wastes CPU on a forecast that will not change in that window.

Production guidance

• Keep one recording rule per workload that captures the next-period forecast. The rule’s value is the dashboard’s source of truth.
• Re-derive the forecast on the cadence that matches the workload’s volatility. Quarterly for a smooth backend; monthly for a weekly-cycling consumer platform; weekly for a workload in adoption.
• Show the forecast and the headroom band on the same panel as the live rate. The line above is “what we used”; the forecast is “what we expect”; the band is “what we plan for”; the difference is the working space.
• Re-derive the compression ratio on real data every quarter, not on the docstring.
• After every structural change (deploy, tenant onboarding, acquisition), reset the forecast window to the post-change regime. The pre-change window is no longer relevant.

Verification

You should now be able to answer:

• What PromQL function produces the next-period forecast, and what window does it use?
• Why is a 30-day window the right compromise for most workloads?
• How does the re-forecast cadence relate to the workload’s volatility?
• What is the failure shape of an un-reforecasted forecast after a structural change?
• How do you detect when the trend has shifted and the forecast is no longer honest?

Quiz