Avoid False Precision

A team plans the new observability platform on a Monday. The plan arrives in a spreadsheet with seven significant figures: 4,287 active series, 11.4 MB/s ingest, 6.7 GB bucket. The plan is presented as a fact. Six weeks later the platform is in production. The active series are 220,000; the ingest is 32 MB/s; the bucket is 18 TB. The plan was off by an order of magnitude on every axis.

The mistake was not the math. The math is the math. The mistake was treating the output of the math as a precise number when every input was an order-of-magnitude guess. This lesson is the discipline of estimating with an error bar, planning at the upper bound, and recognising the estimate for what it is: a guess with a margin.

What estimation is in this context

Estimation in the capacity context is the prediction of four values for a platform that does not yet exist:

1. How many active series the new workload will produce.
2. How many bytes per second the new workload will ingest.
3. How many spans per second the new workload will emit.
4. How much disk and memory the platform will need given the first three.

Every input is a guess. The output is honest when the guess carries an error bar and the plan is sized at the upper bound of that bar. A 2x over-estimate is acceptable; the platform has unused capacity that absorbs the next year’s growth. A 0.5x under-estimate is not; the platform OOMs in the first month and the on-call team is paged during launch.

Why a sysadmin cares

Because the cost of an under-estimate is paid in operations, not in the spreadsheet.

• The OOM at launch. The platform is sized at the estimate; the actual workload is 2x the estimate; the platform OOMs before the launch announcement is finished.
• The dropped-line incident. The per-tenant rate limit is set at the estimate; the actual peak is 2x the estimate; the distributor rejects lines during the launch traffic spike.
• The bucket fills in the first week. The retention is set at the estimate; the actual ingest is 2x the estimate; the bucket hits the volume cap before the first retention cycle.

The cost of an over-estimate is paid in unused capacity — RAM that could run another workload, disk that could hold another tenant. The cost is real but small. The cost of an under-estimate is paid in operations.

How it works: the four-input methodology

Each input is one number with an error bar. The estimate is the upper bound of the bar.

  estimate   =  nominal  *  growth_multiplier

  error_bar  =  +/-  50 percent   (the planning band)

For a new service with a nominal design of 1,000 active series, 5 MB/s log ingest, 200 spans/s, 30 days retention:

  active_series   =  1 000  *  2         =  2 000
  log_ingest      =  5 MB/s  *  2         =  10 MB/s
  span_ingest     =  200  *  2           =  400 spans/s
  memory          =  2 000  *  4 KiB     =  ~8 MiB
  bucket          =  10 MB/s  *  86400  *  30 / 8
                   =  ~3.24 TB

The numbers are guesses. The 2x multiplier is the planning band; the platform is sized at the upper bound of the band. The estimate says “between 4 MiB and 16 MiB of memory, and between 1.6 TB and 6.5 TB of bucket.” The plan sizes at 16 MiB and 6.5 TB. The cost of the unused half is small; the cost of being below the actual is operational.

The four-step methodology:

1. Pick the nominal from the design. The service spec, the load test, the SLA document, the instrumentation library’s defaults. The nominal is the best guess, not the best case.
2. Apply a 2x planning multiplier to each input. The multiplier absorbs the difference between the design and the actual: traffic the team did not anticipate, cardinality the design did not enumerate, retention extensions the SLA required.
3. Run the arithmetic from the upper bound. Memory, disk, ingest rate, bucket size — every term is the upper bound of the planning band.
4. Re-derive from the live metric within 30 days of go-live. The estimate is a guess; the live metric is the truth. The capacity plan is re-derived from the live metric within the first month; the planning multiplier is replaced by the actual ratio.

How to configure it

An estimate lives in a documented file alongside the deployment plan. The file is the answer to “where did this number come from” when the platform is in production and finance asks why the budget is 2x the estimate.

1. The estimate file. Capture the four inputs, the multiplier, and the upper-bound plan in one place:

# estimates/observability-platform.yaml
workload:
  service: checkout-api
  nominal:
    active_series: 50_000
    log_ingest_mbps: 8
    span_ingest_sps: 5_000
    retention_days: 30
  planning_multiplier: 2.0

plan_upper_bound:
  active_series: 100_000
  log_ingest_mbps: 16
  span_ingest_sps: 10_000
  memory_gib: 4
  disk_gib: 6_500          # bucket size at upper bound
  rate_limit_mbps: 32      # 2x the upper bound

re_forecast_after: 30d
review_owner: observability-platform

The file is owned by a named team. The estimate is not the truth; the live metric is. The estimate is the starting point.

2. The capacity alert. Same shape as lesson 1, but with thresholds derived from the upper-bound plan rather than the steady state:

groups:
  - name: capacity-budget
    rules:
      - alert: HeadSeriesAbovePlanUpperBound
        expr: prometheus_tsdb_head_series > 100_000
        for: 30m
        labels: {severity: warning, team: observability}
        annotations:
          summary: 'Active series above the estimate upper bound'
          description: |
            The pre-launch estimate used 50_000 active series
            with a 2x planning multiplier; 100_000 is the
            upper bound. Crossed after 30 days; re-derive
            the forecast.

      - alert: LokiIngestAbovePlanUpperBound
        expr: |
          sum(rate(loki_distributor_bytes_received_total[5m]))
            > 16 * 1024 * 1024
        for: 15m
        labels: {severity: warning, team: observability}

The alert is the early indicator that the upper bound was too low. The first time it fires, the estimate is re-derived from the live metric; the platform is re-sized.

3. The re-forecast rule. A recording rule that captures the live metric for the post-launch review:

groups:
  - name: capacity-forecast
    rules:
      - record: observability:active_series:30d_avg
        expr: avg_over_time(prometheus_tsdb_head_series[30d])

      - record: observability:log_ingest_mbps:30d_avg
        expr: |
          sum(avg_over_time(
            rate(loki_distributor_bytes_received_total[5m])[30d:5m]
          ))  /  1024 / 1024

The 30-day average is the input to the post-launch review. The review compares the average to the nominal estimate; if the average exceeds the upper bound, the estimate was wrong, and the platform is re-sized.

How to validate it

The estimate is honest when the live metric lands within the planning band. The validation runs at three checkpoints.

At design time. The estimate file is reviewed by a second engineer. The review checks that the planning multiplier is appropriate for the workload’s known volatility; that the upper bound is sized for the multiplier; that the rate limits are sized above the upper bound.

# READ-ONLY: review the estimate file.
cat estimates/observability-platform.yaml

At go-live. The first hour of production traffic is compared to the estimate. The active series, the ingest rate, and the span rate are read from the live metric and compared to the upper bound. A value that exceeds the upper bound at go-live is a sign the estimate was wrong before launch.

# READ-ONLY: live active series vs upper bound.
prometheus_tsdb_head_series > 100_000
# Alert fires at the upper bound.

# READ-ONLY: live Loki ingest vs upper bound.
sum(rate(loki_distributor_bytes_received_total[5m]))
  > 16 * 1024 * 1024

At 30 days. The 30-day average of the live metric is the post-launch review. The review answers two questions: did the workload land within the planning band, and is the forecast for next quarter honest?

# READ-ONLY: the 30-day average ingest in MB/s.
observability:log_ingest_mbps:30d_avg

# READ-ONLY: the 30-day average active series.
observability:active_series:30d_avg

A workload that lands at 60 percent of the upper bound was over-estimated; the next deployment can size closer to the nominal. A workload that lands at 110 percent of the upper bound was under-estimated; the platform is re-sized before the next month’s traffic arrives.

How it can fail

1. The estimate presented as a fact. The plan says “4,287 active series” with no error bar; the actual workload is 220,000. Symptom: the platform OOMs before launch is finished.
2. The planning multiplier applied to one input and not the others. Memory is sized at 2x the nominal, but the rate limit is set at the nominal. Symptom: the platform has memory headroom but rejects lines during the launch spike.
3. The estimate not re-derived after launch. The plan says 50,000 series; the live metric says 80,000 after 30 days; nobody re-forecasted. Symptom: the platform OOMs in the third month.
4. The estimate re-derived from the spike, not the average. A launch-day traffic spike is treated as the steady state. Symptom: the next month’s forecast is 2x the actual; the budget is wasted.
5. The planning multiplier frozen at the design number. A 2x multiplier is right for a backend integration; a 5x multiplier is right for a consumer- facing workload. Symptom: the consumer-facing workload OOMs every peak.
6. The estimate that nobody owns. The plan is in a shared drive; nobody is named as the owner. Symptom: the post-launch review does not happen; the plan stays in the shared drive; the platform OOMs when the workload grows.

How to troubleshoot it

Cheap diagnostic first.

1. Where did the number come from? Read the estimate file. The number is honest only if it carries an error bar; a precise number with no provenance is a guess dressed as a fact.
2. What is the live metric? Read prometheus_tsdb_head_series and sum(rate(loki_distributor_bytes_received_total[5m])). The live metric is the truth; the estimate is the starting point.
3. Did the planning multiplier match the workload’s volatility? A 2x multiplier for a 1.2x peak-to- average is over-sized; a 2x multiplier for a 5x peak-to-average is under-sized. The multiplier is a function of the volatility.
4. Has the estimate been re-derived since go-live? A 30-day average that exceeds the upper bound is a sign the estimate was wrong; the platform is re-sized.

Security implications

The estimate file exposes the planned ingest rate, the planned series count, and the planned retention. None of this is sensitive in itself, but each can include a breakdown by service or tenant that hints at the platform shape. Treat the estimate file like every other planning document for access control: same RBAC path, same audit log.

The capacity alert thresholds are derived from the estimate; an attacker who knows the thresholds can plan a cardinality attack that lands just below the alert. The mitigation is to size the alert thresholds below the estimate upper bound, not at it; the planning multiplier absorbs the gap.

Performance implications

• The estimate is a one-time cost. The arithmetic runs once at design time and once at the post-launch review; it does not run on the platform’s hot path.
• The capacity alert is cheap. A handful of PromQL expressions against gauges; cost is in milliseconds.
• The 30-day recording rule is cheap. An avg_over_time over a 30-day window evaluates every five minutes; cost is well within the Prometheus budget.

Production guidance

• Write the estimate as a range with a planning multiplier. A range is honest; a precise number is a lie.
• Pick the planning multiplier from the workload’s known volatility. A 1.2x peak-to-average earns a 1.5x multiplier; a 5x peak-to-average earns a 3-5x multiplier.
• Name an owner for the estimate file and the post-launch review. An estimate without an owner is an estimate that nobody re-derives.
• Re-derive the estimate from the live metric within 30 days of go-live. The estimate is a guess; the live metric is the truth.
• Re-derive the compression ratio, the bytes-per-series figure, and the spans-per-trace figure from the live platform before relying on the prior-engineer’s numbers.

Verification

You should now be able to answer:

• What is the difference between an order-of-magnitude estimate and a precision lie?
• Why is a 2x planning multiplier the right default for most production workloads?
• What three checkpoints validate an estimate (design review, go-live, 30-day post-launch review)?
• Why is the cost of an over-estimate acceptable and the cost of an under-estimate not?
• How do you pick the planning multiplier from the workload’s known volatility?

Quiz