Logs Capacity

It is the 27th of the month. The observability bill has doubled. The on-call engineer opens Grafana, sees Loki ingest trending up and to the right, and reads the email from finance: “please reduce retention.” They search the Loki config for retention_period, find it, edit it to a tenth of the value, and ship the change without computing what the bucket size is now. Within an hour, the compactor is deleting chunks that an open investigation needs.

The mistake was planning capacity against the previous month’s number instead of the live metric. This lesson is the formula, the metric that drives it, and the cadence that keeps the formula honest.

What log capacity is

Log capacity in a Loki 3.x platform is the answer to four questions:

1. What is the steady-state bucket size in object storage given the current ingest rate and the current retention window?
2. What is the peak ingest rate the platform must accept without dropping lines at the distributor?
3. What is the live compression ratio, measured from the chunk store, not assumed from a docstring?
4. How much headroom sits between the steady state and the price-tier breakpoints or volume cap?

The four together describe what the platform will look like at the end of next month if the workload does not change. If the workload will change, the next lesson in this module covers forecasting; this lesson covers the static arithmetic.

Why a sysadmin cares

Three operational pains are specific to log capacity:

• The end-of-month invoice. Loki on object storage is a per-byte, per-request cost. A fleet that doubles ingest without a retention change doubles its storage spend.
• The dropped-line incident. A peak burst exceeds the per-tenant ingest rate limit at the distributor; the distributor rejects lines; dashboards show gaps. The limit was set against the average; the peak was not on the chart.
• The investigation that finds nothing. Retention was shortened to control cost; an incident took longer than the new window to detect; the evidence is gone. The retention change was right; the planning was not.

A monthly review that re-runs the arithmetic against the live metric prevents all three.

How it works: the equation

The bucket-size formula is one line. Every term is a metric the platform already exposes.

  bucket_size_bytes
    = ingest_bytes_per_second
      *  retention_seconds
      /  compression_ratio

For a fleet ingesting 10 MB per second, retained 30 days, with a measured compression ratio of 8:

  ingest       =  10 MB / s  =  10 * 1024 * 1024  B / s
  retention    =  30 d       =  30 * 86400        s
  compression  =  8
  bucket_size  =  (10 * 1024 * 1024 * 30 * 86400) / 8
               ~=  3.24 TB

At S3 Standard pricing of approximately $23 per TB-month, the steady-state storage line for that fleet is roughly $75 per month — and that is before request costs, before the compaction cycle’s LIST cost, and before any storage class transitions.

The arithmetic is linear in ingest and retention, which is what makes it tractable. Doubling ingest doubles the bucket. Halving retention halves it. The formula is honest when the inputs are honest.

How to configure it

A log capacity plan is four numbers: the fleet ingest rate, the per-tenant retention window, the measured compression ratio, and the price-tier breakpoints.

1. Confirm the retention pair. Retention in Loki 3.x requires both the master switch and a period:

# loki-config.yaml
limits_config:
  retention_period: 744h              # 31 days, global default
  overrides:
    payments:
      retention_period: 2160h         # 90 days, PCI window
    noisy-debug:
      retention_period: 168h          # 7 days, drop fast

compactor:
  retention_enabled: true             # required for any deletion
  compaction_interval: 5m
  working_directory: /loki/compactor

The pair is required. retention_period without retention_enabled: true silently does nothing.

2. Confirm the per-tenant rate limit. The distributor enforces ingestion_rate_mb per tenant. Set it above peak, not above average:

limits_config:
  ingestion_rate_mb: 20               # MB / s per tenant
  ingestion_burst_size_mb: 40         # burst tolerance
  per_stream_rate_limit: 5MB          # per-stream safety valve
  reject_old_samples: true
  reject_old_samples_max_age: 168h    # 7 days

A limit set against the average rejects the peak. A limit set against twice the peak is a reasonable upper bound.

3. Confirm the chunk encoding. Bigger chunks compress better; smaller chunks flush faster. The trade-off lives between retention safety and ingest latency:

ingester:
  chunk_encoding: gzip
  chunk_idle_period: 30m              # flush after this idle
  max_chunk_age: 2h                   # max chunk age
  chunk_target_size: 1572864          # 1.5 MiB target

A chunk_target_size of 1.5 MiB is the default and a defensible choice. Going much larger delays the first visibility of a new line and risks larger loss on ingester restart.

4. Confirm the bucket layout. The chunks prefix and the index prefix live in the same bucket but are billed separately:

storage_config:
  aws:
    s3: s3://loki-chunks-eu-west-1
    s3forcepathstyle: true
  boltdb_shipper:
    active_index_directory: /loki/index
    cache_location: /loki/cache
    shared_store_key_prefix: index/

How to validate it

The arithmetic only protects the budget if the operator reads the metrics that drive it.

# Byte ingest rate, summed across distributors.
# This is the fleet-wide "r" in the formula.
sum(rate(loki_distributor_bytes_received_total[5m]))
# Expected units: bytes per second.

# 30-day average, used for the steady-state calculation.
# The 30-day window absorbs business-day spikes.
sum(
  avg_over_time(
    rate(loki_distributor_bytes_received_total[5m])[30d:5m]
  )
)

# Compression ratio, measured against the chunk store.
# loki_ingester_chunk_compression_ratio is a per-chunk gauge;
# average it across the fleet.
avg(loki_ingester_chunk_compression_ratio)
# Expected: somewhere in 3-15 depending on log format.

# READ-ONLY: confirm the bucket size on object storage.
aws s3 ls --recursive s3://loki-chunks-eu-west-1 --summarize \
  --human-readable | tail -5
# Expected:
#   Total Objects: 184,322
#   Total Size: 3.21 TiB

The arithmetic only protects the budget if the four numbers — ingest, retention, compression, and bucket size — agree. When the formula says 3.24 TB and S3 says 3.21 TiB, the plan is honest. When the formula and the bucket diverge by 30% or more, something is accumulating extras — usually orphan streams or a retention gap.

A small shell script captures the loop in one place:

# forecast_bucket.sh
INGEST_BPS=$(curl -s 'http://prometheus/api/v1/query?query='\
'sum(rate(loki_distributor_bytes_received_total[5m]))' \
  | jq '.data.result[0].value[1] | tonumber')
RETENTION_S=$((30 * 24 * 60 * 60))
COMPRESSION=$(curl -s 'http://prometheus/api/v1/query?query='\
'avg(loki_ingester_chunk_compression_ratio)' \
  | jq '.data.result[0].value[1] | tonumber')

BYTES=$(echo "$INGEST_BPS * $RETENTION_S / $COMPRESSION" | bc -l)
TB=$(echo "$BYTES / 1024 / 1024 / 1024 / 1024" | bc -l)
echo "Forecast bucket: ${TB} TB"

How it can fail

1. Ingest rate set against the average, not the peak. A daily 9 a.m. burst is twice the off-peak rate. The distributor is configured for the average; the burst hits the per-tenant limit and is rejected. Symptom: loki_distributor_dropped_bytes_total spiking daily at 09:00; dashboards for the affected tenant show gaps.
2. Compression ratio assumed constant. A migration to structured JSON triples the ratio; a migration to verbose debug logs halves it. The formula now lies. Symptom: the bucket size on S3 diverges from the formula’s prediction by more than 30%.
3. Retention extended without re-running the formula. A single tenant’s window is doubled for a compliance audit; the bucket size ballooned, but the alert on s3_bytes_used only fires after the next billing report. Symptom: month-end invoice is 50% above forecast.
4. The orphan stream. A misconfigured agent emits a stream with no retention — every line ages but is never deleted. The chunk store grows monotonically; the compactor cannot find the index entry to apply retention. Symptom: s3_bytes_used climbing while loki_distributor_bytes_received_total is flat.
5. The price-tier breakpoint. Object storage pricing tiers sometimes have hard breakpoints at 100 TB or 1 PB. A 2% overshoot of the breakpoint is a 5-15% cost jump in the same calendar day. Symptom: month-end invoice is well above the prior month’s, despite the forecast being only modestly above the line.
6. Retention set below the slowest incident detection. An investigation at day 30 reports empty results. The on-call engineer cannot prove what was running. Symptom: post-mortem cites “logs unavailable for the detection window.”

How to troubleshoot it

Cheap diagnostic first.

1. Confirm the ingest metric is fresh and complete. sum(rate(loki_distributor_bytes_received_total[5m])) should be non-zero and should match the sum of per-tenant rates. A zero result means the distributor has no traffic; the metric is fine; the question is upstream.
2. Read the per-tenant breakdown. A single tenant dominating the fleet rate is the first place to look for label bloat, debug-stream leaks, or an outage that is amplifying error logs.
3. Confirm the compression ratio. A ratio that has shifted by more than 2x since the last capacity review invalidates the plan. The shift is usually a logging format change; check the changelog.
4. Compare the formula to the bucket. aws s3 ls --summarize on the chunks prefix. When the two diverge, the plan is wrong; the next lesson in this module covers forecasting, which is the right place to revisit.
5. Check the price tier. If the bucket sits within 10% of a tier breakpoint, the headroom target (next lesson in this module) needs to be above the breakpoint by at least the cost of the jump.

Security implications

• Tenant header trust. Per-tenant overrides are read from the request path. An ingress that trusts an attacker-controlled X-Scope-OrgID can read any tenant’s data. Authentication must terminate at the proxy, not at Loki.
• IAM scope for capacity checks. The credentials used for aws s3 ls --summarize should be scoped to read-only on the chunks prefix. A capacity-planning run that needs more than read is the wrong run.
• Label values as data. Loki labels are indexed. A label that carries a user identifier replicates the identifier into the index, the bucket, and any backup. Treat the index as a data store; classify what may flow through it.

Performance implications

• Compactor CPU. Roughly proportional to the number of streams, not the number of chunks. A fleet with hundreds of thousands of streams can saturate a single compactor; in microservices mode the compactor is sharded via the compactor.ring configuration.
• Compactor working directory. The compactor caches per-(tenant, stream) state on local disk. A 10 TB tenant with millions of streams may require 50-100 GB of working directory. Sizing this wrong is the second most common retention failure.
• Object-store API budget. Every deletion is an API call. The compactor issues LIST requests against the bucket prefix at every cycle; at very high chunk counts, the LIST cost can rival storage cost.

Production guidance

• Plan for peak, not average. The peak-to-average ratio for log ingest is typically 2-5x; a daily peak is the rule rather than the exception.
• Set the per-tenant rate limit at 1.5-2x the observed peak. Headroom on the rate limit is cheaper than dropped lines.
• Re-measure the compression ratio on real data every quarter. The ratio moves with log format; assume it does not and the plan drifts.
• Run the capacity script weekly and check the result against the prior week. A drift of more than 30% in either direction is the trigger for a review.
• Keep retention and storage-class transitions aligned: data that is being deleted in 7 days should not be on STANDARD_IA.

Verification

You should now be able to answer:

• What four terms feed into the bucket-size formula, and which one changes most under workload?
• Why is avg_over_time a better input than the live rate for a steady-state forecast?
• What is the difference between measuring ingest at the distributor and at the ingester?
• Why does the forecast break if the compression ratio is assumed constant?
• What is the failure shape when the per-tenant rate limit is set against the average instead of the peak?

Quiz