Loki Capacity Incident

A Loki cluster has been running at 60 percent of its chunk-store budget for months. The budget is sized for a 30-day retention with a one-week margin. A new tenant is onboarded without a rate-limit conversation. Within a week, the chunk store is at 95 percent. The compactor is evicting old chunks faster than the rate policy intends; queries against the eviction window return incomplete results. A query for a customer report from eleven days ago returns nothing, because the chunks that held those eleven-day-old logs were evicted to make room for the new tenant is output.

This is the capacity incident. It is the most expensive of the spike outcomes because the cost is paid in lost data, and the loss is silent.

What it is

A Loki capacity incident is any condition in which one of the four finite resources of a Loki cluster is approaching or has reached its limit. The four resources are:

• Chunk-store capacity. The object store bucket that holds compressed chunk files. The limit is the bucket’s size or the budget for it.
• Index capacity. The index files (boltdb-shipper files or TSDB index files) that map label sets to chunk IDs. The limit is the disk that holds the index or the rate at which the compactor can keep up.
• Ingester memory. The in-memory map of active streams. The limit is the pod’s memory budget.
• Query concurrency. The number of in-flight queries the queriers and query-frontends can handle. The limit is the CPU and the configured concurrency.

The four are coupled but distinct. The chunk store can be full while the index has plenty of room (a high-cardinality tenant that writes few lines); the index can be saturated while the chunk store has space (a low-cardinality tenant that writes many short lines); the ingester memory can be exhausted while both stores have room (a debug flood before the chunks flush).

Why a sysadmin cares

The capacity incident is the only spike outcome that loses data. A debug flood loses money; a log loop loses money; a new service loses money. A capacity incident loses logs. The loss is paid by the next investigation that needs the evicted data.

Three operational consequences recur:

• Silent eviction. The compactor evicts old chunks to make room for new ones. The eviction is by design; the rate is configured; the consequence is that a query for a time window that overlaps the eviction returns an incomplete result. The query does not fail; it returns less data than it should.
• Cascading rejections. When the chunk store is full, the ingesters cannot flush new chunks; the head block grows; the pod memory climbs; the ingester rejects new writes at the distributor. The rejection is shared by every tenant.
• Compactor catch-up cost. Once the capacity pressure eases, the compactor has to rebuild the index entries for the evicted chunks. The catch-up work competes with the live compaction workload and slows queries for hours after the incident.

How it works

The mechanism is a mismatch between the rate of arrival and the rate of eviction. The chunk store has a fixed budget; the ingest rate sets the rate of arrival; the retention policy sets the rate of eviction. When arrival exceeds eviction, the bucket fills.

                  Chunk store budget
                          |
            +-------------+-------------+
            |                           |
       Rate of arrival             Rate of eviction
       (ingest rate)               (retention policy)
            |                           |
            v                           v
       bytes in / second          bytes out / second
            |                           |
            +-----> compare <-----------+
                          |
                  arrival > eviction?
                          |
                  +-------+-------+
                  |               |
                 yes              no
                  |               |
            bucket fills      bucket stable
                  |
                  v
            compactor evicts
            oldest first
                  |
                  v
            queries against the
            eviction window return
            incomplete data

The three knobs that change the comparison are the arrival rate (raise the per-tenant cap to increase arrival), the eviction rate (shorten the retention to increase eviction), and the budget (grow the bucket to widen the gap). The platform team controls the budget and the eviction rate; the tenants control the arrival rate; the negotiation is the work.

How to configure it

The configuration has three layers: the chunk store, the compactor, and the alerts. Each layer has a role.

The chunk store is sized from the steady-state ingest rate and the retention window. The math is mechanical.

# /etc/loki/config.yaml (Loki 3.x)
common:
  storage:
    s3:
      # The chunk store. The bucket size is sized at 1.5x the
      # expected steady-state volume. The 1.5x margin absorbs
      # the compactor catch-up and any legitimate spike.
      bucketnames: "loki-chunks-prod"
      region: "eu-west-1"

schema_config:
  configs:
    - from: "2024-01-01"
      store: "tsdb"
      object_store: "s3"
      schema: "v13"
      index:
        prefix: "index_"
        period: "24h"

storage_config:
  tsdb_shipper:
    active_index_directory: "/loki/tsdb-index"
    cache_location: "/loki/tsdb-cache"

compactor:
  # The compactor is what reconciles the chunk store with the
  # retention policy. It must keep up with the ingest rate;
  # otherwise the index drifts from the chunks.
  working_directory: "/loki/compactor"
  compaction_interval: "10m"
  retention_enabled: true
  retention_delete_delay: "2h"
  delete_request_store: "s3"

limits_config:
  # The retention window. Shorter windows evict faster; the
  # trade-off is the look-back window for queries.
  retention_period: 744h

  # Per-tenant retention overrides. A tenant with compliance
  # requirements may have a longer window; a tenant with
  # short-lived debug logs may have a shorter one.
  retention_stream:
    - selector: '{service_name="checkout-svc"}'
      priority: 1
      period: "2160h"

  # Per-tenant stream cap. The ceiling that catches the
  # high-cardinality failure shape before it fills the
  # compactor's queue.
  max_streams_per_user: 10000

The alerts are the early-warning system. Each shape has its own alert; the alerts are tiered by severity.

# /etc/prometheus/rules/loki_capacity.yaml
groups:
  - name: loki_capacity
    rules:
      # Chunk-store exhaustion. Paging at 80 percent; warning
      # at 70 percent.
      - alert: LokiChunkStoreWarning
        expr: |
          (
            loki_bucket_stored_chunks_total
              / on() loki_bucket_max_size_bytes
          ) > 0.7
        for: 30m
        labels:
          severity: warning

      - alert: LokiChunkStoreCritical
        expr: |
          (
            loki_bucket_stored_chunks_total
              / on() loki_bucket_max_size_bytes
          ) > 0.9
        for: 5m
        labels:
          severity: critical

      # Index exhaustion. The compactor is falling behind.
      - alert: LokiCompactorFallingBehind
        expr: |
          rate(loki_tsdb_compaction_writes_total[1h])
            < rate(loki_tsdb_index_writes_total[1h]) * 0.5
        for: 30m
        labels:
          severity: warning

      # Ingester memory. Sustained head-block growth.
      - alert: LokiIngesterMemoryHigh
        expr: |
          loki_ingester_memory_chunks
            > 0.8 * loki_ingester_memory_limit
        for: 10m
        labels:
          severity: warning

      # Time-to-full. The most useful alert for the platform
      # team; it converts the current rate and the free space
      # into a number of hours.
      - alert: LokiChunkStoreTimeToFull
        expr: |
          (
            (1 - (
              loki_bucket_stored_chunks_total
                / on() loki_bucket_max_size_bytes
            ))
            * loki_bucket_max_size_bytes
          ) / (1024 * 1024 * rate(loki_distributor_bytes_received_total[1h]) * 3600)
          < 24
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: 'Chunk store fills in {{ $value }} hours'

How to validate it

The validation is three queries. The first confirms the bucket fill level; the second confirms the compactor is keeping up; the third confirms the time-to-full estimate.

# 1. Bucket fill level. The first thing to check.
# Severity: READ-ONLY
aws s3api list-objects-v2 \
  --bucket loki-chunks-prod \
  --output json \
  --query 'sum(Contents[].Size)' \
  | numfmt --to=iec

Expected: well below 80 percent at steady state. A reading above 80 percent means the bucket is in the warning band; above 90 percent means the bucket is in the critical band.

# 2. Compactor lag. The compactor should be keeping up with
# the index writes.
# Severity: READ-ONLY
curl -s 'http://loki-compactor:3100/metrics' \
  | grep -E '^loki_(tsdb_compaction_writes_total|tsdb_index_writes_total)' \
  | awk '{print $1, $2}'

Expected: the compaction rate is close to the index write rate. A gap means the compactor is falling behind; the index drift is growing.

# 3. Time-to-full. The most useful number for triage.
# Severity: READ-ONLY
logcli query --since=1h \
  '(
    (1 - (
      sum(rate(loki_distributor_bytes_received_total[24h]))
        / on() (1024 * 1024 * 1024 * 1024)
    ))
    / (sum(rate(loki_distributor_bytes_received_total[1h])))
  )'

Expected: a number of hours at least three times the retention window. A number below the retention window means the bucket will fill before the oldest legitimate logs are evicted.

How it can fail

Five failure shapes recur at the capacity incident.

1. The unannounced new tenant. A service is deployed without a rate-limit conversation. The first indicator is the bucket fill rate rising without a corresponding rise in any known tenant.
2. The cardinality explosion. A label bug raises the stream count ten times. The first indicator is the index file size growing faster than the chunk size.
3. The retention drift. The retention window was shortened in a config change but the bucket was not grown correspondingly. The first indicator is the time-to-full dropping below the retention window.
4. The compactor outage. The compactor crashed or was scaled down for cost reasons. The first indicator is the index drift growing.
5. The legitimate growth. A new product launch or a seasonal peak. The first indicator is the bucket fill rate correlating with a calendar event.

How to troubleshoot it

1. Confirm the fill level (query 1 above)
        |
        v
2. Identify the dominant tenant (the per-tenant byte ranking)
        |
        v
3. Identify the dominant shape:
        |
        +----> chunk store? -> shorten retention or drop tenant
        |
        +----> index? -> fix cardinality, scale compactor
        |
        +----> ingester? -> fix the streams-not-closing shape
        |
        +----> query concurrency? -> scale queriers
        |
        v
4. Apply the response that matches the shape
        |
        v
5. Validate (queries 1-3 again)
        |
        v
6. Capture the data for the post-incident cost review

Security implications

The capacity incident is not a security event by default, but the eviction shape has a compliance dimension. A retention shortening that evicts logs that a compliance window required is a data incident: the logs are gone, the audit trail has a gap, and the legal owner of the data must be notified. The fix is to never shorten retention without checking the compliance windows; the discipline is to maintain a per-tenant retention table that maps every tenant to its compliance requirements.

Performance implications

The performance cost of a capacity incident is paid in two places. The chunk store pays in write and read IOPS that saturate the bucket’s throughput. The compactor pays in CPU and disk that fall behind the live workload. Both costs are visible in the alerts above; both are bounded by the response that matches the shape.

Verification

You should now be able to answer:

• What are the four canonical shapes of a Loki capacity incident, and which metric identifies each?
• What is the correct immediate response for each shape, and why is the response order important?
• How is the time-to-full alert computed, and what is its threshold?
• What is the compliance dimension of a retention shortening, and what is the correct guard rail?

Quiz