Log Volume from Logs

A team ships a debug log statement that runs on every request, inside a hot path. The line is 1 KiB. The service handles 200 requests per second per pod. The cluster has 12 pods. The volume: 2.4 MiB/s of new log traffic from one service alone, 24/7. The object-store bill arrives two weeks later. The ingester is rejecting chunks at peak because the configured ingestion_rate_mb cap is 16 MiB/s for the entire tenant, and this one service now occupies a third of it. Nobody noticed, because no alert was watching the rate.

A log-volume alert is the simplest production discipline Loki needs and the one most often missing. The shape is small: a bytes-per-second metric per service, a comparison against a budget, and a page that fires before the ingester starts shedding load.

What it is

A log volume metric is a time series whose samples are the volume of log data emitted by a service, computed either as bytes per second or as lines per second. The Loki 3.x query form uses rate() against a window, applied to either the raw line stream or to a chunk-size stream produced by the platform itself.

The query is a metrics query — same shape as the error-rate query, but with a different aggregation:

# Bytes per second over the last 5 minutes, per instance.
sum(rate({job="checkout"} | json [5m])) by (instance)
  * 1024 * 1024

The bytes per second is approximate when computed from the raw line stream (it under-counts by the chunk-encoding overhead). For an exact number, prefer the Loki self-metric loki_ingester_bytes_received_total, which is the platform’s authoritative count.

Why a sysadmin cares

Log volume is the single most expensive dimension of Loki cost. A doubling of log volume is a doubling of:

• Ingest bandwidth between distributor and ingester.
• Chunk-store writes to the object store.
• Index entries (one per stream label combination).
• Compactor work (chunk size, label combinations to merge).
• Query cost (every query that matches the noisy stream pays the parse cost).

A service that grows from 10 KiB/s to 1 MiB/s of logs is a service that has broken the platform budget, even if every other service is healthy. The platform’s first defence is the configured per-tenant ingestion cap (ingestion_rate_mb, ingestion_burst_size_mb); the platform’s second defence is the alert that fires when application volume approaches the cap before the cap is hit.

How it works

The pipeline is the same shape as the error-rate pipeline, but without the level filter. The query selects the stream, applies the rate window, and aggregates.

   log stream             Loki query engine            result
   ----------             -----------------            ------
 every line          --->  rate() over window     --->  bytes/s or lines/s
                       by (instance, job)

Two refinements for production accuracy:

• Bytes vs lines. A line is a discrete event; a byte is the on-the-wire cost. The two metrics correlate, but the ratio varies by log format. A JSON-heavy service averages 500 bytes per line; a syslog-heavy service averages 200 bytes per line. Alert on bytes for cost; alert on lines for behaviour (a service that emits 10x more lines at the same byte volume is emitting shorter lines, which is usually a regression).
• Index entries per label. Loki indexes the unique label combinations; a stream with one extra label is an extra entry in the inverted index for every distinct value. Volume multiplied by label cardinality is the actual platform cost; volume alone is the user-visible symptom.

How to configure it

The recording rule that produces the metric:

# /etc/loki/rules/prod-eu/volume.yaml
groups:
  - name: log_volume
    interval: 1m
    rules:
      # Bytes per second per instance.
      # The constant 1024^2 converts from MiB-rate to bytes/s
      # because Loki reports rate() in bytes-per-second by default
      # when the underlying counter is bytes; the multiplication
      # below is the safety belt for a log-derived metric where
      # the unit is events.
      - record: app:log_bytes:rate5m
        expr: |
          sum(rate({job=~"checkout|api|worker", cluster="prod-eu"} [5m])) by (instance, job)

      # Lines per second per instance (behavioural signal).
      - record: app:log_lines:rate5m
        expr: |
          sum(rate({job=~"checkout|api|worker", cluster="prod-eu"} [5m])) by (instance, job)

      # Tenant-level total — for the capacity alert.
      - record: app:log_bytes:rate5m:tenant
        expr: |
          sum(rate({cluster="prod-eu"} [5m]))

The capacity-driven alert that warns before the ingester cap:

# /etc/loki/rules/prod-eu/volume_alerts.yaml
groups:
  - name: log_volume_alerts
    interval: 1m
    rules:
      # Page when tenant volume is over 75% of the configured cap.
      - alert: LokiIngestionApproachingLimit
        expr: app_log_bytes_rate5m_tenant / 16777216 > 0.75
        for: 5m
        labels:
          severity: warning
          team: observability
        annotations:
          summary: 'Loki ingestion at {{ $value | humanizePercentage }} of cap'
          description: |
            Tenant prod-eu is consuming {{ $value | humanizePercentage }}
            of the configured 16 MiB/s ingestion cap. Investigate
            before the ingester starts rejecting chunks.

      # Page when any service doubles its volume in 10 minutes.
      - alert: LogVolumeSpike
        expr: |
          app_log_bytes_rate5m
            >
          (app_log_bytes_rate5m offset 10m) * 2
        for: 5m
        labels:
          severity: warning
          team: observability
        annotations:
          summary: 'Log volume doubled for {{ $labels.job }}/{{ $labels.instance }}'

A few notes on the shape:

• for: 5m smooths the alert over five minutes; a one-minute spike does not page. The window should match the rate window used for the metric.
• The capacity alert uses / 16777216 because the configured cap is 16 MiB/s = 16,777,216 bytes/s. The division makes the alert threshold a clean percentage.
• The offset comparison catches regressions that double volume without a corresponding business change. A scaling event that doubles the instance count should double volume in step; the offset comparison fires only when the ratio per-instance has changed.

Validate before applying:

# READ-ONLY: confirm the volume query returns a metric.
curl -s -u "$LOKI_USER:$LOKI_PASS" \
  --data-urlencode 'query=sum(rate({job="checkout", cluster="prod-eu"} [5m])) by (instance)' \
  --data-urlencode 'start=2026-08-13T14:00:00Z' \
  --data-urlencode 'end=2026-08-13T14:05:00Z' \
  --data-urlencode 'step=60s' \
  -G http://loki.internal:3100/loki/api/v1/query_range \
  | jq '.data.result[] | {instance: .metric.instance, bytes_s: .values[-1][1]}'
# {"instance":"checkout-7f9c","bytes_s":"12450.32"}
# {"instance":"checkout-8k3d","bytes_s":"11890.71"}

# READ-ONLY: confirm the tenant-level series.
curl -s -u "$LOKI_USER:$LOKI_PASS" \
  --data-urlencode 'query=sum(rate({cluster="prod-eu"} [5m]))' \
  -G http://loki.internal:3100/loki/api/v1/query \
  | jq '.data.result[0].value[1]'
# "3845021.12"  (about 3.7 MiB/s; well below the 16 MiB/s cap)

How to validate it

Three signals confirm the metric is live and useful:

# READ-ONLY: the produced metric has the expected per-instance cardinality.
curl -s -u "$MIMIR_USER:$MIMIR_PASS" \
  --data-urlencode 'query=count(app_log_bytes_rate5m)' \
  -G http://mimir.internal:9009/prometheus/api/v1/query \
  | jq '.data.result[0].value[1]'
# "6"

# READ-ONLY: the metric tracks the platform's authoritative ingestion metric.
# Compare app_log_bytes_rate5m:tenant to loki_ingester_bytes_received_total
# over the same window. A consistent ratio (within ~5%) confirms the
# derivation is right.
curl -s http://loki.internal:3100/metrics \
  | grep '^loki_ingester_bytes_received_total{tenant="prod-eu"}' \
  | head

# READ-ONLY: the alert is loaded and evaluates.
curl -s -u "$LOKI_USER:$LOKI_PASS" \
  http://loki.internal:3100/loki/api/v1/rules \
  | jq '.data.groups[] | select(.name=="log_volume_alerts") | .rules[] | {name, lastEvaluation, state}'

A working volume metric tracks the platform’s authoritative ingestion count within a small margin (the gap is the chunk-encoding overhead that the log-derived metric does not see). A growing gap means the metric is missing streams — the selector is too narrow.

How it can fail

1. The selector is too narrow. A rule whose selector is {job="checkout"} misses every other job in the tenant; the tenant-level total is wrong. Symptom: the capacity alert fires late because the tenant metric under-counts by 90%.
2. Volume grows faster than the rate window can average. A log-spike that lasts 30 seconds is invisible to a 5-minute rate window; the ingester cap fires before the alert does. Symptom: chunks rejected (loki_distributor_dropped_lines_ total), but the volume metric looks healthy.
3. The platform is shedding load. When the ingester cap is hit, lines are dropped before they reach the storage layer. The log-derived metric measures what was stored, not what was sent. Symptom: the metric says 14 MiB/s; the platform actually received 22 MiB/s; the gap is the dropped traffic.
4. Cardinality grows with a new label. A service adds a trace_id stream label; the chunk-store size doubles silently. Symptom: loki_ingester_streams_created_total climbs; the byte metric is unchanged; the storage cost doubles.
5. The rate window is too long for low-volume services. A service at 1 KiB/s averaged over 5 minutes is 300 KiB over the window; the metric floats at the bottom of the resolution. Symptom: the metric shows “0” most of the time and spikes only when traffic actually moves.

How to troubleshoot it

The diagnostic order: is the rule loaded, does the metric track the platform’s authoritative count, is the label set bounded, is the cap being hit.

1. Rule loaded? GET /loki/api/v1/rules. Check the rule group lastEvaluation and confirm the query parses.
2. Metric tracks platform count? Plot app_log_bytes_rate5m:tenant against loki_ingester_bytes_received_total over the same window. A persistent gap means the selector misses streams.
3. Label set bounded? Run count by (__name__) (\{cluster="prod-eu"\}) in Loki; the per-stream count should be in the low thousands, not in the millions. A high count means a new label has been added.
4. Cap being hit? loki_distributor_dropped_lines_total and loki_distributor_dropped_bytes_total rise when the cap is exceeded. The capacity alert should have fired first; if it did not, the rule’s denominator is wrong.
5. Service attribution. When volume grows, attribute to a service with sum(rate({cluster="prod-eu"} [5m])) by (job) over a known window. The job with the steepest slope is the offender.
6. Inspect distributor logs. The distributor logs every 429 rejection with the tenant and the rate at the time; this is the ground truth for what Loki actually saw.

Security implications

• An attacker can drive log volume deliberately. A scripted client sending well-formed JSON logs to the configured agent can inflate volume until the cap is hit, starving legitimate traffic. The fix is rate-limiting at the agent and at the distributor; the volume alert catches the resulting service degradation.
• The metric inherits the secrets in the logs. A volume metric is a count, not a content metric, but the alert annotations may include sample log lines. Audit the alert template to ensure the rendered annotations do not leak.
• The capacity alert is a denial-of-service vector. A misconfigured alert that pages the on-call every five minutes trains them to ignore it. Set thresholds that page only on signal, not on noise.

Performance implications

• Ruler CPU scales with rule cost. A rule whose selector matches the entire tenant opens every stream. A 10,000-stream tenant is a 10,000-stream query; budget the rule accordingly.
• Object-store writes scale with chunk volume. Every MiB/s is a MiB/s of writes plus the index entries. A 100 MiB/s service is a meaningful line item on the storage bill.
• Compactor work scales with stream count. The compactor merges chunks with the same label set; more streams means more merges per cycle.
• The capacity alert is cheap. Two aggregations, one comparison, one threshold. The alert is bounded by the rule cost, not by the volume it monitors.

Production guidance

• Alert on tenant-level volume as a percentage of the configured cap. The page should fire before the ingester starts rejecting, with enough headroom for investigation.
• Alert on per-service volume relative to a known baseline. A doubling over 10 minutes is a regression; a halving is a silence that may be worse.
• Track loki_ingester_streams_created_total weekly. A rising rate under steady traffic is the leading indicator of a cardinality problem the byte metric cannot see.
• Size the cap with headroom for the loudest expected event. A Friday afternoon scaling event should not push the tenant into cap rejection.
• Separate “ingestion approaching limit” (warning, page at 75%) from “ingestion at limit” (critical, page at 95%). Different severities, different on-call rotations, different runbooks.

Verification

You should now be able to answer:

• What is the difference between bytes-per-second and lines-per-second as a volume signal, and which one catches behavioural regressions?
• Why does label cardinality cost more than byte volume per unit, and which Loki self-metric catches a cardinality regression early?
• Which alert fires before the ingester cap is hit, and what threshold should it use?
• What is the failure shape when the selector misses streams, and how does it show up on a dashboard?
• Which self-metric distinguishes what the platform received from what the log-derived metric counted?

Quiz