Log Cost Drivers

The log line on last month’s invoice was 40 percent larger. The S3 bill, separately, had grown 28 percent over two quarters. Nobody on the platform team could say which service, which stream, or which change in a single logging library produced the growth. A week of investigation later the answer was a single line — a JSON blob serialised into every INFO row by a single package.

What log cost is

Loki cost has three additive sub-costs, tied to the log lifecycle:

• Bytes per line. The single most unpredictable part. A short single-field line (status=200) is 16 to 30 bytes after compression. A line with a 4 KiB JSON payload is 4 KiB after compression. The line length distribution of a service can change overnight when a logger is upgraded.
• Lines per second. The volume of new lines arriving at the distributor per tenant per service. Spikes after a deployment can be five times the prior baseline.
• Retention. How long the platform keeps each chunk. Loki splits retention into hot (ingester + boltdb shipper) and cold (long-term store). Hot is roughly ten times more expensive per byte than cold.

The model that ties them together:

daily_ingest_bytes   = bytes_per_line * lines_per_second * 86400
hot_storage_bytes    = ingest_bytes_per_second * hot_retention_seconds
cold_storage_bytes   = ingest_bytes_per_second * cold_retention_seconds
monthly_storage_cost = hot_bytes  * hot_cost_per_byte
                      + cold_bytes * cold_cost_per_byte
                      + query_compute
                      + ingest_compute

Loki is unusual among observability backends because the index itself is small (label hashes per chunk) compared with the chunks. Per-line indexing cost is in the index, per-line storage cost is in the chunk. Both grow with line count, but cost-wise the chunk dominates by an order of magnitude.

Why a sysadmin cares

Three operational pains recur.

1. The post-deploy volume spike. A new release ships with loglevel: debug. For 48 hours the ingest jumps from 50 MB/s to 250 MB/s. The S3 line of the bill catches up at month end; alertmanager latencies rise; the ingester hits its chunk flush limit and starts dropping writes with errors.
2. The single-stream saturation. A misconfigured nginx access log ships every byte to Loki. Other tenants lose headroom; the per-tenant rate limiter kicks in.
3. The undeclared retention. A team changed compactor.retention_period from 7 days to 30 days for a single tenant to chase a post-mortem. The retention stayed at 30 days three months later. Cold storage tripled; nobody noticed until the bill.

Each of these has the same shape: a single lever, on a single tenant, slid past its budget, and the cost surfaced in the month-end invoice, not in the platform.

How the log cost model works

The mental model is a stack of stages, each of which is a cost boundary. The same line can be dropped cheaply at the collector, or dropped expensively at the distributor, or not dropped at all but kept for 90 days.

   service stdout / file         pipeline stages
       |                         (drop, label, limit, parse)
       v
   agent (Alloy / Promtail)      --->  cost boundary 0 (cheap)
       |
       v
   distributor                   --->  cost boundary 1 (rate limit)
       |
       v
   ingester (3x replica)         --->  cost boundary 2 (hot memory)
       |
       v
   boltdb shipper / chunk flush  --->  cost boundary 3 (warm SSD)
       |
       v
   long-term store (S3 / GCS)    --->  cost boundary 4 (cold)
       |
       v
   querier / query-frontend     --->  cost boundary 5 (query-side compute)
       |
       v
   retention compactor           --->  cost boundary 6 (delete)

A cost increase on any stage requires a different control:

• Boundary 0 (collector drop) is the cheapest and the earliest. Drop a noisy debug field there and the data is never written.
• Boundary 1 (distributor rate limit) caps a runaway tenant without dropping the rest.
• Boundary 2 (ingester memory) is the only one with a hard RAM ceiling. Hit it and writes return errors.
• Boundary 3 (warm SSD) is what makes 7-day hot retention affordable. Watch the chunk flush ratio.
• Boundary 4 (cold object store) is where 30 to 90 day retention lives. The compactor deletes by retention_period per stream.
• Boundary 6 (retention) is the deletion lever. Lowering it deletes the corresponding chunks on the next compaction pass.

How to control log cost

The right configuration combines three controls: pipeline drop at the collector, per-tenant rate limit at the distributor, and retention overrides per stream at the compactor.

# File: /etc/loki/config.yaml
# Severity: CONFIGURATION (SIGHUP for limits_config; restart for schema_config)

auth_enabled: true

common:
  ring:
    kvstore:
      store: consul
      consul:
        host: consul:8500
  replication_factor: 3
  path_prefix: /loki

schema_config:
  configs:
    - from: 2026-01-01
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h

storage_config:
  tsdb_shipper:
    active_query_directory: /loki/tsdb-active
    cache_location: /loki/tsdb-cache
  aws:
    s3: s3://eu-west-1/loki
    bucketnames: loki-prod
    region: eu-west-1
    storageclass: STANDARD_IA          # cheaper cold tier for prod

limits_config:
  # Per-tenant rate limit. enforce_metric_name keeps the
  # `loki_ingester_bytes_received_total` counter under the
  # expected ceiling when a tenant exceeds its budget.
  ingestion_rate_mb: 10                # per tenant
  ingestion_burst_size_mb: 20
  max_entries_limit_per_query: 5000
  reject_old_samples: true
  reject_old_samples_max_age: 168h      # 7 days, drop older samples

# Compactor owns retention. Per-tenant override is here.
compactor:
  working_directory: /loki/compactor
  retention_enabled: true
  retention_delete_batch_size: 300
  delete_request_store: s3

# File: /etc/alloy/config.alloy
# Severity: CONFIGURATION (reload required)
# Drop and rewrite at the earliest possible stage. Cost saved here
# never reappears downstream.

loki.source.file "service_logs" {
  targets    = local.file_match.log_targets
  forward_to = [loki.relabel.drop_noisy.receiver]
}

loki.relabel "drop_noisy" {
  forward_to = [loki.write.local.receiver]

  rule {
    # Drop noisy debug lines before they reach the write path.
    # Each line costs roughly 1 KiB in this service; the rule
    # saves roughly 50 MB/s of ingest at peak.
    source_labels = ["level"]
    regex         = "debug|trace"
    action        = "drop"
  }
  rule {
    # Rewrite noisy free-form labels into bounded ones.
    source_labels = ["__meta_kubernetes_pod_label_app"]
    target_label  = "app"
    action        = "replace"
  }
  rule {
    # Stamp the tenant.
    target_label = "tenant"
    value        = "team_alpha"
  }
}

loki.write "local" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
    tenant_id = "team_alpha"
  }
}

The pattern is consistent: drop the data you do not need at the earliest possible stage. Dropping at the collector saves the distributor, the ingester, the boltdb shipper and the S3 line of the bill all at once.

How to validate log cost

Three queries answer the questions that matter.

# Severity: READ-ONLY
# Per-tenant ingest bytes per second, ranked.
logcli instant-query \
  --addr=http://loki:3100 \
  --query='topk(10, sum by (tenant) (rate(loki_ingester_bytes_received_total[5m])))'

illustrative:
{team_alpha="30000000"} {team_bravo="11000000"} {team_charlie="4000000"}

# Severity: READ-ONLY
# Per-stream bytes per second, top 20.
logcli instant-query \
  --addr=http://loki:3100 \
  --query='topk(20, sum by (job, instance) (rate(loki_ingester_bytes_received_total[5m])))'

illustrative:
{job="api-gateway", instance="10.4.2.18:5000"}="8200000"
{job="auth-svc",   instance="10.4.1.7:5000"} = "4400000"
...

# Severity: READ-ONLY
# Per-line byte distribution from a noisy service. A histogram at
# the p99 above 4 KiB is the smoking gun for verbose logging.
logcli instant-query \
  --addr=http://loki:3100 \
  --query='histogram_quantile(0.99, sum by (le) (rate(logql_ingester_line_bytes_bucket[5m])))'

# Severity: READ-ONLY
# Confirm the compactor is honouring the retention periods.
curl -s 'http://loki:3100/loki/api/v1/status/buildinfo' | jq .

illustrative:
{"version":"3.3.0","revision":"...","branch":"HEAD","buildUser":"...","buildDate":"..."}

A runnable Loki 3.x distcompactor output:

# Severity: READ-ONLY
# Per-stream retention override count.
logcli series --match='{job="api-gateway"}' \
  --addr=http://loki:3100 | jq 'length'

The right validation is “the largest per-stream bytes per second falls inside the budget envelope, and the compactor reports retention_enabled: true with the configured period.”

How it can fail

Six shapes repeat.

1. A logger upgrade. A library change moves from text to a JSON payload that includes every HTTP header. Per-line bytes jump from 200 bytes to 4 KiB. Ingest is roughly unchanged; ingest bytes per second is twenty times larger.
2. A panic-loop service. A bug at the logging boundary loops the same ERROR backtrace. Lines per second is roughly a thousand times normal. The distributor rate limit eventually kicks in; until it does, every other tenant’s lines queue.
3. An undeclared tenant. A new pipeline stamp puts every line under tenant="unknown". The rate limit at the unknown tenant is the platform default (effectively zero). One team sees every line dropped silently.
4. A retention override left behind. A post-mortem changed compactor.retention_period for one stream from 7 days to 90 days. It stayed at 90 days a quarter later. Cold storage of that stream grew by an order of magnitude.
5. A drop-stage typo. regex: 'DEBUG' matched nothing because the actual level is debug. The drop stage was decorative; ingest kept rising.
6. The boltdb shipper out of disk. A disk-fill on the shipper volume prevented flushes. Chunks accumulated in ingester memory until the OOM killed writes. The error surfaced as a spike on loki_ingester_chunks_flushed_total failing.

How to troubleshoot runaway log cost

The diagnostic order is the same as the mental model: from the cheapest control to the most expensive.

Symptom (log ingest bytes jumped 3x)
   |
   +-- Per-tenant bytes: which tenant grew?
   |
   +-- Per-stream bytes: which stream grew?
   |     |
   |     +-- Service grep: which app owns the stream?
   |     +-- Label grep: which label is involved?
   |     +-- Sample line: what does one line look like?
   |           |
   |           +-- Long line?  --->  bytes-per-line fix
   |           +-- Many lines?  --->  lines-per-second fix
   |
   +-- Decide: drop in pipeline, rate limit, or retention?
   |
   +-- Verify: did loki_ingester_bytes_received_total fall?
   +-- Document: cost platform change log
   |
Root cause

The decision is rarely “switch vendor.” It is usually one of three: (a) drop the noisy field at the collector, (b) enforce a per-stream rate limit at the distributor, or (c) lower the retention override for a single stream.

Security implications

A log line can leak anything a service prints. PII, secrets, health data and request bodies end up in Loki whenever an engineer adds a print(req.json) line “for the post-mortem.” The control belongs at the pipeline. A drop rule that removes a field at the Alloy collector is far cheaper than an RBAC layer on the query path; the data is never written, so it cannot be leaked. Treat the log retention as a sensitive retention by default: per-stream retention limits shorten the leakage window.

Authentication on the distributor (auth_enabled: true) is mandatory; an unauthenticated Loki accepts log lines from anywhere and stores them under whichever tenant ID the request claims.

Performance implications

The hot ingester is the performance ceiling. Per-stream memory is the binding constraint; once streams exceed ingester's RAM budget, writes return errors and the rate-limit panel trips. Chunk flush throughput is the second ceiling: the boltdb shipper and the in-process shipper compete for disk write bandwidth. The third ceiling is the compactor, which runs on a low-priority node; under-retention workloads can queue compactions, and the long-term store grows behind the curve.

A log budget is therefore a forecast of stream count times bytes per stream times flush-rate. Tune the platform first, then set the budget ceiling to roughly 70 percent of measured headroom.

Production guidance

• Drop the noisy field at the Alloy pipeline. Drop downstream is always more expensive than drop upstream.
• Set per-tenant ingestion_rate_mb and a per-stream override for the largest tenant. A platform default is not enough.
• Configure retention per stream with a config-managed override; do not let engineers change global retention.
• Periodically re-derive the largest tenant’s ceiling from loki_ingester_bytes_received_total. Budgets derived six months ago are fictional.
• Treat any line that exceeds 4 KiB as a budget problem. A long-line detector (a recording rule on the agent side) is cheaper than waiting for the monthly bill.

Verification

You should now be able to answer:

• What three quantities determine the steady-state Loki storage cost, and which is the most unpredictable in production?
• Why is hot ingester memory roughly ten times more expensive per byte than cold S3 retention, and how does that map onto the chunk lifecycle?
• Where does the cheapest cost control live, and why is a drop in the pipeline cheaper than a rate limit at the distributor?
• What is the right diagnostic order when ingest bytes jump three-fold?
• What three Loki runtime metrics are the right set to watch for cost health?

Quiz