Correlation Cost

A team enabled trace_id structured metadata on every log line. Within a month, Loki’s log volume grew 38% (40 bytes per line on average). The Tempo search index doubled (the tail sampler, which used to drop 90% of traces, now keeps 100% so the correlation “always works”). A dashboard panel that pivoted from Loki to Tempo now takes 6 seconds to draw, because the join query is | json | trace_id="..." and the JSON parsing happens against a billion lines per day. The finance team’s quarterly invoice is up by 22%. The team’s response is the lesson: the cost of correlation has four vectors and three failure shapes, and the right discipline keeps both bounded.

The cost is paid in three ways: bytes shipped, queries run, and search index rows stored. The bytes are the obvious one; the queries are the hidden one. The team’s instinct is to enable every correlation and let the storage grow; the operational answer is to size the knobs deliberately and review the cost on the same cadence as the alert budget.

What it is

Correlation cost is the aggregate cost of the four platform choices that together produce the pivot. It is not a single line item on an invoice; it is the sum of:

• Loki ingest bytes. The cost of adding a trace_id field to every log line as structured metadata. Bounded by bytes per line.
• Loki label cardinality. The cost of promoting a resource attribute to a Loki stream label. Bounded by the number of distinct label values across the fleet.
• Tempo trace block storage. The cost of storing the full trace data. Bounded by the number of traces kept and the size of each trace.
• Tempo search index storage. The cost of the search-index sample the operator uses for “find me traces where service.name=X and status=error”. Bounded by the sampling rate and the number of indexed attributes.

A fifth cost vector is on the query path: the time a pivot query (LogQL to filter by trace_id, TraceQL to find by attribute) takes to complete. This is a different shape than the storage cost — it does not appear on invoices — but it is the one that wakes the on-call engineer at 03:00.

Why a sysadmin cares

Two operational pressures depend on the cost being managed:

• Quarterly storage budget. Loki and Tempo storage are the largest operating expense in a small platform and one of the largest in a mid-sized platform. An ungoverned correlation posture pushes the storage bill out by 30% within a quarter; the cost shows up on a finance dashboard the engineering team owns.
• Investigation latency. A pivot that returns in 50 ms is invisible; a pivot that returns in 6 seconds is the reason the on-call engineer keeps a separate browser tab open to Loki and Tempo and copy-pastes between them. The team’s MTTR rises with the slow pivot; the cost of the slow pivot is recomputed in incident time, not in dollars.

The trade-off is real. A trace_id on every line is the single biggest raise in Loki ingest bytes per line; a search index with 100% sampling doubles Tempo’s storage. The right balance for a busy fleet is somewhere between “no correlation” and “every correlation everywhere”. The numbers below are the starting points.

How it works

The four knobs are independent. Each has a default that is defensible; each has a tuning shape:

+-----------------------------------------------+
| Loki ingest bytes                            |
|   per-line overhead = 40 bytes (trace+span)  |
|   100 GB/day of logs = ~5 GB/day overhead     |
|   knob: skip the field if not in a span       |
+-----------------------------------------------+
| Loki label cardinality                       |
|   service_name + deployment_environment      |
|   = 100 services x 3 envs = 300 streams      |
|   knob: bound the number of stream labels     |
+-----------------------------------------------+
| Tempo trace blocks                           |
|   average span payload ~2 KB, 10 spans/trace |
|   1M traces/day x 20 KB = 20 GB/day          |
|   knob: tail-sampling (10% sample rate)      |
+-----------------------------------------------+
| Tempo search index                           |
|   every sampled span has index rows          |
|   sample rate 10% = 10% of trace block size  |
|   knob: separate sampling for search index   |
+-----------------------------------------------+

Three observations:

1. Bytes vs cardinality are different budgets. Adding trace_id as structured metadata costs bytes per line (tens of bytes, not kilobytes). Promoting it to a stream label costs cardinality per line (a row per distinct value per stream in the Loki index). The two are unrelated on the invoice and unrelated on the configuration.
2. Trace blocks vs search index are different budgets in Tempo. Tempo stores trace blocks in object storage (cheap, slow to query by attribute). It also stores a separate search index (faster, more expensive per GB, configurable sample rate). A 100% sample rate to the search index doubles the Tempo storage.
3. The join query cost is on Loki and on Tempo. A pivot that filters Loki by trace_id="..." is a structured metadata lookup; with a proper index, it returns in tens of milliseconds. A pivot that filters Loki with | json | trace_id="..." parses the entire line first and is ten times slower. The choice of log line shape (structured metadata vs free-text) is the biggest query-cost knob on the Loki side.

How to configure it

The four knobs map to four configuration files. They are independent; the team can tune one without touching the others.

The Loki side — structured metadata is the default, label promotion is the exception:

# /etc/loki/config.yaml
# Bounds on the structured metadata the ingester accepts.
limits_config:
  # Maximum number of structured metadata entries per log line.
  # 100 is the default; the team's trace_id + span_id is two.
  structured_metadata_entries_count: 100

  # Per-stream label cardinality budget.
  # Loki rejects new streams when cardinality explodes.
  max_label_name_length: 1024
  max_label_value_length: 4096

# Retention at the chunk store. Configure S3 lifecycle for
# expiry.
compactor:
  working_directory: /tmp/loki-compactor
  retention_enabled: true
  retention_delete_delay_store: "tsdb"
  retention_delete_delay: 10m
  delete_request_store: "tsdb"

The Tempo side — tail-sampling for the trace blocks, separate sampling for the search index:

# /etc/tempo/tempo.yaml
storage:
  trace:
    backend: s3
    s3:
      bucket: tempo-blocks
      endpoint: s3.internal:9000
    wal:
      path: /var/tempo/wal
  # Block retention: delete trace blocks after 14 days.
  trace:
    max_block_duration: 24h

# Tail-based sampling at the collector: 10% of traces get
# stored; the rest are dropped at the collector. Slow traces
# get a higher sample rate.
metrics_generator:
  registry:
    external_labels:
      region: prod-eu
  storage:
    path: /var/tempo/generator

# Trace search index: independent sampling rate. A 25% index
# sample rate provides "find me recent traces for service X
# with status=error" within five minutes, at 1/4 the storage
# of the trace blocks.
search:
  max_duration: 0      # 0 = no retention cap on the index;
                      # override with --override-search-max-duration

The Alloy / OTel collector side — drop the trace_id field if the line is not part of a span (the field is empty in the log record and adds zero bytes — but the upstream work to parse and emit it cost something):

otelcol.processor.transform "traceidcost" {
  trace_statements {
    context = "span"
    statements = [
      # Drop spans that have no parent and no useful attributes.
      # Saves a trace block write per dropped span.
      `delete_span() where span.kind == SPAN_KIND_INTERNAL and attributes["useful"] == nil`,
    ]
  }
  output { traces = [otelcol.exporter.otlp.tempo.input] }
}

The Grafana side — pin the query timeouts so a slow pivot fails loud instead of hanging the dashboard:

# /etc/grafana/provisioning/datasources/loki.yml
jsonData:
  # Timeouts at the Loki data source layer.
  timeout: 30           # seconds before Grafana gives up
  queryTimeout: 60      # seconds for the underlying Loki query
  # Max lines returned per query — the join query cost grows
  # with this number; 1000 is the default; 500 for hot panels.
  maxLines: 500

How to validate it

# READ-ONLY: confirm the Loki chunk store size and the
# ingest rate.
curl -fsS -u "$LOKI_USER:$LOKI_PASS" \
  'http://loki-prod-eu.internal:3100/metrics' \
  | grep -E '^loki_ingester_(bytes_received|chunks_created)_total'
# loki_ingester_bytes_received_total   ...   1.1e+11
# (Use this to compute the structured metadata overhead as
# a percentage of total ingest bytes.)

# READ-ONLY: confirm the Tempo trace blocks size and the
# search index sample rate.
curl -fsS -u "$TEMPO_USER:$TEMPO_PASS" \
  'http://tempo-prod-eu.internal:3200/metrics' \
  | grep -E '^tempo_(ingester_traces_created|search_index|sample)_(total|bytes)'
# (Tempo exposes per-component sample rates as gauges.)
# (Compare tempo_ingester_traces_created_total against
# tempo_metrics_generator_registry_active to confirm the
# search index sample rate matches the configured value.)

# READ-ONLY: confirm the join query latency on the Loki side.
# A healthy pivot returns in < 50 ms.
TRACE=4bf92f3577b34da6a3ce929d0e0e4736
time logcli query --since 1h \
  '{service_name="checkout-api"} | trace_id="'"$TRACE"'"'
# 0.045s
# (A pivot that takes 6 seconds is doing too much JSON
# parsing; the structure of the line is the issue.)

# READ-ONLY: confirm the cost as a Grafana dashboard.
curl -fsS -u "$GRAFANA_ADMIN" \
  'http://grafana.internal:3000/api/datasources/uid/prom-prod-eu' \
  > /dev/null
# (The metric 'loki_ingester_bytes_received_total /
# rate(loki_ingester_bytes_received_total[1h])' is the daily
# ingest rate; the correlation cost is the structured
# metadata bytes per line x lines per day.)

# CONFIGURATION: roll the change out with a budget cap.
# The OTel collector's 'memory_limiter' processor and the
# Tempo ingester's queue depth limit are the operational
# guardrails.

If step 2 shows tempo_metrics_generator_registry_active is counting every trace, the search index sampling is disabled — the metrics index is at full sample and the cost is double the trace block storage. If step 3 returns in 6 seconds, the Loki line format is structured-but-re-parsed (the stage.structured_metadata block is missing in Alloy), and the join query is parsing every line.

How it can fail

1. Unbounded Loki label cardinality. A team promotes trace_id to a Loki stream label “to make joins faster”. Loki’s index grows by one row per request. Within hours, the ingester rejects new streams and the platform appears down. Symptom: Loki metric loki_ingester_streams_created_total grows linearly with request rate; loki_ingester_streams_limit trips.
2. Search index at 100% sampling. A team turns the Tempo search index sampling to 100% so “everything is searchable”. The search index storage equals the trace block storage; the Tempo bill doubles. Symptom: tempo_search_index_storage_bytes is roughly equal to tempo_ingester_bytes_received_total.
3. Tail sampling disabled to “fix” lost correlation. A team disables the collector’s tail sampler so every trace is stored. Trace block storage grows tenfold; the search index with 25% sampling grows tenfold too because the index follows the trace volume.
4. JSON-parsed log lines for join queries. A team re-parses every line on the Loki side with | json to extract the trace_id field, instead of promoting the field to structured metadata. The JSON parser runs per line; the join query takes seconds. Symptom: a pivot panel that took 50 ms now takes 6 s; the dashboard timeout fires first, the panel renders blank.
5. Retention disabled on the trace block store. A team wants every trace “forever”. Object storage grows unbounded; the compactor is not enabled to delete blocks. Symptom: tempo_ingester_blocks_flushed_total keeps adding; object-storage billing doubles month on month.
6. Inverted cost: search index at 10% on the wrong sampler. A team configures the search index sampler to 10% but the metrics-generator’s sampling decision differs from the trace-block decision. The search index misses critical traces and the team concludes “the search index is unreliable” and turns it off, losing the search feature entirely.

How to troubleshoot it

The diagnostic order for “the correlation cost is higher than expected”:

1. What is the per-line Loki overhead? Sample a representative log line. Strip it of trace_id and span_id; compare byte counts. A 40-byte overhead per line at 100 GB/day is 4 GB/day on the bill.
2. How many Loki stream labels? curl /api/v1/series?match[]=\{service_name=~".+"\} | jq '. | length'. A number above a few hundred is an explosion in progress.
3. What is the Tempo trace sample rate? Inspect the collector’s tail-sampler configuration; count the sampling rules and the rule’s policy.late.
4. What is the Tempo search index sample rate? Same procedure; the metrics-generator’s sample-rate is the dial.
5. What is the join query latency? Run the join query from logcli and time it. A latency above 1 s is a structured-metadata promotion problem in Alloy.

Security implications

Correlation surfaces a read-amplification risk. An attacker with read access to one stream can pivot to other streams by trace_id. The same Loki that stores public service logs and the same Tempo that stores public trace spans may, via correlation, expose internal services that were otherwise isolated. The risk is the join, not the streams individually.

Mitigate by reviewing the role assignment on Loki and Tempo together. A reader of service=A should not, implicitly, become a reader of service=B via a pivot. The two roles can be combined, but the combined role must be explicit.

The trace_id itself is not sensitive. The cost surface is the join and the data the join reveals.

Performance implications

The performance implications of correlation cost are:

• Loki chunk store growth. ~ 5% overhead per line for trace_id + span_id. On 100 GB/day of logs, roughly 5 GB of additional chunk storage per day.
• Tempo block store. Ten-fold growth if tail sampling is disabled. The default 10% sample rate is the common starting point.
• Tempo search index. The index size is bounded by the sampling rate times the trace volume. The default 10% / 25% is conservative; a 100% rate doubles Tempo storage for a 100% sample.
• Join query latency. Bounded by Loki’s structured metadata query path. Sub-100ms is the target; the second half is bounded by Tempo’s /api/traces/<id> query.
• Browser cost of derived fields. Covered in lesson 04.

The cost discipline is per-knob, not aggregate. The team reviews each knob on its own monthly cadence and tunes when the budget drifts.

Production guidance

• Loki ingest overhead. 40 bytes per line is the budget; anything above 80 bytes suggests double-stamping or unrelated metadata on the line.
• Loki label cardinality. service_name and deployment_environment are the bounded ones; never promote a per-request value to a stream label. Review the label list quarterly.
• Tempo tail sampling. 10% is the starting sample rate. Tune to the budget; raise the rate for traces the team considers “always keep” (sampled errors, latency outliers).
• Tempo search index. 25% of the tail-sample rate is the starting point; the storage cost is bound to the index sample, not the trace block sample.
• Retention. Configure object storage lifecycle to expire trace blocks after the retention window (14 days is common; longer if compliance requires it). The compactor in Loki handles log retention; the S3 lifecycle handles trace retention.

Verification

You should now be able to answer:

• What are the four cost vectors of a correlation posture?
• Why is “promote trace_id to a stream label” almost always wrong?
• What is the difference between Tempo’s trace blocks and its search index, and why is the latter a separate configuration?
• What is the correct fix for a Loki join query that takes 6 seconds?
• What is the recommended starting sample rate for the Tempo tail sampler and the search index?

Quiz