Trace Cost Drivers

The trace ingest line item of the observability bill was three times larger than the metrics line and four times larger than the logs line. The platform team, on its first look, agreed that “we need to sample” — and then sampled every trace at 100 percent, because the default configuration does. Sampling is the largest single lever on trace cost. Not using it is the most common mistake.

What trace cost is

Tempo cost has four additive components:

• Spans per second. The number of individual span records pushed to Tempo per second. A request that emits ten spans at 5 000 requests/second is 50 000 spans/sec from that endpoint alone. Without sampling this is a constant, request-rate multiplier.
• Bytes per span. On-disk size of a span. Dominated by attributes, events and links. A span with two attributes is about 300 bytes; one with fifty is closer to 4 KiB.
• Sampling rate. The fraction of traces kept. The lever that every other lever multiplies against. The default is 100 percent, which is rarely the right answer for production.
• Retention. Hot and cold retention in the Tempo block store. Hot is roughly seven days of fast queries; cold is the cheap long-term tier.

The model:

spans_per_sec_ingested   = spans_per_request * requests_per_sec
                            * sampling_rate
bytes_per_day_ingested    = spans_per_sec_ingested * bytes_per_span
                            * 86400
monthly_storage_bytes     = bytes_per_day_ingested * retention_in_days
monthly_storage_cost      = hot_bytes  * hot_cost_per_byte
                          + cold_bytes * cold_cost_per_byte
                          + block_search_compute
                          + ingester_compute

The line “spans per request multiplied by request rate, multiplied by sampling rate” is the heart of trace cost. Doubling the sampling rate doubles the ingest. Doubling the request rate doubles the ingest. A new field in the trace context that fans the span count by five multiplies ingest by five. The only control between request rate and ingest is sampling.

Why a sysadmin cares

Three failure shapes dominate.

1. The default sampler is 100 percent. A team enables the OpenTelemetry Collector with the default config. Every request yields every span. At 5 000 req/s and 12 spans per request the platform ingests 60 000 spans/sec. Within a week the trace storage tier is the largest single line on the bill.
2. The 12-span fan-out. A new middleware wraps every database call in an extra span. Spans per request goes from 7 to 12; ingest grows 70 percent. The post-mortem never attributes the growth to the middleware because nobody sampled at the boundary.
3. The 50-attribute span. A new observability library applies every HTTP header as an attribute. Bytes per span triples. Sampling rate is unchanged; ingest bytes triple.

In each shape the fix is the same: bound sampling, keep the interesting traces, drop the rest.

How the trace cost model works

The mental model is two boundaries, both at the collector. The agent chooses what to send; the collector chooses what to keep.

   service runtime         SDK sampler (head-based, decision at start)
       |
       v
   agent (Alloy / otel-agent) --->  cost boundary 0 (early drop, cheap)
       |
       v
   collector (otel-collector / Alloy) --->  cost boundary 1 (tail decision)
       |                                       (tail sampler keeps the
       |                                        interesting traces)
       v
   ingester / distributor (Tempo)
       |
       v
   block store (hot + cold)
       |
       v
   querier / search

Head-based sampling decides at the start of the trace and is cheap; it cannot know which traces are interesting. Tail-based sampling buffers spans and decides at the end; it can keep errors, slow traces, and labelled traces. The cost trade-off is CPU at the collector (tail) versus data the team needs to keep (head).

How to control trace cost

The right configuration combines head-based sampling at the agent with tail-based sampling at the collector. The agent reduces the wire volume; the collector keeps the interesting traces.

# File: /etc/otelcol/config.yaml
# Severity: CONFIGURATION (reload required)
# OpenTelemetry Collector 0.110.x

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

# Drop spans that are obviously cheap to filter: health checks and
# static asset fetches. These span categories are typically 30 to
# 50 percent of an API gateway's span count.
processors:
  filter/health:
    error_mode: ignore
    traces:
      span:
        - 'attributes["http.target"] == "/healthz"'
        - 'attributes["http.target"] == "/readyz"'
        - 'attributes["http.route"] == "/static/*"'

  # Tail-based sampler: 100 percent of error traces and slow
  # traces; 5 percent of success traces. The decision is per-trace.
  tail_sampling:
    decision_wait: 10s           # how long to buffer
    num_traces: 50000           # in-memory trace buffer
    expected_new_traces_per_sec: 1000
    policies:
      - name: errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: slow
        type: latency
        latency: { threshold_ms: 1500 }
      - name: sample-default
        type: probabilistic
        probabilistic: { sampling_percentage: 5 }

  # Cap per-span attribute count to bound bytes-per-span.
  transform/sanitize:
    trace_statements:
      - context: span
        statements:
          - 'truncate_all(span.attributes, "", 32, "")'
          - 'replace_all(span.attributes, "http.request.body", "")'
          - 'replace_all(span.attributes, "db.statement", "")'

  batch:
    send_batch_size: 1024
    timeout: 5s

exporters:
  otlphttp/tempo:
    endpoint: http://tempo:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [filter/health, transform/sanitize, tail_sampling, batch]
      exporters: [otlphttp/tempo]

The companion head-sampling configuration on the SDK side, for high-volume endpoints, looks like:

# File: app-observability-init.py
# Severity: CONFIGURATION (reload required)
# Head-based sampler for the API gateway. Drops 90 percent of
# traces before they reach the collector; the collector sees
# 10 percent and applies its own tail-based policy.

from opentelemetry.sdk.trace.sampling import (
    ParentBasedTraceIdRatio, TraceIdRatioBased,
)

sampler = ParentBasedTraceIdRatio(TraceIdRatioBased(0.10))
provider = TracerProvider(sampler=sampler)

The configuration choices that do the work:

• filter/health removes the health-check span category at the receiver, before the tail sampler buffers the trace.
• tail_sampling keeps the interesting traces and discards the common ones, with bounded in-memory trace count.
• transform/sanitize caps attribute count and removes payload attributes; the bytes-per-span ceiling is enforced by the collector, not the application.
• The head-based sampler at 10 percent is the largest single cost reduction; the tail sampler adds detail on what survives.

How to validate trace cost

Three queries answer the questions that matter.

# Severity: READ-ONLY
# Spans ingested per second by service, top 10.
tempo-cli query metrics \
  --addr=http://tempo:3100 \
  --query='topk(10, sum by (service) (rate(tempo_ingester_spans_received_total[5m])))'

illustrative:
{service="api-gateway"}    "38000"
{service="auth-svc"}       "18000"
{service="payment-svc"}    "12000"
...

# Severity: READ-ONLY
# Bytes per span average, last 1 h.
tempo-cli query metrics \
  --addr=http://tempo:3100 \
  --query='rate(tempo_ingester_bytes_received_total[1h])
           / rate(tempo_ingester_spans_received_total[1h])'

illustrative: 920

# Severity: READ-ONLY
# Tail-sampler dropped ratio. A high drop ratio with a
# low keep ratio means the policies are configured correctly.
curl -s 'http://otelcol:8888/metrics' \
  | grep -E '^otelcol_processor_tail_sampling_(sampled|drop)_spans_total'

illustrative:
otelcol_processor_tail_sampling_sampled_spans_total{...}="320000"
otelcol_processor_tail_sampling_drop_spans_total{...}="6080000"

The validation passes when spans per second is below the budget ceiling, bytes per span is below roughly 1 KiB on average and the tail-sampler drop ratio is high (which is the point).

How it can fail

Six shapes repeat in production.

1. Default sampler at 100 percent. The collector is deployed with the default config; every trace is kept. After a week the block store is the largest line on the bill.
2. Tail sampler buffer too small. num_traces is set to 5 000 while the platform ingests 30 000 traces/sec. The buffer overflows and the tail sampler drops traces without applying the policy.
3. decision_wait shorter than the slowest trace. A trace with a 25-second dependency is dropped by the tail sampler because the buffer expires first. The interesting traces never reach storage.
4. Payload attributes fanout. A new HTTP filter attaches the request body and headers as span attributes. Bytes per span grows 30-fold. The block store swells within hours.
5. Span count doubled by a middleware. A change in instrumentation adds a span around every database call. Span count grows 70 percent with no change in request rate. The monthly bill catches up at month end.
6. http.url left in attributes. A URL with embedded query parameters explodes attribute cardinality at the SDK side and inflates the columnar store. Sampling does not help because the bytes have already been spent.

How to troubleshoot runaway trace cost

The diagnostic order is: count, attribute, decide.

Symptom (spans per second jumped 50 percent)
   |
   +-- Per-service spans: which service owns the growth?
   |
   +-- Per-endpoint spans: which route on that service?
   |     |
   |     +-- Span count per request: did this grow?
   |     +-- Bytes per span: did this grow?
   |     |
   |     |-- Span count grew  --->  new middleware, new SDK init,
   |     |                        or new instrumentation hook
   |     |
   |     +-- Bytes per span grew  --->  attribute volume grew;
   |                                  check http.url, payload attrs
   |
   +-- Decide: head sampling at the agent, tail sampling at the
   |          collector, or attribute cap at the collector
   |
   +-- Verify: did tempo_ingester_spans_received_total fall?
   +-- Document: cost platform change log
   |
Root cause

Sampling is rarely the right first move when only one service grew. Sampling is the right first move when every service grew. Distinguish the two before changing the platform-wide default.

Security implications

A span can carry anything the instrumented code attaches. PII, secrets and request bodies end up in the block store whenever a filter or attribute is set carelessly. Sampling does not help because the bytes are spent in-process before the collector sees them. The control belongs at the SDK or at the transform/sanitize processor: drop payload attributes before they leave the process. Treat the trace retention as a sensitive retention; per-stream retention limits shorten the leakage window.

Authentication on the OTLP receiver is mandatory. An unauthenticated Tempo accepts trace pushes from anywhere and stores them under whichever tenant ID is claimed. Mutual TLS to the collector is the right control for production receivers.

Performance implications

The collector’s tail sampler holds spans in memory for decision_wait. Per-pending-trace memory is the binding constraint for the collector. When tail-sampler memory exceeds budget, the collector OOMs; the OTLP receiver returns 5xx and agents back off. The second ceiling is exporter throughput (Tempo’s ingester has a per-batch span cap); the third is the block store’s compaction throughput. A trace budget is therefore a forecast of pending-trace memory times retention times compaction rate.

Tune the collector first, then set the budget ceiling at roughly 70 percent of measured headroom.

Production guidance

• Head sample at the agent before tail sampling at the collector. Tail sampling without head sampling oversizes the collector.
• Use a tail-sampling policy that combines policy types: errors (status code), slow (latency), labelled (attribute match), default (probabilistic). The four policy types cover the common investigation patterns without 100 percent sampling.
• Cap attributes per span at the collector with transform/sanitize. The cost of one 50-attribute span is the cost of one hundred 1-attribute spans.
• Watch tempo_ingester_spans_received_total and alert at 80 percent of budget. The shape that catches spans is the one that grew 70 percent in a week.
• Set decision_wait to the p99.9 of upstream dependency latency. A wait that is too short drops the interesting traces silently.

Verification

You should now be able to answer:

• What four quantities determine steady-state trace storage cost, and which is the most powerful lever?
• Why is head-based sampling at the agent cheaper than tail- based at the collector, and why is the combination the right answer?
• What three policies are the typical shape of a tail-sampling configuration?
• What is the right diagnostic order when spans per second jump 50 percent after a deploy?
• Why is decision_wait set above the p99.9 of upstream dependency latency?

Quiz