Trace Volume Anatomy

At 02:13 the on-call rotation gets paged. The Tempo ingestion metric has doubled in the last hour. Tempo disk write throughput is flat at the saturation ceiling of the backend volume. The Tempo distributor has started logging rpc error: code = ResourceExhausted on a fraction of incoming pushes. The alert that fired says “tempo ingestion rate above budget”; it does not say why. The on-call engineer is paid to find out.

This is a trace volume incident. A trace volume incident is a production event in which the rate at which spans arrive at the tracing backend exceeds the budget the backend was sized for. The proximate symptom is ingest pressure; the proximate cause is almost always a change in the pipeline that raised the kept span rate rather than a change in user traffic.

What a trace volume incident is

A trace volume incident is not the same as a slow trace. A slow trace is one whose spans take a long time. A trace volume incident is one whose span arrival rate has multiplied: a collector that previously received fifty thousand spans per second now receives two hundred thousand, or the same collector passes twice as many spans downstream because a sampler was turned off, a hash seed drifted, a service mesh sidecar started auto-instrumenting, or a new application started emitting.

Tempo is sensitive to ingest volume for three reasons. First, the distributor fans out to the ingester ring and writes a block per trace; doubling ingest doubles block count. Second, the block builder flushes to object storage on size and age; a higher ingest rate shortens the flush window and produces more blocks per unit time. Third, the search index is built from a sample of spans and grows with retained cardinality. When the rate climbs, all three bind simultaneously.

The shape is recognisable: a step change in tempo_distributor_spans_received_total, a climb in tempo_ingester_blocks_created_total, and eventually distributor back-pressure if the ingester ring cannot keep up.

Why a sysadmin cares

A trace volume incident degrades the entire tracing platform. When the distributor is back-pressuring, spans are dropped at the edge. The investigation loses its primary signal: the “which dependency is on fire” question has no answer. Tempo disk pressure rises and may evict blocks faster than retention intends. The block-builder queue grows. The compactor falls behind. The team that “just turned off sampling” to debug a latency spike has now lost the trace of the latency spike.

The cost of the incident is paid in the next hour of the on-call engineer’s night. The cost of the next incident is paid by everyone who keeps raising the kept-span rate without a volume budget.

How it works

Trace volume is a rate: spans arriving at the collector per second, and spans arriving at Tempo per second. The two are not the same; a collector in the middle can drop, batch, or split the load.

Application SDK  ---span--->  Edge collector  ---span--->  Gateway collector
                                                                       |
                                                                       | (kept)
                                                                       v
                                                                 Tempo distributor
                                                                       |
                                                                       v
                                                                  Tempo ingester ring
                                                                       |
                                                                       v
                                                              Object storage (blocks)
                                                                       |
                                                                       v
                                                            Trace search index

The rate at each arrow is a separate signal. The most common shape of the incident is one arrow rising faster than the rest:

• Edge collector received rate rises. The application emits more spans. A new SDK release or auto-instrumentation added spans.
• Edge to gateway rate rises faster than received rate. A sampler was disabled; the keep rate went from five percent to one hundred percent.
• Tempo distributor rate rises faster than the gateway rate. A tail sampler policy was changed (status code filter widened, latency threshold lowered) and more traces qualify.
• All rates rise together but the application request rate did not. A second source started emitting (a service mesh sidecar, a new exporter in an existing service).

Under the hood

How to configure it

Trace volume is not configured; it is bounded. The configuration that matters is the one that prevents a trace volume incident from becoming an outage. Three settings carry most of the weight: a per-receiver rate limit at the edge, a per-ingester ring size, and the kept-span rate at the sampler.

# /etc/otelcol/config.yaml  (edge collector)
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
    # Refuse a push that exceeds this rate per receiver.
    # Beyond the ceiling, the collector rejects with a
    # ResourceExhausted error; the SDK retries with backoff.
    max_recv_msg_size: 4194304

processors:
  # Tail sampler with explicit kept-span ceiling
  tail_sampling:
    decision_wait: 10s
    num_traces: 50000
    expected_new_traces_per_sec: 1000
    policies:
      - name: keep-errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: keep-baseline
        type: probabilistic
        probabilistic:
          sampling_percentage: 1

  batch:
    timeout: 5s
    send_batch_size: 8192

exporters:
  loadbalancing:
    # Round-robin spans across the Tempo distributor instances.
    # Per-route sizing lives on the routing_key; the exporter
    # itself does not impose a kept-span rate.
    routing_key: traceID
    protocol:
      otlp:
        endpoint: tempo-distributor.observability:4317

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [tail_sampling, batch]
      exporters: [loadbalancing]

# /etc/tempo/tempo.yaml  (Tempo, distributor + ingester)
distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
  # The hard ceiling per distributor instance. Pushes above
  # this are rejected with ResourceExhausted.
  ingester:
    max_block_duration: 30m

ingester:
  max_block_duration: 30m
  # Per-ingester trace ceiling. A trace above this is refused.
  trace_idle_period: 10m

compactor:
  compaction:
    block_retention: 168h   # 7 days

The collector’s loadbalancing exporter is the right pattern for spreading span load across multiple Tempo distributors. The Tempo ingester ring size is the operational lever when the ingester is the bottleneck. The tail sampler’s policy list is the lever when the source is producing too many keepable traces.

How to validate it

When ingest is climbing, the first read is the Tempo distributor rate, the block creation rate, and the producer-side accepted spans. All three come from standard metric endpoints.

# READ-ONLY: Tempo distributor span rate.
curl -s http://tempo:3200/metrics \
  | grep '^tempo_distributor_spans_received_total'

# READ-ONLY: Tempo ingester block creation rate.
curl -s http://tempo:3200/metrics \
  | grep '^tempo_ingester_blocks_created_total'

# READ-ONLY: Collector accepted/sent span rate.
curl -s http://otel-collector:8888/metrics \
  | grep -E 'otelcol_receiver_accepted_spans|otelcol_exporter_sent_spans'

Illustrative output during an incident:

# HELP tempo_distributor_spans_received_total Total spans received
# TYPE tempo_distributor_spans_received_total counter
tempo_distributor_spans_received_total{tenant="default"} 4.92e+10

# HELP tempo_ingester_blocks_created_total Total blocks created
# TYPE tempo_ingester_blocks_created_total counter
tempo_ingester_blocks_created_total{tenant="default"} 1.85e+06

If the distributor rate is climbing but the producer-side accepted rate is flat, the source is upstream of the collector you are looking at. If both are climbing in proportion, the problem is at the producer. If the distributor rate is climbing while the block creation rate is flat, the ingester ring is falling behind — Tempo is the bottleneck.

How it can fail

The six failure shapes that account for the great majority of trace volume incidents:

1. Head sampler turned off. A team sets probabilistic_sampler.sampling_percentage: 100 to debug a spike, ships the config, walks away. The kept-span rate multiplies by twenty. Symptom: Tempo distributor rate climbs in step with the config push.
2. Tail sampler policy widened. A keep-errors policy previously matched status_codes: [ERROR]; a config change adds status_codes: [OK] to “compare healthy traces”. Every healthy trace is now kept. Symptom: kept rate climbs in step with the policy change; trace search index grows.
3. Hash seed drift across edge collectors. Two edge collectors ship different hash_seed values after a partial rollout. The same trace is sampled by one and dropped by another; but each collector ships its own sampled subset, and the union exceeds the budget.
4. Service mesh auto-instrumentation. Istio or Linkerd is configured with meshTracing enabled. Every sidecar emits its own spans for every request, in addition to the application’s SDK spans. The rate roughly doubles.
5. New SDK release adding default spans. An OpenTelemetry SDK upgrade introduces new auto-instrumentations (database driver, message producer). The per-request span count rises from twelve to thirty. Multiply by request rate and the kept-span rate triples.
6. Cardinality drift on span attributes. No single change is large. A hundred small additions each add a new attribute to a span. The trace search index grows; block size grows; compaction falls behind.

How to troubleshoot it

The diagnostic order is fixed: confirm the symptom, locate the collector, locate the policy, then act.

1. Confirm the symptom. Read tempo_distributor_spans_received_total and tempo_ingester_blocks_created_total. Both must be climbing together. If only one is climbing, the bottleneck is between Tempo and the collector.
2. Locate the collector. Check otelcol_receiver_accepted_spans on each collector in the fleet. The collector whose counter is climbing faster than its peers is the entry point of the excess.
3. Locate the policy. Inspect that collector’s processors. A tail_sampling policy was widened, or a probabilistic_sampler was set to 100. Cross-reference with the change log.
4. Form a hypothesis. Identify which producer emits the excess spans. The hypothesis is almost always correlated with a deploy, a config push, or a service mesh config change.
5. Find evidence. Cross-reference with the change log. The culprit is almost always correlated with a deploy or config push in the last few hours.
6. Act. Revert the policy, redeploy the edge collector, watch the distributor rate fall. Do not touch retention or block size until the rate is bounded.

Security implications

Trace volume can be triggered from outside the trust boundary if the collector accepts OTLP from an untrusted source: a port exposed without authentication, a network policy that allows ingress from outside the cluster, or a load balancer that forwards to the gRPC port. An attacker who can push arbitrary spans into the tracing backend can deny the platform to the rest of the team. Mitigation is network policy and mTLS on the OTLP receiver; the rate limits apply after the receiver has accepted the connection.

Span contents carry the same exposure as logs: PII, secrets, and identifiers in span attributes. Higher volume means more of that data sits in the retention window. The fix is to drop attributes at the SDK or the collector, not to redact in the backend.

Performance implications

Trace volume is the dominant performance variable for the tracing backend. The cost shows up in four places: distributor CPU per accepted span, ingester memory per in-flight trace, object storage write throughput per block, and search-index build rate per retained span. A pipeline at 200 000 spans per second writes roughly twice the bytes per second as one at 100 000 spans per second, and the compactor has to do twice the work per retention window.

A useful working ceiling for a single Tempo ingester ring on commodity hardware is roughly fifty thousand spans per second per ingester. Past that, scale the ingester ring horizontally.

Production guidance

• Treat every kept span as volume debt. The span that is cheap to keep today may not be cheap to keep at next year’s traffic.
• Track tempo_distributor_spans_received_total per distributor and alert before the rate reaches 80 percent of the budgeted ceiling.
• Pin the collector binary to otelcol-contrib. The loadbalancing exporter and tail_sampling processor live there; a core-binary deployment fails at startup.
• Document a sampling-override procedure. A time-boxed override (with an end time and a revert step) is the right pattern for debugging a spike. A permanent override is the cause of the next incident.
• Keep a change log that ties deploys to sampling changes. The fastest path from “ingest is climbing” to “edge collector X is at 100 percent” is the diff between the current behaviour and the pre-deploy behaviour.

Verification

• What does tempo_distributor_spans_received_total measure, and what is its companion metric on the ingester side?
• What is the diagnostic order when both Tempo and the collector show a climb?
• What is the most common cause of a trace volume incident?
• Where in the OTel Collector pipeline is the kept-span rate bounded, and where is it amplified?

Quiz