Trace Load Patterns

At 09:00 a marketing email went out to 4 million customers. In the next 12 minutes the platform received 8x normal traffic. The Tempo distributor queue maxed out at 10,000 spans in flight. The ingester’s WAL grew to 80% of its disk. The distributor returned 503 for 40 minutes. The fix was not more ingesters — the ingester could have absorbed the load if the queue had been larger. The fix was raising the distributor queue depth from 10,000 to 50,000 and adding a per-tenant ingestion rate limit.

This lesson is the discipline of sizing Tempo for the load shapes the platform actually produces: spikey vs steady, the buffer sizing for each, and the failure mode for getting it wrong.

What it is

Trace load is the rate of spans and traces arriving at Tempo. Two shapes cover almost every production workload:

• Steady load. Predictable, near-constant rate. The workload of a payment API or a back-office tool. Spans/sec varies by less than 2x over the day.
• Spikey load. Bursts of 5-100x the steady rate over short windows. The workload of a marketing email, a black-friday sale, or a deploy that triggers a thundering herd.

The shapes stress different components:

  Steady load
    |
    v
  +-------------------+   +-------------------+   +-------------------+
  | Distributor       |   | Ingester          |   | Storage           |
  | CPU-bound on      |   | Memory-bound on   |   | Bucket grows      |
  | decoding.         |   | head blocks.      |   | linearly.         |
  | Queue depth low.  |   | WAL drains.       |   | Compactor keeps   |
  |                   |   |                   |   | up.               |
  +-------------------+   +-------------------+   +-------------------+

  Spikey load
    |
    v
  +-------------------+   +-------------------+   +-------------------+
  | Distributor       |   | Ingester          |   | Storage           |
  | Queue saturates.  |   | WAL grows fast.   |   | Compactor backlog |
  | Returns 503 under |   | Disk fills.       |   | during peak.      |
  | overload.         |   |                   |   |                   |
  +-------------------+   +-------------------+   +-------------------+

A platform that has only steady load can right-size for the steady rate. A platform with spikey load must size for the peak or implement explicit load-shedding.

Why a sysadmin cares

Three operational pains are specific to trace load:

1. Distributor 503s under spike. The distributor queue saturates when span arrival rate exceeds the queue capacity. The distributor returns 503 to the client. The OpenTelemetry Collector retries with backoff; an SDK with no buffer drops the spans.
2. WAL growth under spike. The ingester accepts spans faster than it flushes. The WAL grows. A sustained spike fills the disk and the ingester stops accepting spans.
3. Compactor backlog after spike. A spike produces a burst of blocks. The compactor merges them on its cycle. A spike that produces more blocks than the compactor can drain in one cycle produces a backlog; query latency rises until the backlog clears.

The failure shape differs by load type. The diagnosis differs accordingly.

How it works

A trace arrives at Tempo through one of three receivers on the distributor. The distributor decodes the payload, validates it, applies per-tenant rate limits, and forwards it to the ingester ring.

  Client / Collector
       |
       |  spans/sec
       v
  +-----------------------------------+
  | Distributor                       |
  |                                   |
  |   decode OTLP / Jaeger / Zipkin   |
  |   validate                        |
  |   rate limit (per tenant)         |
  |   queue (max queue_depth)         |
  |   hash(trace_id) -> ingester ring |
  +-----------------------------------+
       |
       v
  +-----------------------------------+
  | Ingester                          |
  |                                   |
  |   append to head block            |
  |   fsync to WAL                    |
  |   flush to bucket (background)    |
  +-----------------------------------+
       |
       v
  +-----------------------------------+
  | Object storage                    |
  +-----------------------------------+

The distributor’s queue depth is the buffer between client arrival rate and ingester drain rate. The ingester’s WAL is the buffer between arrival rate and flush rate. Each buffer has a size; the failure shape is the buffer overflowing.

How to configure it

A production distributor config pins the queue depth, the rate limits, and the receivers. The values below match the production defaults for a mid-scale cluster:

distributor:
  receivers:
    otlp:
      protocols:
        grpc: { endpoint: '0.0.0.0:4317' }
        http: { endpoint: '0.0.0.0:4318' }

  # Per-tenant ingestion rate limit. spans/sec per tenant.
  # 0 = unlimited.
  rate_limit:
    ingestion_rate_limit: 0
    ingestion_burst_size: 0
    max_traces_per_second: 0

  # Forward to the ingester ring.
  ring:
    kvstore:
      store: memberlist

ingester:
  # The WAL path. NVMe-backed local SSD.
  # Size for at most 5 minutes of peak load.
  trace_idle_period: 10s
  max_block_duration: 30m
  flush_check_period: 5s
  max_block_bytes: 524288000

storage:
  trace:
    backend: s3
    wal:
      path: /var/tempo/wal

Three details to call out:

• ingestion_rate_limit per tenant. A spikey workload from one tenant should not starve other tenants. The rate limit is the per-tenant ceiling; bursts above the limit are dropped at the distributor with a counter increment.
• WAL disk sizing. The WAL grows at spans_per_sec * bytes_per_span until the next flush. For a 50k spans/sec workload with 1 KiB per span, that is 50 MiB/s. A 15-minute flush cadence produces a 45 GiB peak. The disk must hold at least 2x that for safety.
• OpenTelemetry Collector in front. The Collector’s loadbalancingexporter and tail_sampling processor absorb bursts at the edge before they reach the Tempo distributor. This is the right place for per-tenant sampling decisions.

Severity: CONFIGURATION. Reload requires a process restart for most settings.

How to validate it

Severity: READ-ONLY.

1. Confirm spans are arriving at the distributor:

curl -s http://tempo-distributor:3200/metrics \
  | grep tempo_distributor_spans_received_total
# tempo_distributor_spans_received_total{tenant="single-tenant"} 18423102

2. Confirm the distributor queue depth:

curl -s http://tempo-distributor:3200/metrics \
  | grep tempo_distributor_queue_depth
# (no built-in metric; check via the OpenTelemetry Collector's
#  loadbalancingexporter metric for queue size)

3. Confirm the ingester drain rate:

curl -s http://tempo-ingester:3200/metrics \
  | grep -E '^tempo_ingester_(spans_received|blocks_flushed)_total'
# tempo_ingester_spans_received_total 18423102
# tempo_ingester_blocks_flushed_total  14293

The two should track each other; the ingester should be receiving what the distributor forwards.

4. Confirm the WAL has headroom:

kubectl exec tempo-ingester-0 -- df -h /var/tempo/wal
# Filesystem      Size  Used Avail Use% Mounted on
# /dev/nvme0n1    100G   32G   68G  32% /

5. Confirm 5xx rate is low:

curl -s http://tempo-distributor:3200/metrics \
  | grep tempo_distributor_ingester_client_request
# tempo_distributor_ingester_client_request_duration_seconds_count{status_code="503"} 12

A non-zero 503 count means the ingester ring is not keeping up.

How it can fail

Six shapes appear repeatedly:

1. Distributor queue saturated under spike. The distributor queue fills before the ingester ring drains. Symptom is 503 from the OTLP port and tempo_distributor_dropped_spans_total rising.
2. WAL growth under sustained spike. The ingester accepts spans faster than it flushes. Symptom is the WAL disk filling and tempo_ingester_failed_flushes_total rising once the disk is full.
3. Compactor backlog after a spike. The spike produced a burst of blocks. The compactor cannot drain them in one cycle. Symptom is bucket object count growing faster than tempo_compactor_blocks_compacted_total.
4. Per-tenant starvation. One tenant sends a spike that fills the distributor queue. Other tenants’ spans are dropped because the queue is full. Symptom is the single-tenant rate-limit counter rising on the noisy tenant and 503s on the quiet tenants.
5. OpenTelemetry Collector buffer overflow. The Collector’s memory queue fills before it can forward to Tempo. Symptom is otelcol_exporter_queue_size at the cap and dropped spans logged at the Collector.
6. Distributor OOM under spike. A spike produces a burst of large spans. The distributor decodes each one in memory. Symptom is go_memstats_heap_inuse_bytes growing past the pod limit and the OOM killer firing.

How to troubleshoot it

The diagnostic order:

1. What is the current span rate? tempo_distributor_spans_received_total rate over the last five minutes.
2. Is the distributor queue full? Check the OpenTelemetry Collector’s loadbalancingexporter queue size.
3. Is the ingester WAL draining? Check tempo_ingester_local_checkpoint_manager_last_saved_timestamp.
4. Is the disk filling? df -h on the WAL path.
5. Is the compactor keeping up? Check tempo_compactor_blocks_compacted_total rate.
6. Are tenants starving each other? Check the per-tenant tempo_distributor_spans_received_total counters.

Security implications

The distributor is the only component that accepts public traffic by default. Two attack surfaces:

• Rate limit as DoS protection. A rate_limit setting that is too high (or zero, meaning unlimited) lets a single client consume the entire ingester ring’s capacity. A ingestion_rate_limit of 0 is acceptable for a private cluster; a public-facing OTLP endpoint needs an explicit per-tenant limit.
• Authentication. The OTLP port accepts spans from any source. Without authentication, any caller can write spans to the platform’s Tempo. Treat the OTLP port as service-internal.

Performance implications

Trace load cost has three axes:

• CPU at the distributor. Span decoding scales with spans per second.
• Memory at the ingester. Head blocks scale with active traces.
• Disk at the ingester. The WAL scales with spans per second between flushes.

The right sizing depends on the load shape:

• Steady load. Size for the steady rate with 2x headroom. Three ingester pods with RF=3. Distributor sized for steady_spans_per_sec * 1.5.
• Spikey load. Size for the peak rate, not the steady rate. Five ingester pods with RF=3 (one rolling). Distributor queue depth sized for the peak burst. WAL disk sized for the peak burst between flushes.
• Unbounded spikey load. Implement load-shedding at the OpenTelemetry Collector. Tail-based sampling, head-based sampling, or explicit per-tenant limits.

Production guidance

• Always put the OpenTelemetry Collector (or Grafana Alloy) in front of the Tempo distributor. The Collector absorbs bursts and applies per-tenant sampling.
• Set ingestion_rate_limit per tenant. The default of 0 means unlimited; a single noisy tenant can starve the rest.
• Size the WAL disk for the peak load between flushes, not the steady rate.
• Alert on the distributor 503 rate, the WAL disk fill, and the compactor block-count trend.

Verification

You should now be able to answer:

• What is the difference between spikey and steady load?
• Which component does each load shape stress?
• What is the failure shape when the distributor queue is too small?
• How do you size the WAL disk for a spikey workload?
• When is explicit load-shedding at the OpenTelemetry Collector the right approach?

Quiz