Traces Capacity

The capacity review meeting shows the observability backend holding 4.2 trillion spans across ninety days of retention. The backend is over budget. A team proposes cutting the sampling rate from five percent to one percent across the board. The proposal will reduce spans by roughly a factor of five. It will also drop ninety-five percent of the rare errors the team retained last quarter; the post-mortem from the regression that ran last month referenced a trace captured at five percent, which at one percent would not have been in the sample at all.

The rate is a number. The number has consequences on both sides. The right number is a deliberate choice, not a default. This lesson is the arithmetic that makes the choice defensible.

What trace capacity is

Trace capacity in a Tempo platform is the answer to four questions:

1. What is the spans per second the ingester ring accepts at steady state?
2. What is the bytes per span the workload emits — the average size after span attributes, events and the per-span overhead?
3. What is the retention window the compactor enforces, and how many bytes does that window produce?
4. What sampling rate per service makes the math close without losing the traces the team will need?

The four together describe what the platform will look like at the end of next month. If the workload changes, the next two lessons in this module cover forecasting and estimation.

Why a sysadmin cares

Two failure shapes drive most trace-capacity incidents. Both come from setting the rate by gut feel rather than by traffic mathematics.

• The cost blowout. A team instruments a new service at one hundred percent because “it’s only a small service.” The service is small in count but generates 200 spans per trace and runs at 800 traces per second. The backend ingests 160,000 spans per second. At thirty days of retention the backend holds 414 trillion spans. The cost review names this service as the largest single contributor.
• The blind regression. A team reduces the rate from ten percent to one percent to control cost. A regression that affects one request in a thousand produces zero retained traces: one in a hundred thousand traces is sampled, and the one-in-a-thousand error rate inside that is one in a hundred million. The team discovers the regression only when a user reports it.

The rate is the knob that controls both sides. The discipline is to choose by traffic and investigation value, not by default.

How it works: the equations

Three equations describe the trace capacity math. Each axis is a metric the platform exposes.

  spans_per_second
    = request_traces_per_second  *  spans_per_trace

  retained_spans_per_second
    = spans_per_second  *  sampling_rate

  bytes_in_retention_window
    = retained_spans_per_second  *  bytes_per_span
       *  retention_seconds

For a service at 5,000 traces per second, 30 spans per trace, sampled at 1%, with 400 bytes per span, retained 30 days:

  spans_per_second       =  5 000  *  30              =  150 000
  retained_spans/sec     =  150 000  *  0.01          =  1 500
  bytes_in_window        =  1 500  *  400  *  (30 * 86400)
                          =  1.555 TB

At $0.02 per GB-month on object storage, the storage line for that one service is roughly $30 per month. Multiply by the number of services in the fleet and the arithmetic explains the bill.

The arithmetic is linear in every term. Doubling request volume doubles the bucket. Doubling spans per trace doubles the bucket. Halving the sampling rate halves the bucket. The math is tractable; the rate is the lever.

How to configure it

A trace capacity plan is a sampling rate per service and a per-block retention policy.

1. Per-service sampling rate. Use the OTel Collector’s probabilistic_sampler processor with the routing connector to apply a per-service rate:

# /etc/otelcol/config.yaml
processors:
  probabilistic_sampler/critical:
    sampling_percentage: 25
    hash_seed: 42
  probabilistic_sampler/bulk:
    sampling_percentage: 1
    hash_seed: 42

connectors:
  routing:
    default_pipelines: [traces/bulk]
    error_mode: ignore
    table:
      - context: resource
        statement: route() where attributes["service.name"] == "checkout"
        pipelines: [traces/critical]
      - context: resource
        statement: route() where attributes["service.name"] == "auth"
        pipelines: [traces/critical]

service:
  pipelines:
    traces/critical:
      receivers:  [otlp]
      processors: [probabilistic_sampler/critical, batch]
      exporters:  [otlp/gateway]
    traces/bulk:
      receivers:  [otlp]
      processors: [probabilistic_sampler/bulk, batch]
      exporters:  [otlp/gateway]

A uniform rate across heterogeneous services produces a wrong cost distribution. The bulk service consumes ten times the budget of the auth service for one tenth the investigation value. Per-service rates are the discipline.

2. The block retention window. Tempo’s compactor deletes blocks older than block_retention; the per-tenant override lives in the limits config:

# tempo.yaml
compactor:
  compaction:
    block_retention: 720h        # 30 days, default 14 days
    compacted_block_retention: 240h   # 5 days post-compaction

storage:
  trace:
    backend: s3
    s3:
      bucket: tempo-traces-eu-west-1
      region: eu-west-1

A 30-day window is a defensible default for production; the per-tenant override can shorten it for noisy debug streams.

3. The per-block size. Smaller blocks flush more often but cost more in index overhead. The defaults are correct for most workloads:

ingester:
  max_block_duration: 10m
  trace_idle_period: 10s
  max_block_bytes: 100_000_000    # 100 MB cap per block

4. The ingest protection. Tempo rejects ingests that exceed the configured limits. Set them above peak, not above average:

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317

server:
  grpc_server_max_recv_msg_size: 4194304    # 4 MiB cap per RPC

How to validate it

The rate is correct when the counters and the formula agree.

# CONFIGURATION: parse-check the collector config.
otelcol validate --config=/etc/otelcol/config.yaml
# (no output on success; non-zero exit on error)

# READ-ONLY: per-processor counters.
curl -s http://localhost:8888/metrics | grep probabilistic_sampler

otelcol_processor_probabilistic_sampler_count_traces_sampled{policy="critical"} 312
otelcol_processor_probabilistic_sampler_count_traces_dropped{policy="critical"}  938
otelcol_processor_probabilistic_sampler_count_traces_sampled{policy="bulk"}      87
otelcol_processor_probabilistic_sampler_count_traces_dropped{policy="bulk"}     8613

The ratio sampled / (sampled + dropped) should match the configured rate within sampling noise. For critical, 312 / 1250 = 0.2496 against 25 percent — close enough. For bulk, 87 / 8700 = 0.01 against 1 percent — exact.

# READ-ONLY: confirm the hash_seed is identical across
# the fleet. A drift breaks the consistency invariant.
for h in edge-1 edge-2 edge-3; do
  ssh "$h.observability.internal" \
    "grep hash_seed /etc/otelcol/config.yaml | sort -u"
done

# READ-ONLY: Tempo's live ingest rate.
sum(rate(tempo_ingester_spans_received_total[5m]))
# Expected units: spans per second.

# READ-ONLY: Tempo bucket size on object storage.
aws s3 ls --recursive s3://tempo-traces-eu-west-1 --summarize \
  --human-readable | tail -5
# Compare to the formula. The two should agree within a
# few percent; a 30% divergence means orphan blocks or a
# retention misconfiguration.

How it can fail

1. A uniform rate across heterogeneous services. A bulk telemetry service at 12,000 traces per second at 5 percent ships 600 retained traces per second; the auth service at 1,200 traces per second at 5 percent ships 60. The bulk service consumes ten times the budget for one tenth the investigation value. Symptom: cost review names the bulk service as the largest contributor; rate uniform across the fleet was the wrong default.
2. A rate set by guess, not by measurement. A team sets one percent on a service without measuring its volume. Symptom: at one percent the service ships fewer retained traces than expected; investigation during an incident has no traces; the team cannot reproduce the failure mode.
3. A rate held constant as the service grows. A service grew from 200 to 4,000 traces per second over six months. The sampling rate stayed at one percent. Symptom: the backend ingest doubled without a corresponding budget increase; the cost review names this service as the primary contributor.
4. A rate set against the request rate, not the trace rate. A service generates five traces per request (one parent, four children). The team configures one percent based on request rate rather than trace rate. Symptom: the retained sample is five times larger than expected; the cost exceeds the budget.
5. A high-cardinality span attribute. A service adds user_id as a span attribute. Every distinct user creates a distinct trace; the trace backend’s search index grows linearly with the user count. Symptom: tempo_search_indexed_attributes_total rate rising; query latency climbing; the index budget exhausted before the bucket budget is.
6. A rate that breaks the correlation invariant. A subset of edge collectors runs at ten percent, the rest at one percent. Symptom: traces from the ten-percent subset arrive with full span counts; traces from the one-percent subset arrive with sparse spans; the backend cannot reconstruct a complete picture for any given trace identifier.

How to troubleshoot it

A wrong-rate investigation asks four questions in order.

1. Is the rate what the config says? Re-read the probabilistic_sampler block. A common bug is the field sampling_percentage vs sampling_probability; the processor takes a percentage (5 for five percent), not a probability (0.05).
2. Is the processor actually running? Confirm otelcol_processor_probabilistic_sampler_count_traces_sampled is non-zero. If it is zero, the processor is not in the pipeline. Check service.pipelines.traces.processors.
3. What is the actual request rate? Read otelcol_receiver_accepted_spans and divide by the average span count for the service. A service at ten percent that shows 50,000 spans per second at fifty spans per trace has 1,000 traces per second, not the 5,000 the rate math was based on.
4. Is the rate matched across the fleet? Diff /etc/otelcol/config.yaml across edge collectors. A drift in sampling_percentage or hash_seed produces inconsistent sampling.

Security implications

• Trace identifier as a side channel. The hash of the trace identifier with a fixed seed is deterministic. An attacker who can observe which traces are sampled can infer the seed and predict which of their own traces will be sampled. The mitigation is a randomly chosen seed per environment.
• Sensitive fields in sampled spans. A probabilistic sampler decides on the trace identifier, not on span contents. If a service emits a sensitive field (PII, secrets), the field is retained or dropped with the same probability as the trace. Redaction at the SDK or the collector is the production pattern.
• Rate as a budget for attackers. A high rate makes it easy for an attacker to confirm whether a malicious request was traced. A low rate makes confirmation probabilistic. The right rate is a function of detection need, not just budget.

Performance implications

• Producer CPU. The probabilistic_sampler processor spends a hash per trace. The cost is a fraction of a CPU core at 10,000 spans per second; it is not a bottleneck.
• Backend ingest. The retained sample count is the input to backend ingest. A five percent sample of a five-trace-per-request service at 1,000 requests per second is 250 retained traces per second, which is 21.6 million traces per day. The backend ingest is the line item in the cost review.
• Backend storage. The retained traces per day times retention is the storage footprint. 90 days of 21.6 million traces per day is 1.94 billion retained traces; the storage cost scales with that volume.

Production guidance

• Set the rate per service. A fleet-wide uniform rate is a smell. Use a routing connector or per-host collector configs to apply different rates to different services.
• Measure before choosing. Read the request rate and the trace rate from the receiver counters; set the rate so the retained volume fits the budget; revisit when the traffic grows.
• Match hash_seed across the fleet. A drifted seed breaks the consistency invariant. Use a configuration management tool to enforce the seed.
• Use sampling_percentage (whole numbers), not fractions. The processor takes 5 for five percent. A fraction in the config is the most common typo.
• Re-derive the bytes-per-span figure quarterly. A service that adds attributes pushes the average up; a service that moves to a slimmer SDK pushes it down.

Verification

You should now be able to answer:

• What is the formula that turns request volume, spans per request and sampling rate into retained spans per second?
• Why does a uniform rate across heterogeneous services produce a wrong cost distribution?
• What is the relationship between hash_seed and the correlation invariant across a fleet?
• What is the cardinality cliff, and how does it shape the search index budget?

Quiz