Tail Sampling Cost

The 03:00 page is “tail sampler OOM-killed on gateway-1.” The on-call engineer restarts the collector. Within four minutes the process is gone again. The dashboard shows the gateway has held 50 000 traces in memory and the live trace rate is 8 000 traces per second. The arithmetic is plain: a gateway at 8 000 incoming traces per second and a ten-second decision window is holding 80 000 traces. The num_traces is sized for 50 000. The map is full; the processor is dropping traces on the floor to make room for new arrivals; the kernel kills the process when the resident set exceeds the cgroup limit.

The team enabled tail sampling to keep rare errors. The cost was a gateway with a memory budget that does not exist. The fix is not to disable tail sampling; the fix is to right-size the gateway.

What it is

The cost of tail sampling is the cost of buffering every span of every trace at the gateway until the decision window expires. The buffer is in memory; the cost is the resident set of the collector process. The cost scales with the incoming trace rate, the average trace size, and the decision window.

The math is direct.

in_memory_traces
    = incoming_trace_rate  x  decision_wait

in_memory_bytes
    = in_memory_traces  x  average_trace_size

A gateway at 5 000 traces per second with a ten-second decision window is holding 50 000 traces in memory. If the average trace is 200 kilobytes (twenty spans at ten kilobytes each), the resident set is ten gigabytes. That is the budget for the gateway host; a host with sixteen gigabytes of RAM is already over-subscribed.

The num_traces parameter is the ceiling the operator sets on the in-memory map. The processor drops traces that would push the map over the ceiling. The drop is silent: the trace never reaches the policy evaluator, the export counter, or the backend.

Why a sysadmin cares

Tail sampling is the answer to a question (keep rare errors) that costs a resource (memory proportional to incoming rate). The wrong answer is “turn it on everywhere” because every gateway then needs the memory budget of the entire fleet. The right answer is “turn it on where rare errors matter and the budget exists.”

Two failure shapes dominate tail sampling cost incidents.

1. The gateway that ran out of memory. A team enabled tail sampling at a single gateway that fronts the entire production fleet. The volume is 8 000 traces per second; the decision window is ten seconds. The map fills. The processor drops traces. The kernel OOM-kills the process. The error and slow traces the team turned on tail sampling to keep are dropped with the routine traces; the rare-error coverage is zero.
2. The gateway that became the bottleneck. A team enabled tail sampling at a single gateway that has a 10 Gbit/s network. The fleet generates 200 000 spans per second; each span is five kilobytes; the line rate is 8 Gbit/s. The gateway saturates the link. Spans arrive late; the decision window expires; the policy evaluator runs on partial traces; rare errors are missed.

How it works

The tail_sampling processor holds a map of in-flight traces in memory. Every span that arrives is added to the map by trace identifier. The map is the buffer.

The processor needs three parameters to size the map.

• num_traces — the maximum number of in-flight traces. When the map fills, the processor drops new traces (configurable via drop_policy).
• decision_wait — how long to wait before deciding on a trace that has not been declared complete. A longer wait gives slow traces time to finish; a shorter wait forces decisions on partial traces.
• expected_new_traces_per_sec — a hint the processor uses to size internal data structures. The processor does not enforce a hard cap on this rate; a value that understates the real rate produces suboptimal memory use but does not affect correctness.

The map footprint at steady state is roughly incoming_rate × decision_wait × average_trace_size.

gateway-1
  incoming:        5 000 traces / second
  avg trace size:  200 kB
  decision_wait:   10 s

  in_memory traces = 5 000 x 10 = 50 000
  in_memory bytes  = 50 000 x 200 kB = 10 GB

gateway-1 with 2x traffic spike
  incoming:        10 000 traces / second
  decision_wait:   10 s

  in_memory traces = 10 000 x 10 = 100 000
  in_memory bytes  = 100 000 x 200 kB = 20 GB

The 2x spike doubles the memory. A 4x spike (a noisy deploy) quadruples it. The map is sized for the steady state; the spike blows the budget.

Under the hood

How to configure it

A right-sized gateway for a workload of 1 000 traces per second at an average span size of 100 kilobytes and a ten- second decision window.

# /etc/otelcol/config.yaml  (gateway collector, sized for the workload)

processors:
  tail_sampling:
    decision_wait: 10s
    # num_traces sized at roughly 2x the steady-state population.
    # 1 000 traces / second x 10 s = 10 000; doubled for variance.
    num_traces: 20000
    expected_new_traces_per_sec: 1000
    # Drop the oldest trace when the map fills; the oldest is the
    # most likely to be complete.
    drop_policy: oldest
    policies:
      - name: keep-errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: keep-slow
        type: latency
        latency:
          threshold_ms: 1000
      - name: keep-baseline
        type: probabilistic
        probabilistic:
          sampling_percentage: 5

service:
  pipelines:
    traces:
      receivers:  [otlp]
      processors: [tail_sampling, batch]
      exporters:  [otlp/tempo]

The memory_limiter processor must run before tail_sampling in the pipeline. Without it, the collector has no last-line defence against the tail sampling map growing past the process’s memory budget.

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25

  tail_sampling:
    decision_wait: 10s
    num_traces: 20000

  batch:
    timeout: 5s
    send_batch_size: 8192

service:
  pipelines:
    traces:
      receivers:  [otlp]
      processors: [memory_limiter, tail_sampling, batch]
      exporters:  [otlp/tempo]

The chain order matters. memory_limiter first protects everything downstream. tail_sampling next applies the policy. batch last coalesces the kept traces for export.

How to validate it

Validation confirms the gateway is sized for the workload and that the in-memory map is not dropping traces.

# CONFIGURATION: parse-check.
otelcol validate --config=/etc/otelcol/config.yaml

# READ-ONLY: read the tail sampler counters and the drop metric.
curl -s http://localhost:8888/metrics | grep tail_sampling

otelcol_processor_tail_sampling_count_traces_kept{policy="keep-errors"}     42
otelcol_processor_tail_sampling_count_traces_kept{policy="keep-slow"}       17
otelcol_processor_tail_sampling_count_traces_kept{policy="keep-baseline"} 1230
otelcol_processor_tail_sampling_count_traces_dropped                       24018
otelcol_processor_tail_sampling_traces_dropped_too_early                      0

traces_dropped_too_early is the early-warning signal. A non-zero counter means the map is full and traces are being evicted before the policy evaluates.

# READ-ONLY: read the in-memory map size.
curl -s http://localhost:8888/metrics | grep tail_sampling | grep size

otelcol_processor_tail_sampling_traces_in_queue 8734

A traces_in_queue that consistently approaches num_traces is a gateway that will drop traces on the next spike. The right fix is to raise num_traces or to add another gateway.

# READ-ONLY: read the resident memory of the collector process.
ps -o rss= -p $(pidof otelcol)

10482368   # 10 GB resident

The resident set should stay bounded regardless of the incoming rate. A climbing resident set at constant rate is a leak; a climbing resident set at rising rate is the map filling.

How it can fail

Six failure modes specific to tail sampling cost.

1. num_traces undersized for the volume. A gateway sized for 50 000 traces against a real volume of 100 000. Symptom: traces_dropped_too_early climbs at every traffic spike; rare errors are evicted before the policy evaluates; the rare-error coverage during incidents is zero.
2. decision_wait longer than the slowest trace. A team sets decision_wait: 30s to be safe. The slowest trace is 8s. Symptom: the map holds traces for 30 seconds; the population is rate × 30; the memory budget is three times what it needs to be; the gateway is over-subscribed.
3. expected_new_traces_per_sec understated. The team sized the gateway for 1 000 traces per second; the real rate is 4 000. Symptom: internal data structures are undersized; the processor falls back to slow paths; the CPU climbs; the latency of the policy decision climbs; some traces are decided after the decision window.
4. No memory_limiter in the pipeline. The tail_sampling processor is the only processor. Symptom: the map fills under a spike; the process exceeds the cgroup memory limit; the kernel OOM-kills the process; the host loses its traces.
5. A single gateway fronts the entire fleet. A team deploys one gateway collector for every service. Symptom: the gateway is a single point of failure; one process death loses every rare trace for every service; the post- mortem during the next incident has no evidence.
6. The probabilistic baseline is too high. A team sets the baseline policy to fifty percent. Symptom: the kept volume from tail sampling is half the incoming rate; the backend ingest doubles; the cost review names tail sampling as the largest contributor.

How to troubleshoot it

A saturated-gateway investigation asks five questions in order.

1. Is the map dropping traces? Read otelcol_processor_tail_sampling_traces_dropped_too_early. Non-zero means the map is full; the fix is to raise num_traces or split the traffic across more gateways.
2. What is the resident set? Read the RSS of the collector process. If the RSS is climbing at constant rate, the map is leaking; restart and watch for a return. If the RSS is climbing at rising rate, the map is filling; the budget is the wrong size for the workload.
3. What is the actual incoming rate? Read otelcol_receiver_accepted_spans and divide by the average span count. If the actual rate is higher than expected_new_traces_per_sec, the processor is undersized for the real volume.
4. Is decision_wait matched to the trace p99? Compare the trace p99 from the kept population to decision_wait. A decision_wait much longer than the p99 is wasted memory.
5. Are there enough gateways? A single gateway that fronts every service is a single point of failure. Splitting the traffic across multiple gateways (one per region, one per service tier) reduces the per-gateway memory budget and eliminates the single point of failure.

Security implications

• Resident memory as exposure. Tail sampling holds every span of every trace in memory until the decision. A larger decision_wait is a longer window for the span contents to sit in RAM. A gateway that holds 50 000 traces at 200 kB each is holding ten gigabytes of span data; the security boundary is the process’s memory, not the network.
• Network ingress as attack surface. Tail sampling concentrates the entire fleet’s trace volume at one or two gateways. An attacker who can send malformed spans to the gateway can stress the map and the policy evaluator. The receiver-level limits and the cgroup memory limit are the boundaries.
• Cost as a denial-of-service vector. A misconfigured baseline policy at one hundred percent makes tail sampling equivalent to no sampling. The cost rises in proportion to the incoming rate. A team that observes a sudden cost spike may have a policy misconfiguration or a malicious workload.

Performance implications

• Memory. The dominant cost. The resident set of the collector process scales with incoming_rate × decision_wait × average_trace_size. A right-sized gateway is sized for the steady-state population plus a factor of two for variance.
• CPU. Policy evaluation walks every buffered trace. Three policies means three walks per trace. A longer policy list means more walks; the order of policies matters because the first match short-circuits.
• Network ingress. Every span of every trace arrives at the gateway before the decision. The gateway must accept the full fleet’s trace volume. A single gateway with a 10 Gbit/s link is the ceiling for a fleet at roughly 200 000 spans per second at five kilobytes per span.

Production guidance

• Size num_traces for steady state plus variance. A value of expected_new_traces_per_sec × decision_wait × 2 is the starting point. Watch traces_dropped_too_early to detect under-sizing.
• Always pair tail_sampling with memory_limiter. The chain order is memory_limiter, then tail_sampling, then batch. A tail_sampling without memory_limiter upstream is an OOM waiting for a traffic spike.
• Match decision_wait to the slowest trace. Twice the trace p99 is the starting point. A longer wait is wasted memory; a shorter wait forces partial-trace decisions.
• Set drop_policy explicitly. oldest is the right default; the oldest traces are the most likely to be complete and the most expensive to keep.
• Run multiple gateways. A single gateway is a single point of failure. Split traffic by region or by service tier; each gateway has its own memory budget.

Verification

You should now be able to answer:

• What is the formula for the in-memory footprint of a tail sampling gateway?
• Why does num_traces need to be sized at roughly twice the steady-state population?
• Why must memory_limiter precede tail_sampling in the pipeline?
• What is the signal that the gateway is undersized?

Quiz