Tail Sampler Error

The on-call engineer opens the gateway collector at 11:00. The trace of an error that hit the dashboard two hours ago is in Tempo. So is every other trace for the same service from the last hour. The kept rate is one hundred percent. The tail sampler is not sampling; it is passing everything through.

This is a tail sampler error. A tail sampler error is a production event in which the gateway collector’s tail_sampling processor makes the wrong keep-or-drop decision: every trace is kept when the policy says only a fraction should be, or every trace is dropped when the policy says errors should be retained. The proximate symptom is a per-policy kept-rate counter that does not match the configured intent; the proximate cause is almost always a config bug, a buffer overflow, or a decision window that expires before the trace is complete.

What a tail sampler error is

A tail sampler error is not the same as a sampling misconfiguration. A sampling misconfiguration is at the producer or edge; the team sees per-service rates that have drifted from the documented baseline. A tail sampler error is at the gateway; the team sees per-policy counters that do not match the documented policy. The two look similar in the trace search panel — the kept rate is high in both — but the fix differs. The sampling misconfiguration is reverted at the producer. The tail sampler error is debugged at the gateway.

The shape is recognisable: a per-policy otelcol_processor_tail_sampling_count_traces_kept counter that climbs at the same rate as the incoming trace volume (the sampler is keeping everything), or a otelcol_processor_tail_sampling_traces_dropped_too_early counter that climbs while every policy kept counter is flat (the buffer is overflowing).

Why a sysadmin cares

A tail sampler error takes the tracing platform out of budget when it keeps everything, or out of service when it drops everything. When the sampler keeps everything, the kept rate is twenty to one hundred times the budgeted rate; the compactor falls behind; retention is honoured by evicting the older data faster than intended. When the sampler drops everything, the on-call investigation loses its primary signal: the “which dependency is on fire” question has no answer because the errors were discarded at the gateway.

The cost of the incident is paid by the team whose traces are missing or whose platform is full. The cost of the next incident is paid by the on-call engineer who must rebuild a per-policy baseline under pressure.

How it works

A tail sampler error manifests at one of three points in the gateway collector’s pipeline: the policy list, the in-memory trace map, or the decision window. Each point has its own metric and its own recovery.

                  Edge collector  ---span--->  Gateway collector
                                                          |
                                                          v
                                                  tail_sampling
                                                          |
                                              +----------+----------+
                                              |                     |
                                       policy list           num_traces map
                                       (keep / drop)         (in-memory)
                                              |                     |
                                              +----------+----------+
                                                         |
                                                         v
                                                  decision_wait
                                                         |
                                                         v
                                                       batch
                                                         |
                                                         v
                                                      Tempo

A tail sampler error is present when:

• The policy list matches every trace. The “keep-baseline” probabilistic policy was widened to one hundred percent; the “keep-errors” status_code policy was widened to include OK. Every trace matches and every trace is kept.
• The num_traces map overflows. The in-memory map is sized smaller than the in-flight trace count. Traces that would have matched a policy are dropped before the decision is made.
• The decision_wait expires before the trace is complete. The decision is made on a partial trace; the missing spans arrive later and are silently dropped.

Under the hood

How to configure it

The fix for a tail sampler error is to size the in-memory map correctly, to keep the policy list narrow, and to set decision_wait longer than the slowest trace p99. The prevention is a baseline of expected kept-rate per policy.

# /etc/otelcol/config.yaml  (gateway collector, tail sampling)
processors:
  # num_traces bounds the in-memory trace map. Set it to roughly
  # expected_new_traces_per_sec * decision_wait * 2. The factor
  # of two is for trace-rate variance.
  tail_sampling:
    decision_wait: 10s
    num_traces: 50000
    expected_new_traces_per_sec: 1000
    policies:
      - name: keep-errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: keep-slow
        type: latency
        latency:
          threshold_ms: 1000
      - name: keep-baseline
        type: probabilistic
        probabilistic:
          sampling_percentage: 1

exporters:
  otlp:
    endpoint: tempo.observability.internal:4317

service:
  pipelines:
    traces:
      receivers:  [otlp]
      processors: [tail_sampling]
      exporters:  [otlp]

The diagnostic discipline: every policy should be readable from a single source of truth. A policy that was widened to debug a spike should be time-boxed. The num_traces value should be sized from expected_new_traces_per_sec, not from a guess. The decision_wait should be longer than the slowest trace p99; a value shorter than the slowest trace makes decisions on incomplete traces.

How to validate it

When the tail sampler is misbehaving, the first read is the per-policy counters, the num_traces map fill, and the decision_wait versus p99 trace duration.

# READ-ONLY: per-policy kept and dropped counters.
curl -s http://gateway-collector:8888/metrics \
  | grep '^otelcol_processor_tail_sampling'

# READ-ONLY: confirm num_traces is sized for the current
# traffic. Read the in-memory trace count.
curl -s http://gateway-collector:8888/metrics \
  | grep '^otelcol_processor_tail_sampling_traces_'

Illustrative output during an incident:

otelcol_processor_tail_sampling_count_traces_kept{policy="keep-errors"}     42
otelcol_processor_tail_sampling_count_traces_kept{policy="keep-slow"}       17
otelcol_processor_tail_sampling_count_traces_kept{policy="keep-baseline"} 4812
otelcol_processor_tail_sampling_count_traces_dropped                       24018
otelcol_processor_tail_sampling_traces_dropped_too_early                  12

A keep-baseline counter that climbs much faster than the configured sampling_percentage: 1 indicates a widened policy. A traces_dropped_too_early counter that climbs while policy counters are flat indicates an undersized num_traces.

# READ-ONLY: confirm the slowest trace p99.
# Tempo exposes a generated metric for trace duration p99.
curl -s http://tempo:3200/metrics \
  | grep '^tempo_ingester_traces_duration'

If the trace duration p99 exceeds decision_wait, the decision is being made on incomplete traces. Increase decision_wait to roughly twice the p99 and revalidate.

How it can fail

The six failure shapes that account for the great majority of tail sampler errors:

1. num_traces undersized. The map fills before traces can be decided. The processor logs forced to drop and increments traces_dropped_too_early. The kept rate falls because the errors the policy was meant to keep were dropped before the decision was made.
2. decision_wait shorter than the slowest trace. The decision is made on a partial trace. Spans that arrive after the decision are silently dropped. The kept trace is incomplete; the dropped trace is silent; the policy outcome is decided on a partial trace.
3. Policy widened to one hundred percent. A “debug” policy was added at sampling_percentage: 100 and never removed. The keep-baseline counter climbs at the full incoming rate. The kept rate is twenty to one hundred times the budgeted rate.
4. Policy list reordered. A new policy was added at the top of the list. The first policy to match wins; the existing policies are no longer reached. The new policy keeps different traces than the old.
5. Two policies, mutually exclusive. A trace matches both keep-errors and keep-baseline; the first policy wins, but the trace would have matched the second policy too. The kept count is correct in aggregate but the policy attribution is wrong, which hides a misconfigured keep-errors from the per-policy counter.
6. Tail sampler placed before a processor that adds a label. A resource processor that adds the env label is placed after tail_sampling. The policies cannot filter on env. Every trace matches every policy; the kept rate is the full incoming rate.

How to troubleshoot it

The diagnostic order is fixed: confirm the symptom, locate the policy, locate the buffer, then act.

1. Confirm the symptom. Read otelcol_processor_tail_sampling_count_traces_kept per policy and otelcol_processor_tail_sampling_traces_dropped_too_early. Compare to the baseline. A per-policy counter that has doubled or tripled is the textbook symptom.
2. Locate the policy. Read the on-disk config. A policy that was widened, reordered, or added is the entry point of the error. Cross-reference with the change log.
3. Locate the buffer. Read traces_dropped_too_early. A climb indicates the map is filling. Compare the actual in-flight trace count to num_traces. If the actual is greater than num_traces, the buffer is undersized.
4. Locate the decision window. Compare decision_wait to the slowest trace p99. If decision_wait is shorter, the decision is being made on incomplete traces.
5. Form a hypothesis. Identify when the kept rate started climbing. The hypothesis is almost always correlated with a config push in the change log.
6. Act. Revert the policy, raise num_traces, raise decision_wait. Validate with the metric before moving on.

Security implications

A tail sampler error that keeps every trace exposes more span content to the backend than the documented retention window was sized for. Sensitive fields (PII, secrets, query parameters) sit in the in-memory map for decision_wait plus a buffer, and are exported to Tempo for the retention window. The fix is a narrow policy list and a documented retention window; the tail sampler is not the only control.

A tail sampler that exposes a debug policy in a shared collector can be exploited from outside the trust boundary. A policy keyed on a user-controlled attribute (for example, http.target containing a query parameter) lets an attacker choose whether their traffic is kept. The fix is a documented allow-list and a default probabilistic policy that catches everything the allow-list misses.

Performance implications

A misconfigured tail_sampling policy that keeps every trace multiplies the buffer cost: every trace’s spans sit in the in-memory map for the full decision_wait. A num_traces: 50000 with a decision_wait: 10s and an average trace size of two hundred kilobytes is ten gigabytes of resident memory. A misconfigured policy that keeps every trace raises the per-trace memory and the per-decision CPU proportionally.

A misconfigured num_traces that is undersized for the actual traffic causes the processor to spend CPU dropping traces that would have matched. The CPU cost is unchanged; the useful CPU cost is the cost of the dropped traces that the policy was meant to keep.

Production guidance

• Keep a documented baseline for every policy, every num_traces, and every decision_wait. The baseline is the source of truth; the running config is checked against it.
• Alert on otelcol_processor_tail_sampling_traces_dropped_too_early. A climb is the first signal that the buffer is undersized or the policy was widened.
• Alert on per-policy otelcol_processor_tail_sampling_count_traces_kept. A policy counter that climbs disproportionately fast is a widened policy.
• Document a policy-override procedure with a TTL. A time-boxed override is the right pattern for debugging a spike; a permanent override is the cause of the next incident.
• Right-size num_traces from expected_new_traces_per_sec × decision_wait × 2. The factor of two is for trace-rate variance.

Verification

• Which OTel Collector metric reports the per-policy kept trace count, and what is its sibling counter for traces dropped because the map overflowed?
• What is the diagnostic order when the tail sampler is keeping every trace or dropping every trace?
• What is the most common cause of a tail sampler error?
• Where in the OTel Collector pipeline is the kept-span rate bounded, and what is the metric that confirms the buffer is sized for the actual traffic?

Quiz