Rare Error Traces

A payments team investigates a chargeback spike. The customers report double-charges on a small fraction of transactions. The on-call engineer opens Tempo to find the trace of one failed double-charge. The service runs at one percent probabilistic sampling. The one-in-ten-thousand double-charge exists in roughly one in a million sampled traces. Tempo returns nothing. The investigation ends with log correlation and an educated guess.

The team turns on tail sampling at the gateway. The next day a similar double-charge appears. The trace is in Tempo. The on-call engineer traces the failure to a database transaction that was retried without idempotency. The fix ships; the chargeback spike ends. The investigation succeeded because the rare trace was kept.

This lesson is the policies that make that outcome possible.

What it is

A tail sampling policy is a rule evaluated at the gateway after the full trace has been buffered. The rule says “keep traces that match this condition.” Common conditions are the status code of the root span (the OpenTelemetry status, set by the SDK on error), the duration of the slowest span (a latency threshold), and the presence of a span event with a matching attribute (an error log). A trace that matches any of the configured policies is kept; a trace that matches none is subject to the default policy.

The policy set is the answer to the head-sampling failure shape. Head sampling cannot keep rare errors because it does not know which traces are errors. Tail sampling can, because the gateway sees the status code on the buffered trace before it makes the decision.

The typical policy combination for a critical-path service keeps every trace that meets one of three conditions.

  policy 1 (errors):     status_code = ERROR on any span
  policy 2 (slow):       duration over latency threshold
  policy 3 (baseline):   probabilistic sample of the rest

The first two policies are deterministic: every error and every slow trace is kept. The third policy is statistical: a fraction of the routine traffic is kept for distribution analysis. The combination gives the on-call engineer every trace that needs investigating while keeping a representative baseline.

Why a sysadmin cares

The whole point of a tracing pipeline is the rare error trace. The latency distribution is interesting; the throughput is interesting; the per-service error rate is interesting. None of those values answer the question “what happened to the one customer who was charged twice.” That question requires the trace of the one customer. That trace is rare. A sampling strategy that does not keep rare traces is a strategy that fails at the moment the trace is needed.

Three operational consequences follow.

1. The post-mortem has evidence. A tail-sampled pipeline with a status_code policy keeps every trace whose root span reports an OpenTelemetry error status. The post-mortem can quote the trace identifier, walk the span tree, and identify the failed dependency. The investigation ends with a known cause.
2. The p99 latency is observable. A tail-sampled pipeline with a latency policy keeps every trace whose slowest span exceeds the configured threshold. The on-call engineer can find traces at the p99 and p999 without relying on aggregate metrics to point to them.
3. The baseline is representative. A probabilistic policy at low rate (one to five percent) on the remaining traces keeps a sample of the routine traffic. The latency distribution from the sample reproduces the population within sampling noise.

How it works

The OpenTelemetry tail_sampling processor evaluates policies in declared order. The first policy to match decides keep or drop. A trace that matches no policy falls through to the default policy, which is configured separately.

The three policies the lesson covers:

• status_code — keeps traces whose root span status matches the configured status codes. The most common match is status_codes: [ERROR]. This policy requires that the application SDK set the OpenTelemetry status on error. If the SDK only sets the HTTP status code and not the OTel status, this policy matches nothing.
• latency — keeps traces whose slowest span duration exceeds the configured threshold_ms. The policy evaluates on span duration, not on aggregate metrics; it sees the actual timings of the buffered trace.
• probabilistic — keeps traces whose hash falls under the configured sampling_percentage. This is the baseline policy; it ensures that even the routine traffic is sampled for distribution analysis.

trace 8c4e...
  spans:     14
  duration:  1.4 s
  status:    ERROR (root span)

  policy 1: status_code ERROR  -> MATCH -> keep
  (policies 2 and 3 are not evaluated)

trace a7b1...
  spans:     8
  duration:  1.6 s
  status:    UNSET

  policy 1: status_code ERROR  -> no match
  policy 2: latency > 1000 ms  -> MATCH -> keep
  (policy 3 is not evaluated)

trace 2d8f...
  spans:     6
  duration:  80 ms
  status:    UNSET

  policy 1: status_code ERROR  -> no match
  policy 2: latency > 1000 ms  -> no match
  policy 3: probabilistic 5 %  -> keep (hash in range)

The policy order matters. Place the deterministic policies (errors, latency) before the probabilistic policy; the rare trace is matched early and the policy evaluator does not waste CPU on the hash for a trace it has already decided to keep.

Under the hood

How to configure it

A typical policy combination for a critical-path service.

# /etc/otelcol/config.yaml  (gateway collector, tail sampling policies)

processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 50000
    expected_new_traces_per_sec: 1000
    policies:
      # Policy 1: keep every trace with an OTel error status.
      - name: keep-errors
        type: status_code
        status_code:
          status_codes: [ERROR]

      # Policy 2: keep every trace with a span over 1 second.
      - name: keep-slow
        type: latency
        latency:
          threshold_ms: 1000

      # Policy 3: keep five percent of the rest.
      - name: keep-baseline
        type: probabilistic
        probabilistic:
          sampling_percentage: 5

  batch:
    timeout: 5s
    send_batch_size: 8192

service:
  pipelines:
    traces:
      receivers:  [otlp]
      processors: [tail_sampling, batch]
      exporters:  [otlp/tempo]

A policy set that also matches application-level error logs uses string_attribute to filter on a span event name.

processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 50000
    expected_new_traces_per_sec: 1000
    policies:
      # Match traces whose root span carries a span event named
      # "exception" (the OpenTelemetry convention for error logs).
      - name: keep-on-exception
        type: string_attribute
        string_attribute:
          key: events
          values: [exception]

      - name: keep-slow
        type: latency
        latency:
          threshold_ms: 1000

      - name: keep-baseline
        type: probabilistic
        probabilistic:
          sampling_percentage: 5

The two configurations are not exclusive. A service that uses OpenTelemetry status codes reliably can rely on status_code; a service that emits exception events can rely on string_attribute. A service that does both can chain the two policies: the first matches on status, the second on the event.

How to validate it

Validation confirms the policies are wired and that they are matching the traces they should.

# CONFIGURATION: parse-check.
otelcol validate --config=/etc/otelcol/config.yaml

# READ-ONLY: read the per-policy counters.
curl -s http://localhost:8888/metrics | grep tail_sampling

otelcol_processor_tail_sampling_count_traces_kept{policy="keep-errors"}     42
otelcol_processor_tail_sampling_count_traces_kept{policy="keep-slow"}       17
otelcol_processor_tail_sampling_count_traces_kept{policy="keep-baseline"} 1230
otelcol_processor_tail_sampling_count_traces_dropped                       24018

A non-zero keep-errors counter confirms the policy is matching; a flat counter while errors are firing elsewhere in the stack confirms the policy is not matching and the OTel status is not being set.

# SERVICE-IMPACT: force a known error trace and confirm it is kept.
# (in the test environment only)
curl -X POST http://checkout.test.internal/trigger-error

# READ-ONLY: search Tempo for the trace identifier logged by the
# trigger-error endpoint.
curl -s http://tempo.observability.internal:3200/api/search \
  --data-urlencode 'q={ service.name = "checkout" && status = error }' \
  | jq '.traces | length'

1

A count of one confirms the trace was sampled by the keep-errors policy. A count of zero confirms the policy did not match; the OTel status is not ERROR on the root span.

How it can fail

Five failure modes specific to rare-error tail sampling.

1. The SDK does not set OTel status on error. A team enables the status_code policy but the application only sets the HTTP status code; the OTel status remains UNSET. Symptom: keep-errors counter is flat while HTTP 5xx rates are climbing; the rare error trace is not kept.
2. The latency threshold is too high. A team sets the latency threshold to 5 000 ms. The slow traces they care about are at 1 200 ms. Symptom: keep-slow counter is flat; the on-call engineer has no trace of the slow tail.
3. The probabilistic policy is missing. A team enables error and latency policies but forgets the baseline. Symptom: only error and slow traces are kept; the latency distribution from the routine traffic is empty; the p50/p99 panels in Grafana return no rows.
4. The policy order is wrong. The probabilistic policy is declared first; the status_code policy is declared second. Symptom: a trace that should have been kept by the error policy is sometimes dropped by the probabilistic policy before the error policy evaluates. (The OpenTelemetry Collector evaluates policies in order; the first match wins.)
5. The decision window is shorter than the slowest trace. The decision_wait is 2 seconds; the slowest trace is 8 seconds. Symptom: the policy evaluates on a partial trace; the status code is not yet set on the root span; the error policy does not match; the rare error is dropped.
6. The buffered trace map fills before the decision is made. The num_traces is too small for the volume. Symptom: otelcol_processor_tail_sampling_traces_dropped_too_early climbs; rare traces are dropped before any policy evaluates; the platform looks healthy while the rare-error coverage is gone.

How to troubleshoot it

A missing-error-trace investigation asks five questions in order.

1. Is the OTel status being set? Confirm with a unit test or a manual trigger that the application’s error path sets the span status to ERROR. The status is set by the SDK; a service that throws and catches without setting status reports UNSET even on a thrown exception.
2. Is the policy in the pipeline? Read the service.pipelines.traces.processors list. A tail_sampling not in the list is a tail_sampling not running.
3. Is the policy matching? Read otelcol_processor_tail_sampling_count_traces_kept{policy=...}. A flat counter while the receiver is accepting spans means the policy condition is not being met by the trace contents.
4. Is the decision window long enough? Compare decision_wait to the slowest trace p99. If the p99 exceeds decision_wait, the decision is being made on a partial trace.
5. Is the buffered trace map filling? Read otelcol_processor_tail_sampling_traces_dropped_too_early. A non-zero counter means the map is full and rare traces are being discarded without evaluation.

Security implications

• Exception attributes in span events. The string_attribute policy matches on span event names, but it can also match on any attribute the application sets. If the application sets a sensitive field (PII, secrets) as a span attribute, the policy keeps the trace with the field intact. Redact at the SDK or the collector before tail sampling.
• Status code as a side channel. The status_code policy keeps every error trace. If the error message reveals a sensitive detail (an authentication failure mode, a database constraint name), the policy keeps the trace with the detail. The same redaction discipline applies.
• Decision window as an exposure window. Tail sampling holds every span of every trace in memory until the decision. A longer decision_wait is a longer exposure window for the span contents. Shorter windows reduce exposure but increase the rate of partial-trace decisions.

Performance implications

• Policy evaluation CPU. Each policy that does not match walks the buffered trace. A long policy list with no matches spends CPU on every buffered trace. Place the deterministic policies first to short-circuit on match.
• In-memory trace map. The tail_sampling processor holds spans in memory until the decision. The footprint is num_traces × average_trace_size. A larger num_traces is a larger memory budget.
• Decision window throughput. A longer decision_wait is a larger window for the in-flight trace population. The map fills with traces waiting on the window; the num_traces must be sized accordingly.

Production guidance

• Three policies is the typical minimum. Status code, latency, probabilistic baseline. Add more only when investigation demands.
• Policy order is deterministic first. Errors and latency before probabilistic, so rare traces are matched without the hash.
• Match the decision window to the slowest trace. Set decision_wait to roughly twice the p99 trace duration.
• Verify each policy matches. Force a known error trace in the test environment; confirm Tempo holds it. A policy that has never matched is a policy that does not exist.
• Right-size num_traces. Set num_traces to expected_new_traces_per_sec × decision_wait × 2. Watch traces_dropped_too_early to detect under-sizing.

Verification

You should now be able to answer:

• Why does a status_code policy on its own fail to keep application errors that are reported through exception events?
• What is the role of the probabilistic baseline policy?
• Why is policy order significant in the tail_sampling processor?
• What does decision_wait need to be longer than, and why?

Quiz