Sampling Issues

A customer reports a failed checkout at 03:47. The customer support team has the order ID. The on-call engineer searches Tempo for the customer ID. Nothing. They search for the order ID. Nothing. They search for the timestamp window. Nothing. The trace that documents this exact failure is not in Tempo. Either the trace did not happen, or the sampler decided it was not worth keeping.

This is the lesson. Sampling is the discipline that decides which traces are kept. When the discipline is wrong, the most useful traces are the ones that disappear.

What it is

Sampling is the act of deciding, for every trace, whether to keep it or discard it. The decision is made either at the edge of the system (head-based) or at a central point that has seen the whole trace (tail-based).

A sampling issue is the failure mode where a trace that the operator wants to investigate is not in Tempo because the sampler rejected it. The trace was generated, exported, and either discarded at the SDK or discarded at the collector tail sampler before reaching Tempo.

The two failure shapes are different:

• Head sampling drops early. The decision is made before the trace is complete. A high-volume service that samples 1 in 100 requests drops 99 percent of its traces at the entry point. The traces that fail are not in any system.
• Tail sampling drops late. The full trace is assembled by the collector, the policy decides keep or drop, and the decision is wrong. The trace exists in memory for the policy duration and then disappears.

Why a sysadmin cares

Sampling is the cost-control lever of distributed tracing. Storing every span of every request is operationally infeasible at scale. The compromise is sampling. The compromise is also the failure mode.

Three operational payoffs ride on the sampler:

1. The interesting traces are kept. A 1 percent head sampler keeps 1 percent of successful requests. A failed request is statistically as likely to be dropped as a successful one. The trace the on-call engineer needs is one of the 99 percent that did not make it.
2. The uninteresting traces are dropped. The cost of Tempo scales with the number of stored traces. A service that generates 50 000 spans per second and stores all of them burns through disk in hours. The sampler is the budget.
3. The decision is consistent across services. A trace that the edge sampler kept should be kept by every downstream service. When the consistency breaks, half a trace is in Tempo and the other half is dropped, and the chain is useless.

The right sampling strategy is a business decision, not a default. The wrong strategy is the default that ships in the SDK.

How it works — the mental model

Two sampling architectures exist, and the difference between them is the moment of decision.

Head-based sampling
  +-- decision at the entry point (the API gateway or edge SDK)
  +-- decision before any span exists
  +-- decision is local; no view of the rest of the trace
       |
       v
  Span 1: kept or dropped at entry
       |
       v
  Span 2..N: the sampled flag in traceparent propagates
              the decision; downstream services honour it

Tail-based sampling
  +-- decision at the collector (after the full trace is seen)
  +-- decision can see every span, every status, every duration
  +-- policy decides keep or drop
       |
       v
  Collector receives spans from every service
  Collector buffers spans until the trace is complete
  Collector evaluates policy
  Collector forwards to Tempo (or drops)

Tail sampling is more flexible but has a cost: the collector must buffer the full trace before deciding. The buffering is in-memory and bounded. A trace that takes longer than the buffer window is dropped.

How to configure it

The tail sampler lives in the OpenTelemetry Collector pipeline. The minimum configuration for a useful production policy:

# /etc/otelcol-contrib/config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  # Tail sampler: decide which traces to keep.
  tail_sampling:
    decision_wait: 10s        # wait up to 10s for the full trace
    num_traces: 50000         # in-memory buffer size
    expected_new_traces_per_sec: 1000
    policies:
      # Policy 1: keep all error traces.
      - name: keep-errors
        type: status_code
        status_code:
          status_codes: [ERROR]

      # Policy 2: keep slow traces (p99 latency).
      - name: keep-slow
        type: latency
        latency:
          threshold_ms: 2000

      # Policy 3: keep a baseline of all traces by service.
      - name: baseline
        type: and
        and:
          and_sub_policy:
            - name: baseline-by-service
              type: string_attribute
              string_attribute:
                key: http.target
                values: [/checkout, /cart, /api/v1/order]
            - name: baseline-rate
              type: probabilistic
              probabilistic:
                sampling_percentage: 5

  batch:
    timeout: 5s
    send_batch_size: 8192

exporters:
  otlp/tempo:
    endpoint: tempo.distribution.svc.cluster.local:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      # Order matters: tail_sampling must run before batch.
      processors: [tail_sampling, batch]
      exporters: [otlp/tempo]

Three things matter:

• decision_wait — how long the collector buffers the trace before deciding. Must be longer than the slowest expected request; 10 s is a reasonable default for most HTTP workloads.
• Policy order — the first matching policy decides. Put the cheap, high-signal policies first (status code, latency) and the expensive probabilistic policies last.
• Pipeline order — tail_sampling must run before batch. If batch runs first, the spans are flushed to Tempo before the policy has seen the full trace.

The head sampler is configured per-language in the SDK. The default for most SDKs is ParentBased(TraceIDRatioBased(0.0001)) — keep 0.01 percent of traces plus everything that the parent decided to keep.

// Go: head sampler with parent-based consistency
import "go.opentelemetry.io/otel/sdk/trace"

func main() {
    tp := trace.NewTracerProvider(
        trace.WithSampler(trace.ParentBased(
            trace.TraceIDRatioBased(0.01), // keep 1 percent of roots
        )),
    )
    otel.SetTracerProvider(tp)
}

The 1 percent baseline is reasonable for the common case. The tail sampler then overrides the decision for error and slow traces.

How to validate it

The validation ladder:

# 1. What sampling rate is the SDK using?
kubectl logs deploy/checkout -c app | grep -i sampler
# {"level":"info","msg":"sampler configured",
#  "type":"ParentBased(TraceIDRatioBased(0.01))"}

# 2. Is the tail sampler dropping traces?
curl -sf http://alloy:8888/metrics | grep -E "tail_sampling_(spans|traces)"
# otelcol_processor_tail_sampling_traces_dropped{policy="baseline"} 41283
# otelcol_processor_tail_sampling_traces_kept{policy="keep-errors"} 127
# otelcol_processor_tail_sampling_traces_kept{policy="keep-slow"} 89

# 3. Does Tempo have the failed request?
tctl trace search --service=checkout --status=error --since=1h \
    | grep "$ORDER_ID"
# (the order ID does not match any trace; either the trace was
#  dropped at the SDK or the tail sampler dropped it)

# 4. What fraction of traces is Tempo receiving?
curl -sf http://tempo:3200/api/search?limit=1000&since=1h \
    | jq '.traces | length'
# (rough count; the truth is the rate of traces_created_total)

# 5. Is the tail sampler policy order correct?
grep -A 20 "tail_sampling:" /etc/otelcol-contrib/config.yaml \
    | grep -E "name:|type:|sampling_percentage"
# (the keep-errors and keep-slow policies must come before the
#  probabilistic baseline, otherwise the baseline fires first)

# 6. TraceQL: count traces per status code in the last hour.
# Useful for confirming the error traces are being kept.
# curl -sf -u user:pass 'http://tempo:3200/api/search?limit=1000&since=1h' \
#     | jq -r '.traces[].rootServiceName' | sort | uniq -c

The fifth command is the structural answer. A tail sampler with the wrong policy order is the most common production cause of “the interesting traces never appear.”

How it can fail

Six recurring failure modes.

1. Head sampler too aggressive. The SDK samples 1 in 10 000 requests. The trace the team wants is one of the 9 999 that were dropped. Symptom: Tempo has very few traces per minute; the ones it has look unrepresentative of production traffic.
2. Tail sampler policy order wrong. The baseline policy (probabilistic) runs before the keep-errors policy. Error traces are evaluated by the baseline, get the 5 percent decision, and are dropped on the dice roll. Symptom: error traces appear in Tempo at 5 percent of their true rate.
3. decision_wait too short. The collector decides after 2 s. A request that takes 4 s has its tail spans arrive after the decision. The collector drops the trace because it had no view of the slow span. Symptom: slow traces never appear in Tempo even though the keep-slow policy is in place.
4. Parent-based consistency broken. The edge sampler decides “sampled”. The downstream service is configured with a different head sampler that ignores the parent. The downstream service drops its spans. Symptom: Tempo has traces with the first two spans only; the rest are absent.
5. The buffer overflowed. num_traces is too small for the trace rate. The collector discards the excess. Symptom: the tail_sampling_traces_dropped metric climbs steadily, even for the keep-errors policy.
6. The customer-facing request rate is low. A trace that represents a rare-but-important request (an admin operation, a payment refund) is sampled at 1 percent. The probability of seeing it after sampling is one in a hundred. Symptom: the team searches for a refund trace they know happened, and it is not in Tempo.

How to troubleshoot it

The diagnostic order:

1. Confirm the symptom. A trace the team knows happened is not in Tempo. The first question is whether the trace was ever exported. Check the SDK log for the sampled flag on outgoing requests.
2. Is the head sampler the right rate? 1 percent is a reasonable default for high-volume services. Low-volume services (admin operations, refund flows) often need 100 percent.
3. Is the tail sampler policy in the right order? The keep-errors and keep-slow policies must run before any probabilistic policy. Re-read the configuration.
4. Is decision_wait longer than the slowest trace? A request that consistently takes 8 s with decision_wait: 5s is dropped before the slow span arrives.
5. Is num_traces large enough? The collector drops traces when the in-memory buffer fills. Check otelcol_processor_tail_sampling_traces_dropped.
6. Is the parent-based consistency intact? Check every service in the chain for its head sampler configuration. A service with AlwaysOn will sample everything; a service with TraceIDRatioBased(0.0) will sample nothing.

Security implications

The sampling decision does not see the content of the trace. It sees attributes, status codes, and durations. The risk is around the attributes themselves: a span attribute that contains PII or session tokens is indexed by the policy. The remediation is the same as for any high-cardinality attribute (see lesson High-Cardinality Attributes): the value is hashed or stripped before the policy sees it.

The second-order risk is the sampled flag. A client that controls the traceparent header can set flags=01 (sampled) or flags=00 (not sampled) at the edge. A malicious client that wants to make a trace disappear can set the flag to 00 at the entry point. The remediation is to overwrite the incoming traceparent at the edge with a freshly generated one on a per-request basis, breaking the client’s ability to pin the flag.

Performance implications

Head sampling is free. The SDK decides once at the entry point and propagates the decision. The downstream services honour the flag and do not export their spans. The cost is one bit per request.

Tail sampling is not free. The collector must buffer the full trace in memory until the decision is made. The cost is:

• Memory. Roughly 1 KB per buffered span. A 50 000 trace buffer with an average of 10 spans per trace is 500 MB.
• Decision latency. The decision cannot be made until the trace is complete. The collector adds up to decision_wait to the trace’s end-to-end latency.
• Decision CPU. The policy evaluation is cheap (status code, latency threshold, attribute match) but not free. Roughly 10 µs per trace evaluated.

The capacity is a budgeting question. A 10 000 trace-per-second fleet with decision_wait: 10s and 10 spans per trace needs a 100 000 trace buffer — 1 GB of RAM dedicated to the tail sampler.

Production guidance

• Use tail sampling at the collector. Head sampling at the SDK drops the trace before the failure is visible. The tail sampler can decide based on the full trace.
• Keep error and slow traces unconditionally. A status-code policy and a latency threshold before any probabilistic policy. The order matters.
• Set decision_wait to the slowest expected request. A request that exceeds the wait is dropped before its slow spans arrive.
• Right-size the buffer. num_traces should be larger than expected_new_traces_per_sec * decision_wait. The metric tail_sampling_traces_dropped should be zero in steady state.
• Set the SDK head sampler to parent-based consistency. Every service must respect the parent’s sampled flag. The default ParentBased(TraceIDRatioBased(0.01)) is reasonable for high-volume services. Low-volume services may need AlwaysOn.

Verification

You should now be able to answer:

• What is the difference between head-based and tail-based sampling?
• Why does parent-based consistency matter?
• What is the correct policy order for a tail sampler that keeps error traces?
• What is the most common production cause of “the trace I need is not in Tempo”?

Quiz