Sampling Too Aggressive

A trace lookup returns trace not found. The on-call engineer follows links A through D; all are healthy. The SDK is running, the traceparent is on the wire, the collector is receiving, the exporter is delivering. The next link is the sampler. The collector log shows tail_sampling processor: dropped trace: policy "probabilistic". The Tempo ingestion rate is flat at 5% of the expected volume. The service generates 50,000 spans/sec; only 2,500 spans/sec are being kept. The probabilistic sampler was set to 5% during a cost-cutting exercise six months ago and never raised. Link E is broken in the “too aggressive” direction: too few traces are sampled. The investigation has the same answer in the reverse case: a probabilistic sampler at 100% keeps every trace, saturates the storage budget, and the team is asking why the bill doubled. The two failure shapes share a diagnostic order.

What it is

“Sampling too aggressive” covers two failure shapes on the same link. Link E is the sampling decision; the failure is that the decision is wrong.

• Too few traces sampled. The sampler drops traces that the operator wanted to keep. A request fails; the trace is not in Tempo because the sampler decided not to keep it. The user sees the failure; the platform sees nothing.
• Too many traces sampled. The sampler keeps traces that the operator wanted to drop. The Tempo bucket fills; the bill doubles; the operator’s storage budget is exhausted.

The two shapes share a sampler. The sampler decides RecordAndSample or Drop. The decision is made either in the SDK (head sampling) or in the collector (tail sampling) or both. The wrong decision in either direction costs the operator time and money.

Why a sysadmin cares

The sampler is the policy knob on the trace pipeline. Turning it down to 1% saves storage; turning it down to 1% drops the only trace of the production incident. Turning it up to 100% keeps every trace; the bill triples. The default setting is wrong for almost every fleet.

The diagnostic must distinguish head from tail sampling because the fix differs:

• Head sampling decision is made at span start. The decision is propagated in the traceparent header flags. A trace dropped at head cannot be rescued downstream.
• Tail sampling decision is made after span collection. The collector holds all spans for decision_wait, evaluates policies, and decides per trace. A trace dropped at tail is a collector policy decision.

The wrong fix is to add a tail-sampling policy when the sampler is at the SDK head. The head decision is already made; the tail sampler never sees the spans.

How it works

Head sampling in the SDK

The SDK’s Sampler is consulted at span start. The RecordAndSample decision propagates as flag 01 in the traceparent header. A Drop decision propagates as 00. Downstream services that respect the parent’s decision inherit the same decision.

  +-----------------+    +-----------------+
  | Service A       |    | Service B       |
  |                 |    |                 |
  | Sampler.decide  |    | Sampler.decide  |
  |   flag=01       |    |   inherited     |
  |                 |    |   flag=01       |
  | span emitted    |    | span emitted    |
  +-----------------+    +-----------------+
         |                      |
         +-------+--------------+
                 |
                 v
         flag=01 in traceparent header

A probabilistic head sampler is configured at SDK construction. Typical settings:

  ratio      traces kept   cost shape
  --------   ------------   --------------------------
  1.0        100%           full cost; full coverage
  0.1        10%            10x cheaper; 1-in-10 traces
  0.01       1%             100x cheaper; rare events missed
  0.001      0.1%           only for very high volume

A ratio of 0.01 in a 50,000 spans/sec fleet keeps ~500 spans/sec. The trace of a 1-in-1000 error is sampled with probability 1%. The error is almost never recorded.

Tail sampling in the collector

The collector’s tail_sampling processor holds spans in a trace-id-keyed buffer for decision_wait. At the end of the wait, the policies are evaluated and the trace is sampled or dropped:

  +-----------------+   +-----------------+   +-----------------+
  | receivers       |   | tail_sampling   |   | exporters       |
  |                 |   |                 |   |                 |
  | OTLP gRPC       |-->| buffer spans by |-->| kept traces     |
  | Zipkin          |   | trace_id        |   | dropped traces  |
  | Jaeger          |   |                 |   |                 |
  +-----------------+   | decision_wait   |   +-----------------+
                        | policies:       |
                        |  - errors       |
                        |  - slow         |
                        |  - probabilistic|
                        +-----------------+

The decision_wait must be longer than the slowest expected trace. A typical value is 10 seconds; a long trace may need 30 seconds. The wait is the cost of tail sampling: every trace held in the buffer consumes memory.

Policy precedence

Tail-sampling policies are evaluated in order. The first policy that matches decides. A typical layout:

  policy order   rule                                decision
  ------------   --------------------------------   ---------
  1              errors (status_code ERROR)         sampled
  2              slow (latency > 1500ms)            sampled
  3              probabilistic (ratio 0.1)          sampled
  4              (implicit)                         not sampled

Errors are always kept. Slow traces are always kept. Everything else is sampled at 10%. Traces that match no policy are dropped.

How to configure it

SDK head sampling (Go)

  // Ratio-based head sampler
  tp := sdktrace.NewTracerProvider(
      sdktrace.WithSampler(sdktrace.TraceIDRatioBased(0.01)),
      sdktrace.WithBatcher(exporter),
      sdktrace.WithResource(res),
  )

Severity: CONFIGURATION. Re-build and re-deploy.

SDK head sampling (Java)

  ENV OTEL_TRACES_SAMPLER=traceidratio
  ENV OTEL_TRACES_SAMPLER_ARG=0.01

Severity: CONFIGURATION. Re-deploy.

Collector tail sampling

  processors:
    tail_sampling:
      decision_wait: 10s
      num_traces: 50000
      expected_new_traces_per_sec: 200
      policies:
        - name: errors
          type: status_code
          status_code: { status_codes: [ERROR] }
        - name: slow
          type: latency
          latency: { threshold_ms: 1500 }
        - name: probabilistic
          type: probabilistic
          probabilistic: { sampling_percentage: 10 }

  service:
    pipelines:
      traces:
        receivers: [otlp]
        processors: [tail_sampling, batch]
        exporters: [otlp/tempo]

Severity: CONFIGURATION. Restart the collector.

How to validate it

Severity: READ-ONLY.

Read the SDK sampler log line

  kubectl logs deploy/checkout-svc --since=10m | \
    grep -iE 'sampler|parentbased|traceidratio'
  # TracerProvider[sdk.trace.SdkTracerProvider] sampler:
  #   ParentBased(root=TraceIDRatioBased(0.01))

A TraceIDRatioBased(0.01) confirms 1% head sampling. A AlwaysOn confirms no head sampling.

Read the SDK self-observability counter

  curl -s http://checkout-svc:9464/metrics | \
    grep -E '^otel_sdk_span_(started|sampled)'
  # otel_sdk_span_started_count 124821
  # otel.sdk.span.sampled_count 1247

A sampled_count / started_count ratio of 1% matches the configured sampler. A 0% ratio confirms no spans are sampled; a 100% ratio confirms all are sampled.

Read the tail-sampling decision counters

  curl -s http://otel-collector.observability.svc:8889/metrics | \
    grep -E '^otelcol_processor_tail_sampling_(decision|sampled|dropped)'
  # otelcol_processor_tail_sampling_decision_timer_count{policy="errors",decision="sampled"} 4211
  # otelcol_processor_tail_sampling_decision_timer_count{policy="slow",decision="sampled"} 982
  # otelcol_processor_tail_sampling_decision_timer_count{policy="probabilistic",decision="sampled"} 12842
  # otelcol_processor_tail_sampling_decision_timer_count{policy="probabilistic",decision="dropped"} 115578

A dropped counter that dominates sampled confirms the probabilistic policy is the dominant drop. The fix is to raise sampling_percentage.

Read the queue depth

  curl -s http://otel-collector.observability.svc:8889/metrics | \
    grep -E '^otelcol_processor_tail_sampling_(traces_in_queue|traces_dropped_too_early)'
  # otelcol_processor_tail_sampling_traces_in_queue 4211
  # otelcol_processor_tail_sampling_traces_dropped_too_early 113

A rising traces_dropped_too_early confirms the buffer overflowed before the decision could be made. The fix is to raise num_traces or shorten decision_wait.

Compare expected vs observed trace rate

  # Expected: 50000 spans/sec * 0.01 ratio = 500 spans/sec kept
  # Observed:
  curl -s http://tempo:3200/metrics | \
    grep -E '^tempo_ingester_spans_received_total'
  # tempo_ingester_spans_received_total 500

A 10x gap between expected and observed is the “too aggressive” failure shape. A 10x gap in the opposite direction is “too lenient”.

How it can fail

Six failure shapes, ordered by frequency in production fleets:

1. Probabilistic head sampler at 1%. A cost-cutting change set the SDK’s head sampler to 1% six months ago. Errors are sampled at 1%; rare errors are never recorded. Symptom: otel.sdk.span.sampled_count / started_count ratio matches the sampler; Tempo has very few traces; the on-call opens a trace lookup and gets nothing.

2. Tail sampler’s decision_wait shorter than the slowest trace. The collector decides before all spans arrive. Spans that arrive after the decision are dropped. Symptom: tail-sampled traces in Tempo have only the first few spans; the last span (often the slowest one) is missing.

3. Tail sampler dropped a slow trace. The probabilistic policy evaluated before the slow policy in the list. The trace was sampled at 10% and dropped because the ratio flag was unlucky. Symptom: a slow trace from a specific request is missing even though slow traces are supposed to be kept. Fix: move the slow policy above the probabilistic policy.

4. Head sampler set to AlwaysOff. A defensive ops change set the SDK to OTEL_TRACES_SAMPLER=always_off during an incident. The env var was never unset. Symptom: no spans are sampled; the SDK log shows AlwaysOffSampler; otel.sdk.span.sampled_count is flat at zero.

5. Tail-sampling buffer overflowed. The collector’s num_traces is too small for the input rate. traces_dropped_too_early rises. Symptom: traces are missing despite all five links being healthy; the decision was made too early because the buffer ran out.

6. Tail-sampling policy order wrong. The probabilistic policy is evaluated first. The error policy is evaluated second but never reached because probabilistic decided. Symptom: error traces are sampled at 10%; the team’s expectation of “errors are always kept” is violated.

How to troubleshoot it

The diagnostic order, link E first, cheapest signal first:

1. Read the SDK sampler log line. Confirm the head sampler configuration. A TraceIDRatioBased(0.01) is “1% kept”.
2. Read the SDK self-observability ratio. otel.sdk.span.sampled_count / started_count should match the configured ratio. A flat sampled_count is “no sampling”.
3. Read the tail-sampling decision counters. decision="dropped" for the dominant policy is the cause.
4. Read the queue depth and dropped counter. traces_dropped_too_early rising is the buffer overflow.
5. Compare expected vs observed. Tempo’s tempo_ingester_spans_received_total against the input rate. A 10x gap is the failure shape.
6. Confirm the policy order. The error policy must be evaluated first.

Security implications

The sampler is the privacy boundary for trace data.

• PII in span attributes. A trace that contains a user email or a session token must not be sampled at 100% unconditionally. The probabilistic policy should be combined with a “do not sample” rule on attribute values. The OpenTelemetry attributes processor can redact attributes before sampling.
• Cross-tenant leakage via sampling decisions. A multi-tenant collector with a single tail sampler shares decisions across tenants. A tenant-specific probabilistic policy is required.
• Tail-sampling policy audit. Every policy is a code path that decides to keep or drop. The policies must be reviewed for privacy implications.

Performance implications

The sampler is the cost knob on the trace pipeline.

• Head sampler cost. The decision is made at span start; the cost is a single integer comparison. The cost is negligible.
• Tail sampler cost. The decision is made after the buffer holds all spans for decision_wait. The cost is memory (the buffer) and CPU (policy evaluation at decision time). A 50,000-trace buffer at 1 KiB per trace is 50 MiB.
• Storage cost. The sampled traces land in Tempo. Raising the sampling rate raises the bucket size linearly. The bill rises linearly.

The right sampling rate balances:

• Diagnostic coverage (how often does the team see a representative trace?).
• Storage budget (how much does the bucket cost?).
• Operational noise (how often does the team see traces they would prefer to drop?).

Production guidance

• Always combine head and tail sampling: a lenient head sampler (10%) and a strict tail-sampling policy that drops noise.
• Always put the error policy first; the slow policy second; the probabilistic policy third. The order is the priority.
• Always size decision_wait to be longer than the slowest expected trace. A 30-second wait is appropriate for long-running workflows.
• Always size num_traces to the fleet’s QPS. The buffer must hold expected_new_traces_per_sec * decision_wait traces.
• Always alert on traces_dropped_too_early rising. The buffer overflow is the canary.

Verification

You should now be able to answer:

• What is the difference between head and tail sampling?
• Why does the order of tail-sampling policies matter?
• What is the right decision_wait for a long-running workflow?
• How do you confirm a sampler is dropping too many traces?
• What is the relationship between head sampling and tail sampling in a production fleet?

Quiz