Sampling Misconfig

The on-call engineer opens Tempo on Tuesday morning. The trace search panel is unusually full: every trace for the checkout service from the previous night is present. The volume is many times larger than the documented one percent head sample at the edge. Tempo disk usage has climbed faster than the forecast. Something in the sampling pipeline is keeping more than the configured share of traces.

This is a sampling misconfiguration. A sampling misconfiguration is a production event in which the OTel Collector keeps a different share of traces than the configuration intends. The proximate symptom is per-service trace rate; the proximate cause is almost always a configuration change to probabilistic_sampler, tail_sampling, or the sampling_percentage field that ships with an SDK default.

What a sampling misconfiguration is

A sampling misconfiguration is not the same as a sampling choice the team disagrees with. A disagreeable choice is a deliberate policy that some team wants to revisit. A misconfiguration is a state where the running pipeline does not match the documented intent. The two look similar in the trace search panel — the kept rate is high either way — but the fix differs. The disagreeable choice needs a policy discussion. The misconfiguration needs a config revert.

The shape is recognisable: a step change in otelcol_processor_probabilistic_sampler_count_traces_sampled, or in otelcol_processor_tail_sampling_count_traces_kept for one or more policies, paired with a rate-of-change that does not match the configured percentage.

Why a sysadmin cares

A sampling misconfiguration takes the tracing platform out of budget. When the kept rate climbs, Tempo ingests more blocks; the compactor falls behind; retention is honoured by evicting the older data faster than intended. The platform team’s alert fires on disk pressure; the engineering team investigating the original incident finds their traces are gone before the retention window closes. The team that “just bumped sampling to 100 to debug” three days ago and forgot to revert is now the cause of an unrelated team’s lost-trace investigation.

The cost of the incident is paid by the team whose traces are missing. The cost of the next incident is paid by the on-call engineer who must rebuild a per-service rate baseline under pressure.

How it works

A sampling misconfiguration manifests at one of three points in the pipeline: the producer SDK, the edge collector, or the gateway collector. Each point has its own sampler and its own metric to read.

Application SDK  ---span--->  Edge collector  ---span--->  Gateway collector  ---span--->  Tempo
                       ^                       ^                            ^
                       |                       |                            |
               SDK sampler           probabilistic_sampler          tail_sampling
               (OTEL_TRACES_SAMPLER)  (sampling_percentage)        (policies[])

The kept-span rate at each arrow is observable separately:

• The SDK sampler rate is observable through the otelcol_receiver_accepted_spans rate at the edge collector, divided by the producer’s own counters if it exposes them.
• The edge probabilistic_sampler rate is observable through otelcol_processor_probabilistic_sampler_count_traces_sampled divided by the sum of sampled plus dropped.
• The gateway tail_sampling rate is observable through otelcol_processor_tail_sampling_count_traces_kept per policy, divided by the rate of incoming traces.

A sampling misconfiguration is present when the observed ratio at any of those three points differs from the configured ratio. A common source is a config drift across edge collectors: one collector uses sampling_percentage: 5 while another uses sampling_percentage: 100. The fleet-level mean is meaningless; the per-collector observed rate is the signal.

Under the hood

How to configure it

The fix for a sampling misconfiguration is a config revert; the prevention is a baseline of expected rates. Three settings carry most of the weight: a per-policy sampling_percentage, the hash_seed consistency across edge collectors, and the list of tail_sampling policies.

# /etc/otelcol/config.yaml  (edge collector)
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  # Head sampler. The hash_seed must match across every edge
  # collector in the fleet; a drift between collectors causes
  # some traces to be sampled twice.
  probabilistic_sampler:
    sampling_percentage: 5
    hash_seed: 42

  batch:
    timeout: 5s
    send_batch_size: 8192

exporters:
  otlp:
    endpoint: gateway.observability.internal:4317

service:
  pipelines:
    traces:
      receivers:  [otlp]
      processors: [probabilistic_sampler, batch]
      exporters:  [otlp]

# /etc/otelcol/config.yaml  (gateway collector, tail sampling)
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  # The list of policies is the kept-trace rate budget.
  # Adding a permissive policy raises the kept rate.
  tail_sampling:
    decision_wait: 10s
    num_traces: 50000
    expected_new_traces_per_sec: 1000
    policies:
      - name: keep-errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: keep-slow
        type: latency
        latency:
          threshold_ms: 1000
      - name: keep-baseline
        type: probabilistic
        probabilistic:
          sampling_percentage: 1

  batch:
    timeout: 5s
    send_batch_size: 8192

exporters:
  otlp:
    endpoint: tempo.observability.internal:4317

service:
  pipelines:
    traces:
      receivers:  [otlp]
      processors: [tail_sampling, batch]
      exporters:  [otlp]

The diagnostic discipline: every sampling_percentage field should be readable from a single source of truth (a config management template, a Git repository, an Ansible role). A drift between two edge collectors is invisible to the per-host config and visible to the per-service rate in Tempo.

How to validate it

When per-service trace rate in Tempo is suspicious, the first read is the per-collector sampled/dropped counter pair.

# READ-ONLY: confirm the probabilistic sampler is wired.
curl -s http://edge-collector:8888/metrics \
  | grep '^otelcol_processor_probabilistic_sampler_count_traces'

# READ-ONLY: confirm the tail sampler is wired and read the
# per-policy kept rate.
curl -s http://gateway-collector:8888/metrics \
  | grep '^otelcol_processor_tail_sampling_count_traces_kept'

# READ-ONLY: confirm both edge collectors in the fleet are
# sampling at the same rate.
for host in edge-1 edge-2; do
  sampled=$(curl -s http://$host:8888/metrics \
    | awk '/^otelcol_processor_probabilistic_sampler_count_traces_sampled/ {print $2}')
  dropped=$(curl -s http://$host:8888/metrics \
    | awk '/^otelcol_processor_probabilistic_sampler_count_traces_dropped/ {print $2}')
  echo "$host sampled=$sampled dropped=$dropped ratio=$(echo "$sampled / ($sampled + $dropped)" | bc -l)"
done

Expected output at five percent head sampling:

edge-1 sampled=1287 dropped=24018 ratio=0.0508
edge-2 sampled=1294 dropped=24014 ratio=0.0512

A ratio close to sampling_percentage / 100 is the expected steady state. A ratio wildly different means the deployed config is not the documented one. A divergence between two edge collectors with the same config means hash_seed is drifted across the fleet.

# READ-ONLY: confirm the deployed sampler config matches
# the documented one.
grep -E 'sampling_percentage|hash_seed' /etc/otelcol/config.yaml

    sampling_percentage: 5
    hash_seed: 42

The on-disk config and the running config can diverge. The otelcol validate command confirms parse; the metric confirms running state.

How it can fail

The six failure shapes that account for the great majority of sampling misconfigurations:

1. Sampling percentage raised for debugging, never reverted. A team sets sampling_percentage: 100 on a gateway tail sampler. The debug session ends. The config stays. The kept rate is twenty times the budget.
2. Hash seed drift across edge collectors. A partial rollout ships hash_seed: 7 to half the fleet while the other half still has hash_seed: 42. The same trace is sampled by some collectors and dropped by others; the union of sampled spans is incoherent.
3. Tail policy widened with the wrong comment. A keep-errors policy is changed to status_codes: [OK, ERROR]. The comment says “errors only”. Every healthy trace is now also kept; the per-policy counter still reads “errors” so the team misses it.
4. SDK-side sampler default changed. An OTel SDK upgrade changes the default OTEL_TRACES_SAMPLER from parentbased_always_on to always_on. Every span is kept at the producer. The edge sampler’s probabilistic_sampler is bypassed because the SDK already decided.
5. Two policies, one trace. A trace matches both keep-errors and keep-baseline; the first policy wins, but a reordering of the policy list causes keep-baseline to win on the second match. Kept rate jumps because keep-baseline is the wrong policy for an error trace.
6. Sampling decision made at the wrong layer. A team enables tail_sampling at the producer collector, which only sees the partial trace. Every decision is made on incomplete information; every trace is dropped or kept at random. Symptom: kept rate is wildly variable; error traces are dropped as often as not.

How to troubleshoot it

The diagnostic order is fixed: confirm the symptom, locate the collector, locate the policy, then act.

1. Confirm the symptom. Read tempo_distributor_spans_received_total and per-service span rate. Compare to the baseline. A rate that has doubled or tripled is the textbook symptom.
2. Locate the collector. Inspect each collector’s sampled/dropped counters. The collector whose ratio diverges from the configured sampling_percentage is the entry point of the misconfiguration.
3. Locate the policy. Read the on-disk config and compare it to the documented one. The diff is the change. If the on-disk config matches the documented config, the running config differs from the on-disk config — restart the collector to apply the documented config.
4. Form a hypothesis. Identify when the rate started climbing. The hypothesis is almost always correlated with a config push in the change log.
5. Find evidence. git log on the collector config repository; the diff is the change. If the diff is empty, the running binary is not the on-disk binary.
6. Act. Revert the config, redeploy the collector, watch the per-service rate fall back to the baseline.

Security implications

A sampling misconfiguration can be exploited from outside the trust boundary if the OTel collector is reachable by an attacker who can influence the trace identifier. The probabilistic_sampler hashes the identifier; an attacker who can predict the hash can choose whether their traffic is sampled or dropped. The fix is a hash_seed chosen at deploy time and not reused across distinct security domains.

Tail sampling policies that filter on string attributes can also be exploited if those attributes are user-controlled. A policy keep-strings: { key: tenant, values: [acme] } lets the attacker decide whether their traffic is kept or dropped by choosing the tenant value. The fix is a documented allow-list and a default probabilistic policy that catches everything the allow-list misses.

Performance implications

A misconfigured probabilistic_sampler at one hundred percent turns the processor from a constant-time hash into a constant-time pass-through. The CPU cost is unchanged. The downstream cost is the cost of every span reaching Tempo. A misconfigured tail_sampling policy that keeps every healthy trace multiplies the buffer cost: every trace’s spans sit in the in-memory map for the full decision_wait, which is the right behaviour for a misconfiguration, because the cost is borne by the gateway’s memory budget, not the producer’s CPU.

Production guidance

• Keep a documented baseline for every sampling_percentage and hash_seed. The baseline is the source of truth; the running config is checked against it.
• Alert on otelcol_processor_probabilistic_sampler_count_traces_sampled divided by sampled plus dropped. A ratio outside the expected band is a sampling misconfiguration.
• Alert on per-policy otelcol_processor_tail_sampling_count_traces_kept. A policy counter that climbs disproportionately fast is a widened policy.
• Document a sampling override procedure with a TTL. A time-boxed override is the right pattern for debugging a spike; a permanent override is the cause of the next incident.
• Run otelcol validate on every config before applying. The validate catches syntax errors; the metric catches semantic drift.

Verification

• Which OTel Collector metric reports the per-policy kept rate, and what is its sibling counter for dropped traces?
• What is the diagnostic order when per-service trace rate in Tempo has doubled compared to the documented baseline?
• What is the most common cause of a sampling misconfiguration?
• Where in the OTel Collector pipeline is the kept-span rate bounded, and what is the metric that confirms it is bounded to the configured value?

Quiz