Head vs Tail Sampling

The checkout team investigates a regression. The fix shipped at 14:00. The post-mortem needs the trace of the one customer whose cart failed at 14:03. The on-call engineer opens Tempo. The backend holds traces sampled at one percent; the trace is not there. The investigation reverts to log correlation and best guessing, and the post-mortem is filed with “unable to reproduce” under evidence.

The sampling decision was made at the producer. The producer did not know the request would fail. The trace was discarded before its outcome was visible. A decision made at the producer is a head sample. A decision deferred until the full trace is visible is a tail sample. The location of the decision is the choice this lesson is about.

What it is

Head sampling makes the keep-or-drop decision at the producer, before the trace is complete. The decision is taken from inputs that exist at request entry: the trace identifier, a hash of a service attribute, a configured rate, a per-route flag. The producer then either ships the whole trace or ships nothing.

Tail sampling makes the keep-or-drop decision at a gateway collector, after the full trace has been buffered. The gateway sees the outcome of every span and applies a policy: keep all errors, keep traces over a latency threshold, sample five percent of the rest. The buffered spans of rejected traces are discarded; the spans of accepted traces are exported.

The two decisions live at different points in the pipeline.

HEAD SAMPLING

  producer (SDK / edge collector)
        |
        |  decision: keep? (hash + rate)
        |
        +-- drop --> [no further work]
        |
        +-- keep --> OTLP --> gateway --> backend

TAIL SAMPLING

  producer (SDK / edge collector)
        |
        |  decision: ship everything (or cheap head sample)
        |
        v
  gateway collector
        |
        |  buffer full trace until decision_wait expires
        |
        |  apply policy: errors / latency / rate
        |
        +-- drop --> spans discarded
        |
        +-- keep --> backend

Why a sysadmin cares

Two failure shapes dominate sampling incidents. Both are about location of decision.

1. The blind incident. A one-percent head sample is fine while the service is healthy: a randomly chosen one percent reproduces the population. The moment a specific failure mode appears, the sample no longer reproduces it. A rare error at one in ten thousand exists in roughly one in a million sampled traces. The platform team has a trace of the error only if the failure happened to be sampled. Most of the time it was not. The investigation ends with no evidence.
2. The buffered gateway that ran out of memory. A team turns on tail sampling to keep all errors. The gateway now receives every span of every trace and buffers until the trace is complete. A traffic spike at five times normal fills the buffer. The collector is OOM-killed; traces are dropped wholesale, including the errors the team turned on tail sampling to keep.

Both failures are predictable. The fix in the first case is tail sampling on a policy that keeps errors. The fix in the second case is capacity for the buffer and a per-collector volume ceiling. The lesson returns to each.

How it works

The mental model is straightforward once the trace is understood as a tree of spans with a shared trace identifier.

A trace identifier is a 128-bit number. The OTel SDK attaches it to every span. The number is generated at the entry point and propagated to every downstream service. Every span of a given trace carries the same identifier.

Head sampling computes a hash of the trace identifier, takes the low-order bits, and compares them against a rate. A five percent sample keeps traces whose low-order bits land in the lowest five percent of the value range. The decision is the same for every service that hashes the same identifier, so a head sample applied at the producer agrees with a head sample applied at an edge collector, provided both use the same hash and the same seed.

Tail sampling makes no decision at the producer. Every span arrives at the gateway. The gateway holds the spans in memory until either the trace is complete (no more spans arrive for the trace identifier within the decision window) or the configured decision_wait expires. At that point the gateway evaluates the policies against the buffered trace and either drops it or exports it.

The decision point changes what information is available. At producer entry, no span has run; the outcome is unknown. At gateway completion, every span has run; the outcome is known. Head sampling cannot keep errors because it cannot see them. Tail sampling can keep errors because it can.

Trace 7a2f... at 0.05 sampling (head)

  producer entry: hash(7a2f...) mod 100 = 17
                  17 not in [0..5) -> drop
                  (the request was an error; the trace is gone)

Trace 8c4e... at full rate into tail sampler (tail)

  producer entry: ship everything
  gateway:        buffer 14 spans for 10s (decision_wait)
  gateway:        status_code = ERROR on root span
  gateway:        policy: status_code ERROR -> keep
                  (the error is exported)

Under the hood

How to configure it

A producer-side head sample uses the probabilistic_sampler processor. The decision is per-trace; consistent hashing on the trace identifier means every service agrees.

# /etc/otelcol/config.yaml  (edge collector, head sampling)

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  # Head sample at 5 percent. The hash_seed is the same across
  # every collector in the fleet; if it differs, the sample
  # population differs.
  probabilistic_sampler:
    sampling_percentage: 5
    hash_seed: 42

  batch:
    timeout: 5s
    send_batch_size: 8192

exporters:
  otlp:
    endpoint: gateway.observability.internal:4317
    tls:
      insecure: false
      ca_file: /etc/ssl/certs/ca-certificates.crt

service:
  pipelines:
    traces:
      receivers:  [otlp]
      processors: [probabilistic_sampler, batch]
      exporters:  [otlp]

A gateway-side tail sample uses the tail_sampling processor. The producer no longer samples; it ships every span. The gateway decides.

# /etc/otelcol/config.yaml  (gateway collector, tail sampling)

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  # Bounds the in-memory trace map. A value too small silently
  # drops traces when the map fills.
  tail_sampling:
    decision_wait: 10s
    num_traces: 50000
    expected_new_traces_per_sec: 1000
    policies:
      - name: keep-errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: keep-slow
        type: latency
        latency:
          threshold_ms: 1000
      - name: keep-baseline
        type: probabilistic
        probabilistic:
          sampling_percentage: 5

  batch:
    timeout: 5s
    send_batch_size: 8192

exporters:
  otlp:
    endpoint: tempo.observability.internal:4317
    sending_queue:
      enabled: true
      queue_size: 5000

service:
  pipelines:
    traces:
      receivers:  [otlp]
      processors: [tail_sampling, batch]
      exporters:  [otlp]

The two configs are not exclusive. A fleet can run an edge collector with probabilistic_sampler to drop ninety-five percent of bulk traffic, and a gateway collector with tail_sampling to apply policy to the five percent that survived. The combined pipeline is the production pattern; the next lesson names it.

How to validate it

Validation confirms the right processor is wired and that the metrics show the expected decisions.

# CONFIGURATION: parse-check the collector config.
otelcol validate --config=/etc/otelcol/config.yaml

# (no output on success; non-zero exit on error)

# READ-ONLY: confirm the binary ships both processors.
otelcol components | grep -E 'probabilistic_sampler|tail_sampling'

  - name: probabilistic_sampler
    type: processor
  - name: tail_sampling
    type: processor

# READ-ONLY: read the probabilistic sampler metrics.
curl -s http://localhost:8888/metrics | grep probabilistic_sampler

otelcol_processor_probabilistic_sampler_count_traces_sampled{policy="probs"} 1287
otelcol_processor_probabilistic_sampler_count_traces_dropped{policy="probs"} 24018

A count_traces_sampled to count_traces_dropped ratio close to the configured percentage is the expected steady state. A ratio wildly different from the configuration means the sampling_percentage is not what was intended or the hash_seed does not match the fleet.

# READ-ONLY: read the tail sampler counters.
curl -s http://localhost:8888/metrics | grep tail_sampling

otelcol_processor_tail_sampling_count_traces_kept{policy="keep-errors"}     42
otelcol_processor_tail_sampling_count_traces_kept{policy="keep-slow"}       17
otelcol_processor_tail_sampling_count_traces_kept{policy="keep-baseline"} 1230
otelcol_processor_tail_sampling_count_traces_dropped                       24018

A trace dropped by the tail sampler never reaches Tempo. If the dropped counter climbs while the kept counters are flat, the trace map (num_traces) may be saturated or the decision_wait may be expiring before all spans arrive.

How it can fail

Five failure modes specific to the head-versus-tail decision.

1. Head sampling on a service that needs to keep errors. A checkout service runs at five percent head sampling. A regression introduces a one-in-ten-thousand error. The sampled population contains roughly one error per two million traces. Symptom: Tempo returns no traces of the error during the incident; the post-mortem has no evidence; the regression repeats.
2. Tail sampling without enough num_traces. A gateway collector is configured with num_traces: 5000 against a real volume of 50 000 in-flight traces. Symptom: the in-memory map fills; the processor logs forced to drop; the dropped traces include the very errors the policy was written to keep.
3. Decision window shorter than the slowest trace. A decision_wait: 1s is set against a backend whose p99 trace duration is 8 seconds. Symptom: spans still arrive after the decision is taken; the kept trace is incomplete; the dropped trace is silent; the policy outcome is decided on a partial trace.
4. Inconsistent hash seed between edge collectors. One edge collector uses hash_seed: 42, another uses hash_seed: 7. Symptom: the same trace identifier is sampled by one edge collector and dropped by another; the trace arrives at the backend with missing spans; the latency picture is incomplete.
5. Tail sampler placed before the policy processor. The tail_sampling processor runs before a resource processor that would add the env label. Symptom: the policies cannot filter on env; the same policy applies to production and staging; staging traffic floods the production backend.
6. Head sample on a service that has no fallback. A service runs at one percent head sampling with no tail sampler downstream. Symptom: a one-off bug that affects fewer than one in a hundred requests produces zero retained traces; the on-call engineer has nothing to investigate.

How to troubleshoot it

A missing-trace investigation asks three questions in order.

1. Is the producer sampling? Check otelcol_processor_probabilistic_sampler_count_traces_sampled on the edge collector. If the counter is climbing, the producer is sampling. If the counter is flat and the receiver is accepting, the producer config is not loaded.
2. Is the gateway tail sampling? Check otelcol_processor_tail_sampling_count_traces_kept per policy. If every policy counter is flat while the receiver is accepting spans, the tail sampler config is not loaded or the service.pipelines.traces.processors does not include tail_sampling.
3. Is the decision window long enough? Compare the slowest trace p99 to decision_wait. If the trace p99 is greater than decision_wait, the decision is being made on incomplete traces. Increase decision_wait to roughly twice the p99 and re-validate.
4. Is the in-memory map filling? Watch otelcol_processor_tail_sampling_traces_dropped_too_early and the trace count metric on the processor. If either climbs, raise num_traces or add additional gateway collectors.

Security implications

• Trace contents. Tail sampling keeps every span until the decision window expires. Sensitive fields (PII, secrets, query parameters) sit in memory for decision_wait plus a buffer. Shorter windows reduce the exposure; longer windows give the policy time to evaluate. The trade-off is between privacy and completeness.
• Hash seed. The hash_seed of a probabilistic_sampler determines which trace identifiers are sampled. If the seed is predictable and an attacker can influence trace identifiers, they can choose whether their traffic is sampled or not. A random seed chosen at deploy time and not reused across distinct security domains is the production pattern.
• PII in span attributes. Tail sampling with a string_attribute policy can keep traces whose spans contain customer identifiers. The export then carries the identifiers into Tempo. The downstream retention and redaction rules apply.

Performance implications

• Head sampling cost. The probabilistic_sampler processor is constant-time per span. It does not buffer; it does not grow with traffic. The cost is the cost of a hash. A producer running at 10 000 spans per second spends a fraction of a CPU core on the decision.
• Tail sampling cost. The tail_sampling processor grows with the in-memory trace map. A rough ceiling is num_traces × average_trace_size_bytes; fifty thousand traces at an average two hundred kilobytes is ten gigabytes of resident memory. The CPU cost is dominated by policy evaluation on every decision. A gateway running tail sampling is a single-purpose host in most production deployments.
• Network. Head sampling reduces network egress at the producer: dropped traces never leave the host. Tail sampling increases network ingress at the gateway: every span arrives before the decision. The trade-off is between edge bandwidth and gateway bandwidth.

Production guidance

• Pick the location per service. Critical-path services (auth, payments, search) get tail sampling at the gateway. Bulk services (telemetry, internal cron, background workers) get head sampling at the producer.
• Pin the binary. The tail_sampling processor is in otelcol-contrib. A core-binary deployment will fail at startup. Run otelcol components and confirm.
• Hash seed consistency. Every edge collector in the fleet must use the same hash_seed. A drifted seed changes which traces are sampled and breaks correlation between services.
• Right-size num_traces. Set num_traces to roughly expected_new_traces_per_sec × decision_wait × 2. The factor of two is for trace-rate variance. Watch otelcol_processor_tail_sampling_traces_dropped_too_early to detect under-sizing.

Verification

You should now be able to answer:

• What is the operational difference between head sampling and tail sampling?
• Why does the producer not have enough information to make a tail-style decision?
• What is the cost trade-off when tail sampling is enabled?
• Which processor (probabilistic_sampler or tail_sampling) lives at the gateway, and why?

Quiz