Sampling — head, tail, and what you stop being able to see

The previous lesson worked out that 500 spans a second for a fortnight is roughly 240 GB. Nobody keeps 100% of production traces for long. Sampling is how you reduce that, and it is the one observability decision that permanently removes information — you cannot go back and un-sample last Tuesday.

The two places a decision can be made

Head sampling decides at the start of the trace, in the application, before any work is done. It is cheap: the spans that lose the coin flip are never created, never serialised, never sent. It is also blind: at the moment of the decision, nobody knows whether this request will fail or take nine seconds.

Tail sampling decides at the end, in a collector, after every span of the trace has arrived. It can keep exactly the interesting traces — all the errors, all the slow ones — because by then it knows which those are. It is expensive: every span must be created, sent, and buffered in memory until the decision is made, so you pay the full production and network cost and save only on storage.

Head sampling, done consistently

The trap in head sampling is not the ratio; it is disagreement. If the API samples 10% and the payments service independently samples 10%, then a request crossing both is fully captured only 1% of the time and partially captured 18% of the time. Partial traces are worse than no traces: they show a gap where a service should be, and the obvious reading of that gap is “the service was never called”.

The fix is that the decision must be made once, at the root, and obeyed everywhere else. That is what parentbased_traceidratio does:

services:
  api:
    image: example.com/api:1.4.2
    environment:
      OTEL_SERVICE_NAME: api
      OTEL_TRACES_SAMPLER: parentbased_traceidratio
      OTEL_TRACES_SAMPLER_ARG: '0.1'
  payments:
    image: example.com/payments:2.0.1
    environment:
      OTEL_SERVICE_NAME: payments
      OTEL_TRACES_SAMPLER: parentbased_traceidratio
      OTEL_TRACES_SAMPLER_ARG: '0.1'

parentbased_* means: if this request arrived with a parent context, respect the parent’s decision; only if there is no parent do you apply the ratio. Every service in the request path therefore reaches the same answer.

Tail sampling in the collector

Tail sampling lives in the tail_sampling processor, which is a contrib component — you need otel/opentelemetry-collector-contrib, not the core image.

processors:
  tail_sampling:
    # How long to hold a trace's spans before deciding. Must exceed
    # your slowest realistic request, or slow traces get judged on
    # partial data — which is exactly the population you wanted.
    decision_wait: 30s
    num_traces: 50000
    expected_new_traces_per_sec: 500
    policies:
      - name: keep-errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: keep-slow
        type: latency
        latency:
          threshold_ms: 2000
      - name: baseline-sample
        type: probabilistic
        probabilistic:
          sampling_percentage: 5

Policies are evaluated as an OR: a trace is kept if any policy says keep. So the configuration above means “every errored trace, every trace slower than two seconds, and 5% of everything else” — which is close to the right default for most services.

To require two conditions together, use the and policy type explicitly; listing two policies side by side does not intersect them.

Verifying the sampler is doing what you think

The collector counts what the sampling processor kept and dropped:

$ curl -s http://localhost:8888/metrics | grep -E 'otelcol_processor_tail_sampling_(count_traces_sampled|global_count_traces_sampled)'

otelcol_processor_tail_sampling_count_traces_sampled{policy="keep-errors",sampled="true"} 412
otelcol_processor_tail_sampling_count_traces_sampled{policy="keep-slow",sampled="true"} 187
otelcol_processor_tail_sampling_count_traces_sampled{policy="baseline-sample",sampled="true"} 3104
otelcol_processor_tail_sampling_count_traces_sampled{policy="baseline-sample",sampled="false"} 58996

If keep-errors sits at zero while your error-rate metric is clearly non-zero, the application is not setting span status to ERROR — a very common gap, because many auto-instrumentation libraries record an HTTP 500 as an attribute without marking the span as failed. The policy is fine; the instrumentation is not.

For head sampling there is no such counter, because the spans were never created. You verify head sampling by arithmetic: compare your request-rate metric with the trace count in Tempo over the same window and check the ratio is roughly what you configured.

What you can no longer ask

This is the part that gets skipped, and it is the part that hurts.

Two further consequences worth knowing before you turn sampling on:

• The rare bug is the sampled-away bug. A failure affecting one request in ten thousand, at 5% sampling, appears in your trace store roughly once every 200,000 requests. Keep errors unconditionally and this mostly goes away — which is the strongest argument for tail sampling over head sampling.
• Sampled traces cannot be recovered. Unlike a log level you can turn up during an incident, the spans were never stored. Some teams keep a “debug” sampler override — a header or attribute that forces a trace to be kept — precisely so they have a lever during an incident.

Knowledge check

Where next

That completes the observability part: pillars, tracing, correlation, the collector, the backend, and the volume decision. The next part covers health and failure detection — how the system notices that something is wrong in the first place.