Break/Fix: Sampling Removes Important Traces

Reported symptoms

A customer was charged twice on a refund. The support ticket has a request identifier, a timestamp, and a twenty-minute window. The on-call engineer opens Tempo, searches { resource.service.name = "checkout-api" && status = error } over the window, and gets three traces. None of them is the one.

This is the fourth time in six weeks. The team enabled tail sampling at the gateway specifically so that this would stop happening, and the change is live.

Four other things are on the ticket, each of which was taken as reassurance.

The tail sampler looks healthy. The gateway reports the keep-errors policy keeping about forty traces an hour, steadily, all day. Every time somebody has checked, the counter was climbing. Nothing is being dropped too early.

The canary lands every time. A platform smoke test injects a synthetic trace every five minutes; it is queryable in Tempo within a minute, without exception, and has been for months. That is treated as end-to-end proof that the pipeline works.

Another service is fine. Error traces for edge-gateway are reliably present for the same windows. Two teams, the same gateway, the same Tempo. That made this look like a checkout-api instrumentation problem, and two sprints went into the SDK integration.

The traces that do arrive are perfect. Full span trees, every service present, no holes where a dependency should be. Nothing about them suggests anything is being discarded.

One observation contradicts all of that and was never followed up. When tail sampling was enabled six weeks ago, Tempo’s ingest volume and object-storage growth did not change. Not by a little - at all. The note in the change ticket reads “the tail sampler is efficient”, and the ticket was closed.

Two weeks ago somebody raised decision_wait from 5s to 30s to see if traces were being decided before they were complete. No effect.

Evidence provided

The cheapest useful thing here is not a collector metric. It is counting the same quantity twice, from two systems that do not share a failure.

$ logcli query --quiet --limit 0 --stats \
'{service="checkout-api"} |= "ERROR" | json | trace_id != ""' \
--from 2026-08-17T13:40:00Z --to 2026-08-17T14:00:00Z \
| wc -l

147

One hundred and forty-seven errors happened, and the application knew the trace identifier of every one of them. Tempo holds three.

That ratio is about two percent, and two percent is a number that should make anyone reach for a sampler. But the gateway insists it is keeping every error it sees.

$ curl -s http://otel-gateway-01.internal.example.com:8888/metrics \
| grep -E 'tail_sampling_(count_traces_kept|traces_dropped_too_early)'

otelcol_processor_tail_sampling_count_traces_kept{policy="keep-errors"} 3841
otelcol_processor_tail_sampling_count_traces_kept{policy="keep-slow"} 902
otelcol_processor_tail_sampling_count_traces_kept{policy="keep-baseline"} 71204
otelcol_processor_tail_sampling_traces_dropped_too_early 0

Nothing in the collector fleet is head sampling either.

$ grep -c probabilistic_sampler /etc/otelcol/config.yaml

0

So the decision is being made before the pipeline starts. The producer is the only place left.

$ ssh checkout-01.internal.example.com \
'systemctl show checkout-api --property=Environment' \
| tr ' ' '\n' | grep OTEL_TRACES

OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.02

$ ssh edge-gw-01.internal.example.com \
'systemctl show edge-gateway --property=Environment' \
| tr ' ' '\n' | grep OTEL_TRACES

OTEL_TRACES_SAMPLER=always_on

Work the evidence before reading on

The tail sampler keeps every error it sees. One hundred and forty-seven errors occurred. Three traces exist. All three statements are true.

1. A head sampler decides not to record. What does the gateway receive for a trace the SDK declined to record, and what could any downstream policy do about it?
2. keep-errors is keeping forty traces an hour and that looks healthy. Healthy compared to what? Name the denominator the counter does not contain.
3. /v2/refunds served 147 errors in twenty minutes and produced no trace; the two busiest routes produce error traces regularly. Both are sampled at the same rate. Why does one look broken and the other does not?
4. The sampler is parentbased_traceidratio. Given that, predict whether the traces that do arrive should be complete or full of holes, and reconcile your answer with the symptom that made everyone rule out sampling.
5. Ingest volume did not move when tail sampling was enabled. If the producers had been shipping everything, what should that graph have done - and what does a flat line tell you?

Before continuing: state what would happen to the gateway if you set OTEL_TRACES_SAMPLER=always_on on every service this afternoon.

Root cause

1. A head decision is not a filter, it is an absence

parentbased_traceidratio with an argument of 0.02 hashes the trace identifier at the entry point and, for ninety-eight percent of requests, decides not to sample. That is not a decision to record a trace and discard it later. The spans are not recorded, nothing is exported, and nothing arrives anywhere.

The tail sampler is downstream of that. It cannot apply a policy to data it was never sent, and there is no configuration of the gateway that changes this. Six weeks of tail-sampling work were applied to two percent of the problem, and applied correctly.

2. A keep-rate has no denominator in it

keep-errors at forty an hour was the strongest evidence the team had that the pipeline was doing its job, and it is not evidence of anything. The counter reports how many of the traces that reached the gateway matched the error policy. It has no opinion about how many errors the service served, because it never saw them.

Set beside the log count, forty an hour is about two percent of reality. The same number, unqualified, reads as success. This is the single most reusable idea in the incident: a ratio computed entirely inside the thing you are testing tells you that the thing is internally consistent, not that it is right.

3. Rarity made it look endpoint-specific

Every row is sampled identically. The busy routes produce enough errors that two percent still yields a visible handful, so the pipeline looks fine when anyone spot-checks it. The low-volume route yields an expected three and an observed zero, which is unremarkable for a sample of that size and looks exactly like a route whose instrumentation is broken.

The routes that matter most in an investigation are usually the rare ones. Sampling degrades them first and degrades them invisibly.

4. Every check the team ran was insensitive to the cause

The canary sets always_on for its own requests, so it never passes through the sampler that production traffic passes through. It proves the exporter, the network, the gateway and Tempo all work. It says nothing about the decision, which is the part that was broken.

The flat ingest volume was the loudest available evidence and was read backwards. If the producers had begun shipping every span for the gateway to decide on, ingest should have jumped by roughly fifty times on the cutover date. A graph that does not move after a change of that magnitude means the change did not reach the producers.

The decision_wait increase addresses traces being decided before they are complete. That was not happening - traces_dropped_too_early is zero - so the change had no effect, and it quietly multiplied the gateway’s buffer requirement by six.

Resolution

1. Say plainly, in the channel, that the tail sampler is healthy and has been throughout. Six weeks of work went into the wrong component because a green counter was read as a green pipeline, and the next person will repeat it unless the correction is explicit.
2. Do the buffer arithmetic before changing anything. num_traces needs to be about expected new traces per second multiplied by decision_wait, with headroom for variance, and the memory that implies is roughly that count multiplied by the average trace size. Get both numbers at the target ratio, not at the current one.
3. Bring decision_wait back toward the measured p99 trace duration. It was raised to 30s chasing a cause that was not there, traces_dropped_too_early proves the original value was adequate, and every second of it is now buffer you have to pay for at fifty times the volume.
4. Raise the ratio on the one service under investigation rather than fleet-wide, in stages - 0.02, then 0.1, then 0.5, then 1.0 - holding at each step long enough to read gateway memory, refused spans, dropped-too-early traces and exporter queue depth through at least one traffic peak.
5. Treat stopping short as a legitimate outcome. If the gateway cannot hold full rate today, holding at an intermediate ratio with a named owner and a review date is a better answer than a heroic cutover, and it is a much better answer than an out-of-memory kill.
6. Add gateway capacity horizontally rather than buying headroom by trimming decision_wait below the p99. One of those choices costs money and the other silently corrupts the decision.
7. Move the sampler setting out of the base-image default into declared per-service configuration in the same change, so that the fleet-wide claim "we use tail sampling" becomes something a review can check.
8. Correct the smoke test so it traverses the production sampler path instead of setting always_on for itself. A canary that exempts itself from the mechanism under test is worse than no canary, because it produces a green signal.
9. Tell the investigating team that the 147 errors from the incident window are gone permanently. A trace that was never recorded does not exist in any buffer, backend or backup, and the investigation has to proceed from logs.

Verification

1. The producer is configured as intended. Read the sampler settings back from a running host rather than from the deployment manifest; the whole incident is the difference between those two.
2. The independent counts agree. Traces matching status = error in Tempo must now land within a few per cent of the count of error log lines carrying a trace identifier over the same window. This is the comparison that exposed the fault, and it is the one that confirms the fix.
3. Ingest moved. otelcol_receiver_accepted_spans at the gateway has stepped up by roughly the factor the ratio change implies. If the graph is flat, the change did not reach the producers and nothing else you measure means anything.
4. The gateway is not failing quietly under the new load. traces_dropped_too_early stays at zero and refused spans stay at zero through a full traffic peak, not merely in the ten minutes after the change.
5. The rare case works, tested directly. Emit twenty errors on the low-traffic route through the ordinary request path and require all twenty to be queryable in Tempo. Nineteen is a failure: the entire point is the one you did not get.
6. The baseline policy still discards routine traffic. Confirm the tail sampler is not now keeping everything, or the storage bill becomes the next incident in a fortnight.
7. The detector can fail. In staging, set the sampler back to 0.02 and confirm the produced-versus-accepted alert fires. A guard that has only ever been green has never been tested.

Prevention

• Publish one ratio: spans produced against spans accepted at the gateway. If any head sampler sits anywhere in the path, that number is not one. It is the only signal capable of seeing a decision made before the pipeline begins, because a trace that was never recorded emits nothing at all.
• Never read a keep-rate without its denominator. Forty errors an hour kept is meaningless until it is set beside the errors the service actually served. The log-derived count is independent, already collected, and costs one query.
• Make sampler configuration declared and per-service. A base-image default is not configuration; it is an assumption that survives every review because it appears in no diff.
• Canaries must traverse the mechanism under test. A synthetic trace that sets always_on validates transport and storage and asserts nothing at all about the sampling decision.
• A change to decision_wait is a change to the memory budget. Put the buffer arithmetic in the same review, every time.
• After a sampling cutover, the acceptance test is a measured step in gateway ingest. A flat graph after a change of that magnitude is a finding, not a compliment.
• Run a rare-error canary on a low-volume route and alert when its trace is not queryable within a stated interval. The failure this class produces is the absence of something nobody is looking for, and only a standing test looks for it.