Mitigation and Recovery

The Tempo distributor rate has doubled. The ingester is back-pressuring. The block-builder queue is climbing. The compactor is falling behind. The on-call engineer is paid to stabilise the platform now, not to debate the right sampling policy for the next quarter. The mitigation is a sequence of bounded actions; the recovery is the post-mortem and the baseline update that follows.

This is the trace volume mitigation and recovery. The mitigation is the immediate work to bring the ingest rate back inside the Tempo budget. The recovery is the work to make the next incident cheaper.

What mitigation and recovery means for a trace volume incident

Mitigation is the act of stabilising the platform. Recovery is the act of returning the platform to a known state. The two are not the same. Mitigation accepts that the rate is high and bounds it. Recovery accepts that the rate should be lower and identifies the cause.

A trace volume incident has three mitigation levers, in order of preference:

1. Lower sampling rate. The cheapest lever. The probabilistic_sampler.sampling_percentage is reduced at the edge collector; the kept rate at the gateway falls proportionally.
2. Narrow tail sampling policies. The second lever. The gateway tail_sampling policy list is reviewed; a permissive policy is removed; the kept rate at Tempo falls proportionally.
3. Reject excess at the Tempo distributor. The third lever. The Tempo distributor’s per-receiver rate limit is lowered; pushes above the limit are rejected with ResourceExhausted; the SDK retries with backoff; the rate at the distributor falls.

The three levers are not interchangeable. Lowering sampling rate is the cheapest, but it loses rare traces. Narrowing tail policies is more selective, but it requires understanding which policy was widened. Rejecting at the distributor preserves all sampling decisions upstream but risks losing errors at the edge if the SDK retry budget is exhausted.

Why a sysadmin cares

The mitigation choice determines the cost of the incident to the team whose traces are missing. The wrong mitigation silences the team that is investigating the original incident and saves the platform at the cost of the investigation. The right mitigation bounds the rate while preserving the errors.

The recovery determines the cost of the next incident. A recovery that does not update the baseline (the documented sampling_percentage, the documented policy list, the documented distributor rate limit) repeats the same incident in a quarter when the team has rotated.

How it works

A trace volume incident has three mitigation points, in order of preference and in order of how early they fire in the pipeline.

Application SDK  ---span--->  Edge collector  ---span--->  Gateway collector  ---span--->  Tempo distributor
                       ^                       ^                            ^                            ^
                       |                       |                            |                            |
               lower SDK sampling       lower probabilistic         narrow tail policies       lower distributor
               (OTEL_TRACES_SAMPLER)    (sampling_percentage)        (policies[])               (rate limit)

The cheapest lever is the leftmost; the most expensive is the rightmost. The mitigation order matches: lower the producer-side rate first, narrow the gateway policies second, and only as a last resort lower the distributor rate limit.

Lower sampling at the edge
   |
   +-- kept rate at gateway falls proportionally
   +-- rare traces are lost
   +-- cheap; no SDK change
   |
Narrow tail policies at the gateway
   |
   +-- kept rate at Tempo falls proportionally
   +-- errors and slow traces are still kept (if status_code policy remains)
   +-- requires understanding which policy widened
   |
Reject excess at the distributor
   |
   +-- distributor rate falls
   +-- SDK retries with backoff
   +-- risks losing errors at the edge
   +-- last resort

Under the hood

How to configure it

The mitigation is a sequence of bounded actions. The recovery is a baseline update. The two are configured together.

# STEP 1: lower sampling at the edge collector.
# CONFIGURATION: edit /etc/otelcol/config.yaml and reload.
sed -i 's/sampling_percentage: 5/sampling_percentage: 1/' \
  /etc/otelcol/config.yaml
kubectl rollout restart deploy/edge-collector

# STEP 2: narrow tail policies at the gateway.
# CONFIGURATION: remove the permissive policy.
# Edit the policy list in /etc/otelcol/config.yaml on the
# gateway collector and reload.
kubectl rollout restart deploy/gateway-collector

# STEP 3: reject excess at the distributor.
# CONFIGURATION: lower the distributor rate limit.
# Edit /etc/tempo/tempo.yaml and reload Tempo.
yq -i '.distributor.receivers.otlp.rate_limit = 50000' \
  /etc/tempo/tempo.yaml
kubectl rollout restart deploy/tempo-distributor

The mitigation order is: step 1, then validate, then step 2 if step 1 was not enough, then validate, then step 3 if steps 1 and 2 were not enough. Each step is bounded; the validation between steps is the per-service trace rate in Tempo.

# /etc/otelcol/config.yaml  (edge collector, mitigated)
processors:
  probabilistic_sampler:
    sampling_percentage: 1   # was 5; lowered during the incident
    hash_seed: 42

  batch:
    timeout: 5s
    send_batch_size: 8192

# /etc/tempo/tempo.yaml  (Tempo distributor, mitigated)
distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
      # Rate limit per receiver. Lowered during the incident
      # to bound the maximum ingest rate. The SDK retries with
      # backoff; effective rate falls to this limit.
      rate_limit: 50000   # was 100000; lowered during the incident

How to validate it

When the mitigation is applied, the first read is the per-service trace rate, the Tempo distributor rate, and the ingester block creation rate. All three should fall within minutes of the mitigation.

# READ-ONLY: Tempo distributor span rate after mitigation.
curl -s http://tempo:3200/metrics \
  | grep '^tempo_distributor_spans_received_total'

# READ-ONLY: Tempo ingester block creation rate.
curl -s http://tempo:3200/metrics \
  | grep '^tempo_ingester_blocks_created_total'

# READ-ONLY: per-service rate.
curl -s http://tempo:3200/metrics \
  | grep '^tempo_ingester_traces_per_service'

Expected behaviour after step 1 (lower sampling):

tempo_distributor_spans_received_total 1.92e+10
   # falling; was 4.92e+10 before step 1

tempo_ingester_blocks_created_total 6.4e+05
   # falling; was 1.85e+06 before step 1

If the distributor rate is not falling after step 1, the edge collector reload did not pick up the new sampling_percentage. Read the collector log for the "config loaded" line and confirm the value matches the mitigated one.

# READ-ONLY: confirm the mitigated collector is running.
kubectl logs deploy/edge-collector | grep 'config loaded'

How it can fail

The six failure shapes that account for the great majority of mitigation and recovery errors:

1. Lowered sampling on the wrong service. A team lowers sampling_percentage on the critical-path service to save the platform. The investigation loses its traces; the platform is saved; the post-mortem has no evidence.
2. Mitigation not time-boxed. The mitigation is applied and never reverted. Six months later the team is running at one percent sampling because of an incident that has been closed for half a year.
3. Mitigation reverted before the recovery is complete. The team lowers sampling to bound the rate, then reverts to the original percentage before the baseline update is applied. The next incident hits with the same cause.
4. Distributor rate limit too aggressive. The rate_limit is lowered to a value below the SDK retry budget can sustain. Spans are dropped at the SDK; the error rate at the SDK is high; the platform is silent on the errors the team was trying to keep.
5. Mitigation applied without validating. The team lowers sampling but does not read the per-service trace rate after the reload. The mitigation did not take effect; the rate is unchanged; the team thinks the mitigation worked.
6. Recovery not documented. The mitigation is applied; the incident closes; no baseline entry is created. The next incident hits with the same cause.

How to troubleshoot it

The mitigation order is fixed: lower sampling first, narrow policies second, lower distributor rate limit third.

1. Confirm the symptom. Read tempo_distributor_spans_received_total and tempo_ingester_blocks_created_total. Both should be climbing before the mitigation and falling after.
2. Apply step 1: lower sampling at the edge. Edit the edge collector’s sampling_percentage; reload; validate the per-service rate has fallen.
3. Apply step 2: narrow tail policies. Remove the permissive policy at the gateway; reload; validate the per-policy counter has fallen.
4. Apply step 3: lower distributor rate limit. Lower the distributor’s rate_limit; validate the SDK retry rate has not exceeded its budget.
5. Document the recovery. Update the baseline: the new sampling_percentage, the new policy list, the new rate_limit. The baseline is the source of truth for the next incident.
6. Validate the recovery. Confirm the per-service trace rate has stabilised; confirm the error rate at the SDK is within the retry budget; confirm the compactor has caught up.

Security implications

A trace volume mitigation that lowers sampling on the critical-path service may also lower the rate at which errors are kept. The investigation loses its evidence; the audit trail loses its traces. The fix is a documented mitigation policy that preserves the status_code and latency policies even when the probabilistic policy is lowered.

A distributor rate limit that rejects pushes with ResourceExhausted is observable from outside the trust boundary if the receiver is exposed. An attacker can use the rejection rate to identify when the platform is under pressure. The fix is a documented receiver configuration that does not expose the rejection rate to untrusted clients.

Performance implications

A trace volume mitigation is, by definition, a reduction in the per-request tracing cost. The cheapest lever (lower sampling) reduces the cost proportionally without changing the per-span cost. The most expensive lever (lower distributor rate limit) reduces the cost at the cost of SDK-side retries. The mitigation choice is a trade-off between per-span cost and per-request visibility.

Production guidance

• Document a mitigation policy per service. The policy identifies which services are critical-path (status_code and latency policies preserved) and which are bulk (probabilistic policy lowered first).
• Document a sampling override procedure with a TTL. A time-boxed override is the right pattern; a permanent override is the cause of the next incident.
• Document a baseline entry for every mitigation. The entry identifies the new sampling_percentage, the new policy list, and the new rate_limit. The baseline is the source of truth for the next incident.
• Run otelcol validate on every config before applying. The validate catches syntax errors; the metric catches semantic drift.
• Pin the SDK propagator to W3C TraceContext. B3, Jaeger, and other propagators are not compatible with the sidecar’s default.

Verification

• What is the right mitigation order for a trace volume incident?
• Which mitigation lever is the cheapest, and which is the most expensive?
• What is the most common shape of a mitigation error?
• Where in the OTel Collector pipeline is the mitigation applied, and what is the metric that confirms the mitigation took effect?

Quiz