Sampling Rate

The capacity review meeting shows the observability backend is holding 4.2 trillion spans across ninety days of retention. The backend is over budget. A team proposes cutting the sampling rate from five percent to one percent across the board.

The proposal will reduce spans by roughly a factor of five. It will also drop ninety-five percent of the rare errors the team retained last quarter. The post-mortem from the regression that ran last month referenced a trace captured at five percent; at one percent the regression would not have been in the sample at all.

The rate is a number. The number has consequences on both sides. The right number is a deliberate choice, not a default.

What it is

The sampling rate is the fraction of traces (or spans) the sampler keeps. For a probabilistic sampler, it is a percentage applied to the trace identifier. A rate of one percent keeps one trace in a hundred; a rate of five percent keeps five in a hundred; a rate of one hundred percent keeps every trace.

The rate and the request volume together determine the retained sample count.

retained_traces_per_second
    = request_traces_per_second  x  rate

A service at 5 000 traces per second sampled at one percent ships fifty retained traces per second. The same service at five percent ships 250. The same service at one hundred percent ships 5 000, which is the full traffic.

The math is not the cost. The cost is the cost of those retained traces multiplied by their average span count multiplied by their retention. The math is the bottleneck number to plan against.

Why a sysadmin cares

Two failure shapes drive most sampling-rate incidents. Both come from setting the rate by gut feel rather than by traffic mathematics.

1. The cost blowout. A team instruments a new service at one hundred percent because “it’s only a small service.” The service is small in count but generates 200 spans per trace and runs at 800 traces per second. The backend ingests 160 000 spans per second. At thirty days of retention the backend holds 414 trillion spans. The cost review the next quarter names this service as the largest single contributor.
2. The blind regression. A team reduces the rate from ten percent to one percent to control cost. A regression that affects one request in a thousand produces zero retained traces: one in a hundred thousand traces is sampled, and the one-in-a-thousand error rate inside that is one in a hundred million. The team discovers the regression only when a user reports it.

The rate is the knob that controls both sides. A high rate gives more traces and costs more; a low rate costs less and keeps less. The discipline is to choose by traffic and investigation value, not by default.

How it works

The probabilistic sampler applies the rate to a hash of the trace identifier.

trace_identifier    = 7a2f0e3d...  (16 bytes / 128 bits)
hash(traced_id,
     hash_seed)      = 0x8c4e1f...
hash mod 10000       = 1932
1932 / 10000         = 0.1932

sampling_percentage  = 5  (i.e. 0.05)
0.1932 < 0.05 ?      = false  -> drop

sampling_percentage  = 25  (i.e. 0.25)
0.1932 < 0.25 ?      = true   -> keep

The hash is consistent. Given the same trace identifier and the same seed, every collector in the fleet computes the same hash and reaches the same decision. This is what makes head sampling useful in a distributed system: every service that sees the same trace identifier decides the same way.

The relationship between rate and retained volume is linear.

service:    checkout
rate:       5 %
volume:     5 000 traces / second
retained:   5 000 x 0.05 = 250 traces / second
            21.6 M traces / day
            648 M traces / month

service:    background-scheduler
rate:       0.1 %
volume:     12 000 traces / second
retained:   12 000 x 0.001 = 12 traces / second
            1.04 M traces / day
            31 M traces / month

service:    auth
rate:       100 %
volume:     1 200 traces / second
retained:   1 200 x 1.0 = 1 200 traces / second
            104 M traces / day
            3.1 B traces / month

The auth service at one hundred percent ships every trace; the scheduler at 0.1 percent ships one in a thousand. The retained volume from auth is one hundred times the scheduler’s retained volume despite auth having one tenth the request volume. Rate is the lever; volume is the input.

The retention multiplies the math again. The same 250 traces per second kept for ninety days is 1.94 billion retained traces; at an average span count of fifty and an average span size of two kilobytes, that is 194 TB of indexed spans.

Under the hood

How to configure it

The probabilistic_sampler processor is a single-line decision in the OTel Collector.

# /etc/otelcol/config.yaml  (edge collector, per-service rate)

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  # Bulk service: keep 1 percent.
  probabilistic_sampler:
    sampling_percentage: 1
    hash_seed: 42

  batch:
    timeout: 5s
    send_batch_size: 8192

exporters:
  otlp:
    endpoint: gateway.observability.internal:4317
    tls:
      insecure: false
      ca_file: /etc/ssl/certs/ca-certificates.crt

service:
  pipelines:
    traces:
      receivers:  [otlp]
      processors: [probabilistic_sampler, batch]
      exporters:  [otlp]

A per-service rate uses the routing connector to send each service’s traces through its own pipeline.

# /etc/otelcol/config.yaml  (per-service rate via routing connector)

connectors:
  routing:
    default_pipelines: [traces/bulk]
    error_mode: ignore
    table:
      - context: resource
        statement: route() where attributes["service.name"] == "checkout"
        pipelines: [traces/critical]
      - context: resource
        statement: route() where attributes["service.name"] == "auth"
        pipelines: [traces/critical]
      - context: resource
        statement: route() where attributes["service.name"] == "telemetry-collector"
        pipelines: [traces/bulk]

service:
  pipelines:
    traces/critical:
      receivers:  [otlp]
      processors: [probabilistic_sampler/critical, batch]
      exporters:  [otlp/gateway]
    traces/bulk:
      receivers:  [otlp]
      processors: [probabilistic_sampler/bulk, batch]
      exporters:  [otlp/gateway]

processors:
  probabilistic_sampler/critical:
    sampling_percentage: 25
    hash_seed: 42
  probabilistic_sampler/bulk:
    sampling_percentage: 1
    hash_seed: 42

The two pipelines are evaluated independently. The critical pipeline keeps twenty-five percent of checkout and auth traces; the bulk pipeline keeps one percent of every other service. The retained population per service is the service’s request volume times its rate.

How to validate it

Validation confirms the rate produces the expected retained volume and that the rate is the same across the fleet.

# CONFIGURATION: parse-check the collector config.
otelcol validate --config=/etc/otelcol/config.yaml

# (no output on success; non-zero exit on error)

# READ-ONLY: read the per-processor counters.
curl -s http://localhost:8888/metrics | grep probabilistic_sampler

otelcol_processor_probabilistic_sampler_count_traces_sampled{policy="critical"} 312
otelcol_processor_probabilistic_sampler_count_traces_dropped{policy="critical"}  938
otelcol_processor_probabilistic_sampler_count_traces_sampled{policy="bulk"}      87
otelcol_processor_probabilistic_sampler_count_traces_dropped{policy="bulk"}     8613

For the critical pipeline, the ratio sampled / (sampled + dropped) is 312 / 1250 = 0.2496, matching the configured twenty-five percent within sampling noise. For the bulk pipeline, 87 / 8700 = 0.01, matching the configured one percent exactly.

# READ-ONLY: confirm the hash_seed is identical across the fleet.
for h in edge-1 edge-2 edge-3; do
  ssh "$h.observability.internal" \
    "grep hash_seed /etc/otelcol/config.yaml | sort -u"
done

        sampling_percentage: 25
        hash_seed: 42
        sampling_percentage: 25
        hash_seed: 42
        sampling_percentage: 25
        hash_seed: 42

A drift in hash_seed between hosts breaks the consistency property. Different hosts sample different trace identifiers; some traces arrive at the backend with missing spans; the correlation across services is broken.

How it can fail

Five failure modes specific to the sampling rate.

1. A uniform rate across heterogeneous services. Every service runs at five percent. The bulk telemetry service at 12 000 traces per second ships 600 retained traces per second; the auth service at 1 200 traces per second ships 60. The bulk service consumes ten times the budget for one tenth the investigation value. Symptom: cost review shows the bulk service as the largest contributor; rate uniform across the fleet was the wrong default.
2. A rate set by guess rather than measurement. A team sets one percent on a service without measuring its volume. Symptom: at one percent the service ships fewer retained traces than expected; investigation during an incident has no traces; the team cannot reproduce the failure mode.
3. A rate held constant as the service grows. A service grew from 200 to 4 000 traces per second over six months. The sampling rate stayed at one percent. Symptom: the backend ingest doubled without a corresponding budget increase; the cost review names this service as the primary contributor.
4. A rate set against the request rate, not the trace rate. A service generates five traces per request (one parent, four children). The team configures one percent based on request rate rather than trace rate. Symptom: the retained sample is five times larger than expected; the cost exceeds the budget.
5. A rate changed without informing the consumers. The platform team drops the rate from five percent to one percent to control cost. Symptom: dashboards that show “traces per service” appear to show a five-times reduction in traffic; the on-call engineer cannot tell whether the service is failing or the sampler changed.
6. A rate that breaks the correlation invariant. A subset of edge collectors runs at ten percent, the rest at one percent. Symptom: traces from the ten-percent subset arrive with full span counts; traces from the one-percent subset arrive with sparse spans; the backend cannot reconstruct a complete picture for any given trace identifier.

How to troubleshoot it

A wrong-rate investigation asks four questions in order.

1. Is the rate what the config says? Re-read the probabilistic_sampler block. A common bug is the field sampling_percentage vs sampling_probability; the processor takes a percentage (5 for five percent), not a probability (0.05). Five in the config means five percent, not five one-thousandths.
2. Is the processor actually running? Confirm otelcol_processor_probabilistic_sampler_count_traces_sampled is non-zero. If it is zero, the processor is not in the pipeline. Check service.pipelines.traces.processors.
3. What is the actual request rate? Read otelcol_receiver_accepted_spans and divide by the average span count for the service. A service at ten percent that shows 50 000 spans per second at fifty spans per trace has 1 000 traces per second, not the 5 000 the rate math was based on.
4. Is the rate matched across the fleet? Diff /etc/otelcol/config.yaml across edge collectors. A drift in sampling_percentage or hash_seed produces inconsistent sampling.

Security implications

• Trace identifier as a side channel. The hash of the trace identifier with a fixed seed is deterministic. An attacker who can observe which traces are sampled can infer the seed and predict which of their own traces will be sampled. The practical risk is low; the mitigation is a randomly chosen seed per environment.
• Sensitive fields in sampled spans. A probabilistic sampler decides on the trace identifier, not on span contents. If a service emits a sensitive field (PII, secrets), the field is retained or dropped with the same probability as the trace. Redaction at the SDK or the collector is the production pattern.
• Rate as a budget for attackers. A high rate makes it easy for an attacker to confirm whether a malicious request was traced. A low rate makes confirmation probabilistic. The right rate is a function of detection need, not just budget.

Performance implications

• Producer CPU. The probabilistic_sampler processor spends a hash per trace. The cost is a fraction of a CPU core at 10 000 spans per second; it is not a bottleneck.
• Backend ingest. The retained sample count is the input to backend ingest. A five percent sample of a five-trace-per- request service at 1 000 requests per second is 250 retained traces per second, which is 21.6 million traces per day. The backend ingest is the line item in the cost review.
• Backend storage. The retained traces per day times retention is the storage footprint. Ninety days of 21.6 million traces per day is 1.94 billion retained traces; the storage cost scales with that volume.

Production guidance

• Set the rate per service. Use a routing connector or per- host collector configs to apply different rates to different services. A fleet-wide uniform rate is a smell.
• Measure before choosing. Read the request rate and the trace rate from the receiver counters; set the rate so the retained volume fits the budget; revisit when the traffic grows.
• Match hash_seed across the fleet. A drifted seed breaks the consistency invariant. Use a configuration management tool to enforce the seed.
• Use sampling_percentage (whole numbers), not fractions. The processor takes 5 for five percent. A fraction in the config is the most common typo.

Verification

You should now be able to answer:

• What is the formula that turns request volume and sampling rate into retained traces per second?
• Why does a uniform rate across heterogeneous services produce a wrong cost distribution?
• What is the relationship between hash_seed and the correlation invariant across a fleet?
• What is the trade-off between rate and investigation value?

Quiz