Sampling Strategy Per Service

A platform team owns thirty services. The trace volume is 38 000 spans per second at peak. The budget review shows the backend is the second-largest line item in the observability spend. The team has three weeks to cut the cost by forty percent without losing the rare-error coverage the on-call engineers rely on.

The team tries four strategies. The first is a fleet-wide one percent head sample; it cuts cost by ninety-five percent and loses every rare error. The second is tail sampling at every gateway; it preserves the rare errors and the gateway OOMs at peak. The third is per-service rates chosen by traffic; it cuts cost by sixty percent and keeps the rare errors on the critical path. The fourth layers head and tail sampling; it cuts cost by seventy percent and keeps the rare errors on every service that matters.

The lesson is the fourth strategy: the layered, per-service, per-environment plan.

What it is

A sampling strategy is the plan that decides which traces are kept, where the decision is made, and at what rate. The plan is layered: a head sampler at the producer drops the bulk volume, a tail sampler at the gateway applies policy to the survivors, and the choice of layer and rate is per-service.

producer SDK (or edge collector)
        |
        v
probabilistic_sampler (head)
        |
        | keep = small fraction of full rate
        |
        v
OTLP to gateway
        |
        v
tail_sampling (policy)
        |
        | keep = errors / slow / probabilistic baseline
        |
        v
backend (Tempo)

The head sampler is the volume control. The tail sampler is the policy control. A team that wants to cut cost raises the head sampler rate down; a team that wants to keep rare traces adds tail sampler policies at the gateway.

The per-service dimension is the lever that makes the plan affordable. Critical-path services (auth, payments, checkout) earn higher rates; bulk services (telemetry collectors, internal schedulers) earn lower rates. The per-environment dimension adds the staging-versus-production split.

Why a sysadmin cares

Three operational consequences follow from a layered strategy.

1. The budget is bounded. Head sampling at the producer drops the bulk of the volume before the spans leave the host. The gateway ingest is a fraction of the fleet volume; the memory budget for tail sampling is a fraction of the fleet memory. The cost review sees the backend ingest decline; the gateway sizing becomes a manageable calculation.
2. The rare traces are kept. Tail sampling at the gateway keeps every trace whose policy matches, regardless of the head sampler rate. A critical-path service at five percent head sampling with a tail-sampled error policy at the gateway keeps every error trace from the five percent; the on-call engineer has the trace at 03:00.
3. The plan is portable across environments. Production and staging share the same pipeline structure; only the rates differ. A staging environment at lower rates costs less while preserving the same sampling semantics. A staging environment at one hundred percent validates the policy without obscuring the production budget.

How it works

The OTel Collector routes traces by service.name to different pipelines using the routing connector. Each pipeline has its own probabilistic_sampler and (where it matters) its own tail_sampling.

The three pipeline classes that make up the strategy:

• Critical pipelines. Services whose rare traces must be retained (auth, payments, checkout). High head rate (ten to twenty-five percent) and tail sampling with error, latency, and baseline policies.
• Bulk pipelines. Services whose traces are aggregated into latency distributions and throughput metrics (telemetry collectors, internal schedulers, batch jobs). Low head rate (one percent or less) and no tail sampling.
• Internal pipelines. Services that exist primarily for observability of the platform itself (the OTel Collector itself, the Tempo ingester). Either head sampling at very low rate (0.1 percent) or no sampling at all, depending on the value of the trace.

The per-environment split is a separate decision layered on top. Production runs the policy at the rates described above. Staging runs the same pipeline structure but at lower rates; the goal in staging is to validate the policy without generating production-scale trace volume.

production (env=prod)
  checkout:    head 25 %, tail {error, slow, baseline 5 %}
  auth:        head 25 %, tail {error, slow, baseline 5 %}
  telemetry:   head  1 %, no tail
  scheduler:   head  1 %, no tail

staging (env=staging)
  checkout:    head  5 %, tail {error, slow, baseline 5 %}
  auth:        head  5 %, tail {error, slow, baseline 5 %}
  telemetry:   head  0.1 %, no tail
  scheduler:   head  0.1 %, no tail

The rates are deliberately lower in staging; the goal there is to exercise the pipeline, not to retain traces for investigation. A staging environment at production rates is a staging environment that costs as much as production.

Under the hood

How to configure it

A layered, per-service strategy via the routing connector.

# /etc/otelcol/config.yaml  (gateway collector)

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

connectors:
  routing:
    default_pipelines: [traces/bulk]
    error_mode: ignore
    table:
      - context: resource
        statement: route() where attributes["service.name"] == "checkout"
        pipelines: [traces/critical]
      - context: resource
        statement: route() where attributes["service.name"] == "auth"
        pipelines: [traces/critical]
      - context: resource
        statement: route() where attributes["service.name"] == "payments"
        pipelines: [traces/critical]
      - context: resource
        statement: route() where attributes["service.name"] == "scheduler"
        pipelines: [traces/bulk]
      - context: resource
        statement: route() where attributes["service.name"] == "telemetry-collector"
        pipelines: [traces/bulk]

processors:
  probabilistic_sampler/critical:
    sampling_percentage: 25
    hash_seed: 42
  probabilistic_sampler/bulk:
    sampling_percentage: 1
    hash_seed: 42

  tail_sampling/critical:
    decision_wait: 10s
    num_traces: 20000
    expected_new_traces_per_sec: 500
    policies:
      - name: keep-errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: keep-slow
        type: latency
        latency:
          threshold_ms: 1000
      - name: keep-baseline
        type: probabilistic
        probabilistic:
          sampling_percentage: 5

  batch:
    timeout: 5s
    send_batch_size: 8192

exporters:
  otlp/tempo:
    endpoint: tempo.observability.internal:4317
    tls:
      insecure: false
      ca_file: /etc/ssl/certs/ca-certificates.crt

service:
  pipelines:
    # Stage 1: receive, route by service.name.
    traces:
      receivers: [otlp]
      exporters: [routing]

    # Stage 2a: critical services get head sample then tail sample.
    traces/critical:
      receivers: [routing]
      processors: [probabilistic_sampler/critical, tail_sampling/critical, batch]
      exporters: [otlp/tempo]

    # Stage 2b: bulk services get head sample only.
    traces/bulk:
      receivers: [routing]
      processors: [probabilistic_sampler/bulk, batch]
      exporters: [otlp/tempo]

The pipeline graph is two stages. Stage one routes by service.name. Stage two applies the per-service strategy. The bulk pipeline ends with the exporter; the critical pipeline applies the tail sampling policy before exporting.

A per-environment split uses a second routing expression on the deployment.environment attribute.

connectors:
  routing:
    default_pipelines: [traces/bulk]
    error_mode: ignore
    table:
      # Critical services in staging use lower head rates.
      - context: resource
        statement: route() where attributes["service.name"] == "checkout"
          and attributes["deployment.environment"] == "staging"
        pipelines: [traces/critical-staging]
      - context: resource
        statement: route() where attributes["service.name"] == "checkout"
          and attributes["deployment.environment"] == "prod"
        pipelines: [traces/critical-prod]

The two traces/critical-* pipelines have the same tail_sampling/critical configuration but different probabilistic_sampler/critical-* percentages. The production pipeline keeps twenty-five percent; the staging pipeline keeps five percent.

How to validate it

Validation confirms the routing is sending each service to the right pipeline and that the rates are correct.

# CONFIGURATION: parse-check.
otelcol validate --config=/etc/otelcol/config.yaml

# READ-ONLY: read the routing connector counters.
curl -s http://localhost:8888/metrics | grep connector_routing

otelcol_connector_routing_spans_routed{destination="traces/critical"} 14532
otelcol_connector_routing_spans_routed{destination="traces/bulk"}      98412

The counter for traces/critical matches the expected proportion of critical services. A flat counter against traces/critical while service.name == "checkout" is firing means the routing expression is not matching; the trace is falling through to default_pipelines.

# READ-ONLY: read the per-pipeline sampling counters.
curl -s http://localhost:8888/metrics | grep probabilistic_sampler

otelcol_processor_probabilistic_sampler_count_traces_sampled{policy="critical"} 3633
otelcol_processor_probabilistic_sampler_count_traces_dropped{policy="critical"} 10899
otelcol_processor_probabilistic_sampler_count_traces_sampled{policy="bulk"}      984
otelcol_processor_probabilistic_sampler_count_traces_dropped{policy="bulk"}    97428

For the critical pipeline, the ratio sampled / (sampled + dropped) is 3633 / 14532 = 0.25, matching the configured twenty-five percent. For the bulk pipeline, 984 / 98412 = 0.01, matching the configured one percent.

# READ-ONLY: read the tail sampler counters.
curl -s http://localhost:8888/metrics | grep tail_sampling

otelcol_processor_tail_sampling_count_traces_kept{policy="keep-errors"}     42
otelcol_processor_tail_sampling_count_traces_kept{policy="keep-slow"}       17
otelcol_processor_tail_sampling_count_traces_kept{policy="keep-baseline"}  182

The tail sampler is consuming the output of the critical head sampler; the kept volume is a fraction of the head rate. The discipline is to watch the kept counter for each policy and to confirm the policy is matching the right traces.

How it can fail

Six failure modes specific to a layered, per-service strategy.

1. The uniform rate across the fleet. Every service runs at five percent head sampling, no tail sampling. Symptom: the bulk service consumes ten times the budget for one tenth the investigation value; rare errors are sampled away at one in twenty.
2. The routing expression not matching. A team adds a new service (recommendations) but forgets to add a routing rule. Symptom: the trace falls through to default_pipelines (the bulk pipeline); the new service is sampled at one percent; rare errors are lost.
3. The critical pipeline and the bulk pipeline using the same processor name. The team defines probabilistic_sampler (no per-pipeline suffix) and references it from both traces/critical and traces/bulk. Symptom: the collector refuses to start with “duplicate processor name”; the fix is to suffix every per-pipeline processor (probabilistic_sampler/critical).
4. The staging environment running at production rates. A team deploys the same config to staging and production. Symptom: staging trace volume equals production trace volume; the staging cost review surprises the platform team; the staging budget overruns.
5. The tail sampler buffering for staging traffic as well. The team runs the same gateway for staging and production. Symptom: the staging traffic fills the same num_traces budget as production; rare production errors are evicted under staging load; the production rare-error coverage is reduced.
6. The hash_seed drifting between critical and bulk pipelines. The two pipelines use different hash_seed values. Symptom: a trace from checkout (routed to critical) and a trace from scheduler (routed to bulk) sample at different rates with no cross-service correlation; the bulk pipeline’s sampled population is unrelated to the critical pipeline’s.

How to troubleshoot it

A wrong-strategy investigation asks five questions in order.

1. Is the routing expression matching? Read otelcol_connector_routing_spans_routed against each pipeline. A flat counter against a pipeline whose service is firing means the expression is not matching.
2. Is each service landing in the right pipeline? For a known service, inspect the routing destination in the logs. The collector logs each routing decision at debug level. Confirm the destination matches the plan.
3. Is the per-pipeline rate what the config says? Read the probabilistic_sampler_count_traces_sampled and ..._dropped per pipeline. The ratio should match the configured rate.
4. Is the tail sampler buffering the right population? Read otelcol_processor_tail_sampling_count_traces_kept per policy. The kept volume should be a fraction of the head-sampled volume of the critical pipeline; if the kept volume is suspiciously large, the tail sampler may be running on bulk traffic too.
5. Is the per-environment split in effect? Read attributes["deployment.environment"] from a sample trace in each pipeline. A staging trace in the production pipeline means the staging routing expression is wrong.

Security implications

• Sensitive attributes in routing expressions. A routing expression on service.name is innocuous. A routing expression on a sensitive attribute (PII, customer ID) exposes the attribute to the routing decision and to the collector logs at debug level. Prefer the innocuous attribute.
• Pipeline isolation. The critical pipeline and the bulk pipeline share the gateway collector but use different exporters. A misconfigured exporter on the critical pipeline can leak critical-path traces to the wrong backend. The discipline is to verify each pipeline’s exporter against the plan, not against the others.
• Per-tenant separation. A multi-tenant gateway that routes by tenant ID and uses one tail_sampling policy for every tenant is a tenant isolation failure: a noisy tenant fills the map and evicts the quiet tenant’s rare traces. Per-tenant gateways (or per-tenant policies) are the production pattern.

Performance implications

• Routing CPU. The routing connector evaluates the OTTL expression on every span that arrives. A long expression list or an expensive expression (regex against a large attribute) is a per-span CPU cost. The discipline is to keep the routing table short and the expressions simple.
• Pipeline parallelism. The per-service pipelines run as separate processor chains. A bulk pipeline with cheap processors can run in parallel with a critical pipeline with expensive processors. The routing connector fans out the trace to both; the bulk pipeline does not wait for the critical pipeline.
• Memory. The memory budget is dominated by the tail sampling in-memory map. The critical pipeline owns that budget; the bulk pipeline is stateless.

Production guidance

• Use the routing connector. The per-service strategy is impractical without it. A fleet-wide uniform rate is the wrong default.
• Suffix per-pipeline processor names. Every per-pipeline processor must have a unique name (probabilistic_sampler/critical, probabilistic_sampler/bulk). The collector refuses to start with duplicate names.
• Match the hash_seed across pipelines. A drifted seed breaks the correlation invariant within a single trace that crosses pipelines.
• Stage in staging first. A new sampling strategy should run in staging at the planned production rates before production. A staging validation that confirms the policy matches a forced-error trace is the gate for production.
• Make default_pipelines deliberately empty. An unmatched service then produces a routing error that surfaces in the logs; the team discovers the new service before the on-call engineer does.

Verification

You should now be able to answer:

• Why is a layered strategy (head at producer, tail at gateway) cheaper than tail sampling alone for the same rare- error coverage?
• What is the role of the routing connector in a per-service strategy?
• Why must every per-pipeline processor be suffixed?
• What is the right place to set a different sampling rate for staging?

Quiz