Re-tuning Sampling

The cost review meeting opens with the observability line item. The backend ingest grew forty-three percent over the quarter. The team that owns auth deployed a new microservice without notifying the platform team; the new service ships 9 000 spans per second at one hundred percent. The cost review names the platform team as the budget owner and the auth team as the budget spender.

The platform team has not looked at the sampling strategy for eight months. The traffic has grown; the rates have not. The quarterly review that was supposed to happen did not happen.

The lesson is the process that catches this before the cost review names it.

What it is

Re-tuning sampling is the iterative discipline of observing the current strategy’s outcome, deciding on a change, applying the change in version control, validating the change in staging, rolling out to production, and documenting the outcome. The discipline runs on a cadence (typically monthly or quarterly) and on triggers (a cost spike, an incident with no trace, a service whose traffic has grown materially).

The process has five steps.

observe   -->  decide   -->  change   -->  validate   -->  document
  |             |             |              |               |
metrics,    policy       collector      staging run,      change log,
incidents,  decision     config in      force-error      rationale,
capacity    (raise rate, git, PR,      test,            review date,
            lower rate,  reviewed       metrics match    next review
            add policy)                expected          date

The cadence is the rhythm. The triggers are the interruptions. Both matter. A cadence without triggers misses the urgent changes; triggers without a cadence miss the gradual drift.

Why a sysadmin cares

Sampling strategy is not a one-time decision. Traffic grows; services change; the budget moves; the rare-error coverage drifts. A team that sets the strategy once and never revisits it will see the strategy become wrong within months.

Three failure shapes dominate re-tuning incidents.

1. The cost surprise. The backend ingest has grown linearly with traffic over six months. The strategy was never revisited. The cost review names the platform team as the budget owner. The team is now in the position of cutting rates reactively, under pressure, without time to validate the change in staging.
2. The incident with no trace. A regression appears. The on-call engineer opens Tempo. The trace is not there. The service grew; the head sampler rate held; the rare-error coverage decayed below the threshold needed for investigation.
3. The undocumented change. A team member changed a sampling rate during an incident (“we need more traces now”). The change was not committed; the next deploy reverted it; the next incident had no traces again. The team’s memory of the change is the only record.

The re-tuning discipline prevents all three. The cadence catches the cost drift; the triggers catch the incident gap; the documentation prevents the rollback surprise.

How it works

The process is mechanical once the metrics and the cadence are agreed.

Observe

The observation step reads the metrics that indicate the current strategy is misaligned with the traffic.

The metrics that matter:

• Backend ingest rate. Tempo’s ingest counter, split by service. A service whose ingest has doubled over the quarter is a candidate for a rate review.
• Cost per service. The cost allocation for the backend, split by tenant or by label. A service whose share of the ingest has grown materially is a candidate for a rate review.
• Tail sampler dropped-too-early. The traces_dropped_too_early counter on the gateway. A non- zero counter is the gateway being undersized; the fix is either to raise the rate and so reduce the volume, or to right-size the gateway.
• Rare-error coverage. A weekly test that forces a known error trace and confirms it is retained. A test that starts failing is the rare-error coverage decaying.
• Per-service p99 trace duration. A growing p99 means traces are taking longer; the decision_wait may now be shorter than the slowest trace; partial-trace decisions are increasing.

Decide

The decision step interprets the observation. A service whose ingest has doubled is a candidate for either a rate cut (to control cost) or a tail sampling addition (to preserve rare-error coverage at the higher volume). A gateway with non-zero dropped-too-early is a candidate for either a rate cut or a gateway resize.

The decision should be a sentence:

• “Lower the head sample rate for scheduler from one percent to 0.5 percent to control cost.”
• “Raise num_traces on gateway-2 from 20 000 to 40 000 to eliminate dropped-too-early.”
• “Add a string_attribute policy on the payments gateway to keep traces with span events named exception.”

A decision that is not a sentence is a decision that is not yet a decision.

Change

The change step applies the decision as a version-controlled edit to the collector config.

The change goes through a pull request with the decision sentence as the description. The reviewer confirms the sentence matches the diff. The change is merged; the rollout plan is the next step.

Validate

The validation step confirms the change has the expected effect in staging.

The validation has three parts:

• Parse-check. otelcol validate confirms the config parses.
• Component check. otelcol components confirms the processors named in the diff are present in the binary.
• Behaviour check. Force a known error trace; confirm the trace is retained by the expected policy. Read the metrics; confirm the rate matches the new configuration.

The validation is a stage gate. The change does not go to production until the validation passes.

Document

The documentation step records the change and the rationale in a place the next reviewer can find.

The minimum documentation is:

• The decision sentence. What changed and why.
• The metrics before. The numbers that triggered the decision.
• The expected metrics after. The numbers the change is expected to produce.
• The next review date. When the change should be revisited.

A change log in the repo (CHANGELOG-sampling.md), an entry in the runbook, or a row in the metrics-tracking spreadsheet are all acceptable forms. The discipline is that the next reviewer can find the rationale without asking the team that made the change.

Under the hood

How to configure it

The configuration of re-tuning is not a config file; it is a process. The collector’s role is to publish the metrics that the process reads.

A minimal Prometheus scrape config that pulls the collector metrics for the re-tuning review.

# /etc/prometheus/prometheus.yml

scrape_configs:
  - job_name: otelcol-edge
    static_configs:
      - targets:
          - edge-1.observability.internal:8888
          - edge-2.observability.internal:8888
          - edge-3.observability.internal:8888

  - job_name: otelcol-gateway
    static_configs:
      - targets:
          - gateway-1.observability.internal:8888
          - gateway-2.observability.internal:8888

A Grafana dashboard that shows the per-service empirical sampling rate and the dropped-too-early counter is the operational view for the review.

  Panel 1:  Backend ingest per service  (traces / second)
            Tempo ingest counter by service.name

  Panel 2:  Empirical sampling rate    (ratio)
            sampled / (sampled + dropped) by service.name

  Panel 3:  Tail sampler kept counter  (per policy)
            otelcol_processor_tail_sampling_count_traces_kept

  Panel 4:  Dropped-too-early counter  (per gateway)
            otelcol_processor_tail_sampling_traces_dropped_too_early

  Panel 5:  Cost per service           (currency / day)
            Tempo cost allocation by tenant label

The dashboard is the operational view; the re-tuning review reads the dashboard, identifies the divergences, and writes the decision sentences.

How to validate it

Validation confirms the re-tuning change has the expected effect in production.

# READ-ONLY: confirm the new head sample rate is in effect.
curl -s http://gateway-1.observability.internal:8888/metrics \
  | grep probabilistic_sampler

otelcol_processor_probabilistic_sampler_count_traces_sampled{policy="scheduler"}  29
otelcol_processor_probabilistic_sampler_count_traces_dropped{policy="scheduler"} 5771

The ratio 29 / (29 + 5771) = 0.005, matching the new 0.5 percent configuration. The empirical rate matches the configured rate.

# READ-ONLY: confirm the gateway is no longer dropping too early.
curl -s http://gateway-1.observability.internal:8888/metrics \
  | grep traces_dropped_too_early

otelcol_processor_tail_sampling_traces_dropped_too_early  0

A zero counter confirms the gateway is sized for the volume.

# READ-ONLY: confirm the rare-error coverage is preserved.
# (a forced-error test trace in the test environment)
curl -X POST http://checkout.test.internal/trigger-error
sleep 5
curl -s http://tempo.observability.internal:3200/api/search \
  --data-urlencode 'q={ service.name = "checkout" && status = error }' \
  | jq '.traces | length'

1

A non-zero result confirms the rare error trace was retained by the policy. A zero result confirms the policy did not match; the change did not preserve rare-error coverage.

How it can fail

Five failure modes specific to the re-tuning process.

1. No cadence. A team that owns the sampling strategy but never schedules a review. Symptom: the strategy drifts out of alignment with the traffic; the cost review or the next incident names the misalignment.
2. Watching the wrong metric. A team reads the backend ingest rate but not the empirical sampling rate. Symptom: the team sees the ingest climbing and assumes the traffic grew; the actual cause is the rate drifting to a higher value due to a config change no one documented.
3. Reactive cuts under pressure. A team receives a cost- cut directive from management and cuts rates fleet-wide without validation. Symptom: rare-error coverage drops; the next incident has no trace; the post-mortem names the cost cut as the cause.
4. No rollback plan. A team changes the rate but does not record the previous value. Symptom: the new rate causes an unexpected problem; the team cannot revert without guessing at the previous value.
5. Change without documentation. A team member changes a rate during an incident but does not commit the change. Symptom: the next deploy reverts the change; the next incident has no trace again; the team’s memory is the only record.
6. Re-tune triggered by a noisy metric. A team reacts to a single-day spike and cuts the rate, then discovers the spike was a one-off deploy. Symptom: the rate is now too low for steady state; the rare-error coverage drops for the next month.

How to troubleshoot it

A wrong-strategy investigation asks four questions in order.

1. What is the empirical sampling rate? Read sampled / (sampled + dropped) from the collector metrics. The empirical rate should match the configured rate within sampling noise. A divergence means the config has drifted from what is running.
2. What is the backend ingest by service? Read the Tempo ingest counter by service. A service whose ingest has grown materially is the candidate for a rate review.
3. Is the gateway saturated? Read traces_dropped_too_early on every gateway. A non-zero counter is the gateway being undersized; the fix is either to resize the gateway or to cut the upstream rate.
4. Is the rare-error coverage preserved? Run a forced- error test in staging; confirm the trace is retained. A failure means the policy is not matching; the fix is either to fix the SDK or to revise the policy.

Security implications

• Sampling change as a credential change. A sampling change is a behaviour change at the gateway; the change must go through the same review and approval as a credential change. The PR must have a second reviewer; the change must be deployable from version control, not from a manual edit on a running host.
• Audit trail. The re-tuning process produces an audit trail: who changed the rate, when, why, and what the metrics were. The audit trail is the record the next reviewer reads; the absence of an audit trail is a smell.
• Rate as a budget for sensitive services. A sampling rate for a sensitive service (auth, payments) is a security decision as much as a budget decision. A rate cut that loses rare-error coverage is a security regression; the rate cut must be reviewed by the team that owns the service, not only by the platform team.

Performance implications

• Re-tune cost. The re-tune process itself has a cost: the time to read the metrics, write the decision sentences, open the pull request, validate in staging, roll out to production, document the change. The cost is roughly four to eight hours per quarter for a small fleet; the benefit is the alignment that prevents the cost surprise.
• Pipeline latency. A change to a head sampler rate takes effect on the next request. A change to a tail sampler policy takes effect on the next trace; traces that are in flight at the time of the change use the previous policy until they complete.
• Gateway sizing. A change to a head sampler rate changes the gateway ingest; a change to a tail sampler policy changes the gateway CPU. The re-tune review reads the gateway metrics to confirm the new rate does not push the gateway past its budget.

Production guidance

• Schedule the cadence. A re-tune review on the calendar (monthly for high-growth fleets, quarterly for stable fleets). The cadence is the rhythm that prevents drift.
• Watch the right metrics. Empirical sampling rate, backend ingest by service, dropped-too-early on the gateway, rare-error coverage from a forced-error test. These four metrics are the operational view for the review.
• Validate in staging first. Every change goes through otelcol validate, otelcol components, and a forced- error behaviour check before production.
• Document every change. The decision sentence, the metrics before, the metrics after, the next review date. A change log in the repo is the minimum.
• Make the change reversible. Record the previous value; roll back from the change log if the new value is wrong.

Verification

You should now be able to answer:

• What is the re-tuning cadence for a sampling strategy, and what triggers it?
• What four metrics drive the re-tuning review?
• What three validation steps confirm a re-tune change is correct?
• Why must every re-tune change be documented with the previous value?

Quiz