Tracing Cost Considerations

The team turns on 100% trace sampling across the production fleet. Three weeks later the Tempo storage bucket is at 18 TiB and the monthly cost has tripled. The investigation finds that a single service is emitting 80 spans per request and 70 of them are duplicates of the same DB query inside a loop. The cost is not in the platform; the cost is in the application that is over-instrumenting a hot path.

This lesson is the arithmetic that bounds the platform, the cardinality cliff that makes the arithmetic explode, and the sampling strategies that keep the platform within budget without losing the traces that matter.

What it is

The cost of tracing has three components: the storage cost of the spans themselves, the index cost of the trace_id → block mapping, and the ingest cost of the network and processor time to accept the spans. The three components are all linear in the number of spans per second, but the index and ingest costs scale with the per-span size while the storage cost scales with the per-span size plus the per-span overhead.

The storage arithmetic:

spans_per_second
  = request_rate * spans_per_request

bytes_per_second_in
  = spans_per_second * bytes_per_span

bytes_in_retention_window
  = bytes_per_second_in * retention_seconds

storage_cost
  = bytes_in_retention_window * dollars_per_gigabyte_month / window_seconds

The variables:

• request_rate — requests per second to the service.
• spans_per_request — average number of spans in a request.
• bytes_per_span — average size of a span in bytes (200 to 500 is typical for a well-instrumented service).
• retention_seconds — how long the spans are kept.
• dollars_per_gigabyte_month — the cost of the storage backend.

For a service at 1000 requests per second, 20 spans per request, 400 bytes per span, 14 days retention:

spans_per_second    = 1000 * 20                  = 20 000
bytes_per_second_in = 20 000 * 400               = 8 MB/s
                     = 8 MB/s * 86400 s/day       = 691 GB/day
                     = 691 GB/day * 14 days       = 9.7 TB
                     = 9.7 TB / 1024              = 9.5 TiB

Nine and a half tebibytes of spans over the retention window. At $0.02 per gigabyte-month on object storage, the monthly cost for the spans alone is roughly $190 per month for one service. Multiply by the number of services in the fleet and the arithmetic explains the bill.

Why a sysadmin cares

Three operational pains are specific to running traces without a cost model.

1. The storage bill that arrives at the end of the month. The team turns on tracing; the cost grows linearly with traffic; the bill arrives after the fact. The team disables tracing to control cost; the team loses the investigation tool. The cost was predictable from the first day of sampling; nobody ran the arithmetic.
2. The service that emits 200 spans per request. The service is over-instrumented; a hot loop emits a span per iteration; the span rate is bounded by the request rate times the loop count. The cost grows with the request rate; the investigation value of the loop spans is zero because they are duplicates of the same operation.
3. The high-cardinality attribute that explodes the index. A service adds user_id as a span attribute. Every distinct user creates a distinct trace. The trace backend’s block count grows linearly with the user count; the query cost grows linearly with the block count; the retention cost grows linearly with the storage size. The team cannot sample their way out; the cardinality is in the data.

How it works

The cost model has three levers: the number of spans, the size of each span, and the retention window. The sampling strategy chooses which spans are kept; the cardinality budget caps the per-span size; the retention policy caps the retention window.

The cardinality cliff

The cardinality cliff is the point at which a per-span attribute’s distinct value count overwhelms the storage and index budgets.

Attribute value count   Cost in Tempo block index   Cost in trace ID index
---------------------   --------------------------   ----------------------
10                       trivial                      trivial
1 000                    low                          low
10 000                   measurable                   measurable
100 000                  high                         high
1 000 000                prohibitive                  prohibitive

A span attribute that has 1 000 000 distinct values creates 1 000 000 distinct traces (assuming every value is seen once per request). The trace backend’s block index grows by 1 000 000 entries; the query cost to find any one trace grows with the index size; the retention storage grows with the number of blocks.

The OTel semantic conventions provide a list of attributes that are safe to record and a list that are unsafe. The unsafe attributes are the ones that grow with user input: user.id, session.id, request.id, http.url with embedded identifiers.

The right discipline is to record the attribute as a resource attribute (one per service) rather than a span attribute (one per span), or to record a low-cardinality bucketed value rather than the raw value. user.tier = "premium" is a safe attribute; user.id = "u-12345" is not.

Head-based vs tail-based sampling

The sampling decision is the second lever. The decision has two shapes:

• Head-based sampling decides at the start of a trace whether to keep it. The decision is made in the SDK or at the first collector; the decision is the same for every span in the trace. Head-based sampling is cheap: the decision is a probability check, the cost is negligible. The disadvantage is that the decision cannot see the outcome — a failing trace and a successful trace have the same probability of being kept.
• Tail-based sampling decides at the end of a trace whether to keep it. The decision can use the trace’s outcome, duration, error status, or any other attribute. Tail-based sampling is more useful for catching rare failures. The disadvantage is that every span must be forwarded through the collector to make the decision; the network and processor cost is the full trace volume, not the sampled volume.

The choice between the two is a cost-versus-investigation trade-off:

A small fleet with rare failures uses tail-based sampling at the collector. A large fleet with budget pressure uses head-based sampling at the SDK and accepts the lost rare failures in exchange for the lower cost.

The cost-versus-investigation trade

The trade between cost and investigation depth is the central decision. The arithmetic bounds the platform; the sampling strategy determines how much of the trace space is preserved; the retention policy determines how far back the investigation can reach.

Total monthly cost
  = request_rate * spans_per_request * bytes_per_span
    * retention_seconds * dollars_per_gigabyte_month
    / 86400 / 1024^3

Investigation depth
  = spans_per_request * sampling_rate * retention_days

A team that wants 30 days of investigation depth at 1% sampling of a 20-span trace has 6 spans available per request type on average. A team that wants 7 days of investigation depth at 100% sampling has all 20 spans available. The first team has one-third the storage cost and one-third the investigation depth.

Under the hood

How to configure it

A sampling policy is the configuration that bounds the cost. For a Python service using the OTel SDK, the head-based sampler:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.sampling import (
    ParentBased, TraceIdRatioBased, ALWAYS_ON
)

# ParentBased(TraceIdRatioBased(0.01)) keeps 1% of traces
# and respects the parent's sampling decision for child spans.
sampler = ParentBased(TraceIdRatioBased(0.01))

tracer_provider = TracerProvider(sampler=sampler)
trace.set_tracer_provider(tracer_provider)

ParentBased is the discipline: every child span respects the parent’s sampling decision. A trace that was sampled out at the root does not become a partial trace because a child span decided to keep itself. TraceIdRatioBased(0.01) keeps roughly 1% of traces.

For tail-based sampling at the collector, the loadbalancing and tail-sampling processors in the OpenTelemetry Collector:

# /etc/alloy/config.alloy
otelcol.receiver.otlp "default" {
  grpc { endpoint = "0.0.0.0:4317" }
  output { traces = [otelcol.processor.tail_sampling.default.input] }
}

otelcol.processor.tail_sampling "default" {
  decision_wait = "10s"
  num_traces     = 50000
  expected_new_traces_per_sec = 1000

  policy {
    name = "errors-and-slow"
    type = "and"
    and {
      policy {
        name = "errors"
        type  = "status_code"
        status_code { status_codes = ["ERROR"] }
      }
      policy {
        name = "slow"
        type  = "latency"
        latency { threshold_ms = 1000 }
      }
    }
  }
  policy {
    name = "baseline"
    type  = "probabilistic"
    probabilistic { sampling_percentage = 1 }
  }

  output { traces = [otelcol.exporter.otlp.tempo.input] }
}

The policy says: keep every trace that has an error status and is over 1 second; keep 1% of the rest. The cost is the full trace volume to the collector; the stored volume is the sampled subset.

For retention, Tempo’s storage configuration:

# Tempo configuration
storage:
  trace:
    backend: s3
    s3:
      bucket: tempo-traces
    wal:
      path: /var/tempo/wal

The retention is set at the bucket level (S3 lifecycle rules) or at the Tempo level via the compactor’s block_retention setting.

How to validate it

Three checks confirm the cost is within budget.

# READ-ONLY: confirm the current span ingest rate.
curl -s http://localhost:8889/metrics \
  | grep otelcol_receiver_accepted_spans \
  | grep -v "^#" | head -3

otelcol_receiver_accepted_spans{receiver="otlp",service_name="checkout"} 18423
otelcol_receiver_accepted_spans{receiver="otlp",service_name="cart"} 9120
otelcol_receiver_accepted_spans{receiver="otlp",service_name="payment-svc"} 22015

The counters advance at the ingest rate. The next check is the per-span size distribution:

# READ-ONLY: confirm the average span size is within budget.
curl -s http://localhost:8889/metrics \
  | grep -E "^otelcol_exporter_sent_(spans|bytes)" | head -5

otelcol_exporter_sent_spans{exporter="otlp/tempo"} 18423
otelcol_exporter_sent_bytes{exporter="otlp/tempo"} 7369200

Average span size is 7369200 / 18423 = 400 bytes per span. The check against the budget:

# READ-ONLY: confirm the bucket size is within the budget.
aws s3api list-objects --bucket tempo-traces \
  --query 'sum(Contents[].Size)' --output text

9673825012

The bucket holds roughly 9 GiB. The arithmetic from earlier predicted 9.5 TiB; the actual is 9 GiB because the OTel SDK’s compression gives a 1000x compression ratio on the network and the bucket compresses further. The arithmetic is the upper bound; the actual is the lower.

The third check is the cardinality budget:

# READ-ONLY: confirm the trace_id count for a high-cardinality
# attribute is bounded.
curl -s -u "${TEMPO_USER}:${TEMPO_PASS}" \
  'https://tempo.internal.example.com/api/search?query={user.id="u-12345"}&limit=1' \
  | jq '.traces | length'

1

A non-zero count means the attribute was searchable; a zero count means the attribute was not indexed. The discipline is to verify that attributes that should not be indexed are not indexed.

How it can fail

Six failure modes specific to trace cost.

1. The 100% sampling default. The SDK is initialised with ALWAYS_ON sampling. Symptom: every request is sampled; the storage cost grows linearly with traffic; the bill arrives at the end of the month. Cause: the default sampler was not changed during the SDK initialisation.
2. The high-cardinality attribute that is not bounded. The application records request.id as a span attribute on every span. Symptom: every trace is unique; the sampling rate is irrelevant because no two traces share a trace_id prefix; the storage is essentially 100% unique. Cause: a developer added the attribute without thinking about the cardinality cost.
3. The retention window that grows with the storage budget. The team increases retention from 14 days to 90 days to keep more history. Symptom: the storage cost multiplies by 6.4; the investigation value does not. Cause: a change to the retention policy without a corresponding change to the sampling rate.
4. The tail-based sampling that is too generous. The collector’s tail-sampling policy is “keep every trace over 100 ms and every error”. Symptom: the storage cost matches the request rate; the collector’s CPU and memory cost matches the full request rate times the trace size. Cause: the policy’s thresholds are too low; the “interesting” set is too large.
5. The hot loop that emits a span per iteration. A service iterates over a list of items and emits a span per item. Symptom: the per-request span count is 200 for a request that should be 10. Cause: a developer added per-iteration instrumentation for debugging and forgot to remove it.
6. The attribute that should be a resource. The application records service.version on every span instead of as a resource attribute. Symptom: the per-span size is inflated by a constant string; the cost is measurable but the investigation value is zero. Cause: the attribute was set with span.set_attribute instead of resource.merge.

How to troubleshoot it

When the cost is higher than expected, the order matters.

1. Measure the current rate. The arithmetic in the “How to validate it” section is the baseline. A rate that is higher than the baseline is the first regression indicator.
2. Find the service with the highest span rate. The otelcol_receiver_accepted_spans metric has a service_name label. The service with the highest rate is the largest contributor to the cost.
3. Find the spans with the highest per-span size. The otelcol_exporter_sent_bytes metric divided by otelcol_exporter_sent_spans gives the average per-span size per service. The service with the largest per-span size has the highest-cardinality attributes.
4. Find the attributes with the highest cardinality. A TraceQL query that selects by a candidate attribute and counts the distinct values over a time window gives the cardinality. The attribute with the highest cardinality is the first candidate for removal.
5. Adjust the sampling policy. A change from 1% to 0.1% head-based sampling reduces the storage cost by 10x. A change from “keep all errors” to “keep errors over 500 ms” reduces the storage cost by another factor.

Security implications

The cost discipline and the security discipline overlap on the cardinality budget. A high-cardinality attribute is both a storage cost and a potential information leak: an attribute that records user.email creates a unique trace per user; the trace contains the email; the trace is searchable in Tempo.

Three operational rules:

• Redact high-cardinality PII at the SDK. The same redaction discipline that applies to logs applies to spans. Email addresses, authorisation headers, and tokens must be redacted before the span is exported.
• Cap the cardinality budget per attribute. The cardinality budget is a number; the team should write it down; the team should audit the attributes against the budget.
• Apply access control on the trace backend. The retention storage contains the same data the application handles. The ACL on Tempo’s read API must match the data classification.

Performance implications

The cost model has three performance implications on the production fleet.

• CPU on the SDK host. The SDK adds CPU cost for sampling decisions, attribute allocation, and batching. Head-based sampling at 1% adds negligible cost; 100% sampling adds noticeable cost on a hot path.
• Network between SDK and collector. The OTLP export carries every span that the SDK records; the cost is the span size times the rate. Compression on the OTLP export is the SDK’s default.
• Storage on the backend. The cost is the per-span size times the rate times the retention. The arithmetic is the budget.

Production guidance

• Write down the sampling policy as a contract. The policy is the answer to “why is the cost what it is” and “why did we miss this rare failure”. A policy that lives only in the SDK configuration is a policy that nobody can change safely.
• Audit the cardinality budget per attribute. The attribute list is finite; the cardinality budget is finite; the audit is a script that compares the two.
• Measure the per-span size per service. The per-service breakdown is the actionable view; the fleet total is the budget view.
• Set the retention to match the investigation need. A retention that is longer than the longest investigation is over-provisioned; a retention that is shorter is under-provisioned. The investigation need is the input; the retention is the output.

Verification

You should now be able to answer:

• What four variables determine the storage cost of traces?
• Why does a high-cardinality span attribute create a unique trace per value?
• What is the difference between head-based and tail-based sampling, and what is the cost trade-off?
• What is the first metric to inspect when the storage cost is higher than expected?
• What is the right place to apply redaction for high- cardinality PII: the SDK or the collector?

Quiz