High-Cardinality Attributes

The on-call engineer opens a TraceQL query that worked last week. The query is { resource.service.name = "checkout" }. The Grafana panel spins for forty-five seconds and returns a “query timeout”. They open the Tempo metrics endpoint. The metric tempo_search_external_lookups_total is climbing. The metric tempo_index_search_result_count is enormous. The Tempo ingester’s blocklist has more entries than last week. Something about the indexed attributes has changed.

A colleague diffs the application’s instrumentation. Last week, the checkout service stamped user.email on every span as a debugging aid during the previous incident. The PR was merged. The attribute is in production. The user.email value space is unbounded — every new user is a new value. Tempo has indexed every distinct value. The index has blown.

This is the lesson. Attribute cardinality is the cost driver of distributed tracing. The cost is invisible until the index blows.

What it is

A high-cardinality attribute is a span attribute whose value space is large or unbounded. The cardinality of an attribute is the number of distinct values the attribute takes across all spans. An attribute with a value space of three (success, failure, pending) has cardinality 3. An attribute with a value space of one value per user has cardinality equal to the user count.

Tempo indexes span attributes for search. The index is a per-attribute inverted map from value to spans. The size of the map is proportional to the number of distinct values times the number of spans that contain each value. A high-cardinality attribute makes the index large, the search slow, and the storage cost high.

The cost shape is non-linear. A service with 10 000 distinct users and 100 spans per user has 1 million spans and a cardinality-100 000 attribute. The search { user.id = "..." } is O(1) per match, but the index itself is many gigabytes.

Why a sysadmin cares

Tempo’s design budget assumes bounded cardinality. The OTel semantic conventions explicitly call out attributes that are acceptable to index (http.method, http.status_code, service.name) versus attributes that are not (user.id, request.id, raw URLs, raw error messages).

Three operational payoffs ride on the cardinality budget:

1. Query latency. A TraceQL query that selects on a high-cardinality attribute has to scan the inverted index. The scan is O(matches), not O(spans), but the index is large. The query is slow. The Grafana panel times out.
2. Index size. The inverted index is in memory at the querier. A cardinality explosion is a memory exhaustion. Tempo starts swapping; queries slow by an order of magnitude.
3. Storage cost. Every distinct value in every block is stored on disk. The block size grows with the cardinality. The retention math changes — what looked like a 30-day retention at 1 TB is now a 7-day retention at 5 TB.

The cost is paid in the next query, the next retention window, and the next capacity review.

How it works — the mental model

Tempo stores traces as blocks. A block contains every span received in a window (typically 5 to 30 minutes). Within the block, Tempo builds an inverted index over the span and resource attributes.

Incoming spans (from the collector)
  +-- block for window [T, T+15m)
       +-- inverted index:
            span.http.status_code = 200  -> [span_a, span_b, ...]
            span.http.status_code = 500  -> [span_x, span_y]
            resource.service.name = "checkout" -> [every span]
            user.id = "u12345"          -> [span_a, span_c]
            user.id = "u12346"          -> [span_b]
            ...
            (the size of the index grows with the cardinality
             of every attribute)
       +-- the block is flushed to storage
       +-- the inverted index lives in memory at the querier

A TraceQL query { span.http.status_code = 500 } is a lookup against the inverted index. The lookup is O(matches). A query { user.id = "u12345" } is also a lookup, but the inverted index for user.id has as many entries as there are users.

How to configure it

The fix is in two places: the application (don’t stamp high-cardinality attributes) and the collector (strip or transform them).

The OTel Collector processor for stripping attributes:

# /etc/otelcol-contrib/config.yaml
processors:
  # Drop high-cardinality attributes before they reach Tempo.
  attributes/limit:
    include:
      # Only allow these attributes; drop everything else.
      match_type: strict
      attributes:
        - service.name
        - service.namespace
        - service.instance.id
        - http.method
        - http.status_code
        - http.target        # the path, not the URL
        - http.route
        - db.system
        - db.statement        # consider the cost
        - error.type
    actions:
      - key: db.statement
        # Truncate long SQL statements to bound the cardinality.
        action: truncate
        max_value_len: 200

  # Hash high-cardinality identifiers that are useful for the
  # application but should not be indexed by Tempo.
  transform:
    trace_statements:
      - context: span
        statements:
          - replace_pattern(attributes["user.id"], "user.id", "$$user_id_hash")
          - hash(attributes["user.id"], "user.id_hash")
          - delete(attributes, "user.id")

Three patterns to apply:

• Allow-list. Only stamp the attributes the team has agreed to index. The attributes/limit processor drops everything else. The application discipline catches what the collector misses.
• Truncate. Long strings (SQL statements, error messages, URLs) have unbounded cardinality by virtue of their length. Truncate to a fixed maximum length; the first 200 characters of a SQL statement are usually enough to identify the query.
• Hash. High-cardinality identifiers (user.id, order.id) are useful for application-level debugging but useless for cross-cutting search. Hash the value to a fixed-size string; the cardinality is bounded by the hash space.

The Grafana Alloy equivalent:

otelcol.processor.attributes "limit" {
  include {
    match_type = "strict"
    attributes = [
      "service.name",
      "service.namespace",
      "service.instance.id",
      "http.method",
      "http.status_code",
      "http.target",
      "http.route",
      "db.system",
      "error.type",
    ]
  }
  output { traces = [otelcol.processor.batch.default.input] }
}

How to validate it

The diagnostic ladder:

# 1. What is the cardinality of the indexed attributes?
# Use Tempo's metric endpoint to inspect the highest-cardinality
# columns. (The exact metric name depends on the Tempo version;
# in 2.x it is tempo_index_search_result_count and the cardinality
# must be derived from the blocklist inspection.)
curl -sf http://tempo:3200/api/metrics | grep -E "tempo_index|cardinality"
# tempo_search_external_lookups_total{...} 41283
# tempo_search_external_lookup_result_count{...} 18432
# (the lookup result count tells you how many spans a search
#  found; a query that returns a huge result count for a
#  "specific" filter is a high-cardinality signal)

# 2. Which attribute is the culprit?
# TraceQL with stats by is the operational diagnostic.
tctl trace search --query='{ resource.service.name = "checkout" } | stats by (span.http.status_code, http.method) (count)' --since=1h
# (small table: cardinality is bounded)
#
# Repeat with a suspected high-cardinality attribute:
tctl trace search --query='{ resource.service.name = "checkout" } | stats by (span.user.id) (count)' --since=1h
# (huge table: user.id has exploded)

# 3. What is the Tempo block size in storage?
du -sh /var/tempo/blocks/
# (compare to last week's baseline; a sudden growth is a
#  cardinality-explosion signal)

# 4. Is the OTel Collector stripping attributes?
kubectl logs deploy/alloy -c alloy | grep -i "attributes"
# (the collector logs the matched and dropped attributes at
#  startup)

# 5. TraceQL search latency.
# Run a representative query and time it.
time tctl trace search --query='{ span.http.status_code = 500 }' --since=1h --limit=10
# real    0m4.213s
# (a healthy query is under 1 second; sustained multi-second
#  queries on "specific" filters are a cardinality signal)

# 6. Is the allow-list enforced?
grep -A 20 "attributes/limit:" /etc/otelcol-contrib/config.yaml
# (the include list should match the team's agreed attribute set;
#  a missing list means everything is indexed)

The TraceQL stats by query is the structural answer. A stats by on a high-cardinality attribute returns an enormous table; a stats by on a bounded attribute returns a small one.

How it can fail

Six recurring failure modes.

1. user.id stamped on every span. The application developer wanted to debug a per-user issue. They added attributes["user.id"] = user.ID to the span. The value space equals the user count. Symptom: Tempo’s index size grows with the user base; queries on user.id are slow even when the result set is small.
2. Raw URL stamped as http.target. The application stamps the full URL — query string included — on every span. The value space equals the number of distinct URLs, which grows with each campaign, each A/B test, each search query. Symptom: http.target cardinality climbs; the index bloats.
3. Full SQL statement stamped as db.statement. The application stamps the parameterised SQL on every span. Long statements, dynamic table names, and dynamic column lists push the cardinality up. Symptom: db.statement cardinality climbs; the index bloats.
4. Error messages stamped as error.message. The application stamps the exception message on every span. The message includes per-request details (the input that failed, the user that hit the error). Symptom: error.message cardinality grows with the failure surface.
5. The OTel Collector allow-list is missing. The collector forwards every attribute the SDK stamps. Without an allow- list, the team has no enforcement layer; the discipline is entirely in the application. Symptom: the index grows with whatever the application happens to stamp.
6. The application was migrated from logs. The team traditionally logged request_id, session_id, correlation_id. They “moved it to tracing” by stamping the same identifiers as span attributes. Symptom: the trace index now carries the same cardinality as the log label space — which was already a budget problem.

How to troubleshoot it

The diagnostic order:

1. What is the cardinality of each indexed attribute? Run stats by on the suspect attributes. The output table size is the cardinality.
2. What is the change? Diff this week’s metrics from last week’s. A sudden growth is a recent code change; a slow growth is a long-running drift.
3. Where is the attribute stamped? The application source. A PR that adds an attribute is the structural fix.
4. Is the allow-list in place? The collector config. If the allow-list is missing, add it. If the allow-list is in place but the attribute is in the list, the application discipline is the fix.
5. Can the attribute be hashed? A user.id is useful for per-user debugging but useless for cross-cutting search. Hash it to a fixed-size string; the cardinality is bounded.
6. Can the attribute be truncated? A db.statement is useful for query identification; the first 200 characters usually suffice. Truncate it.

Security implications

High-cardinality attributes are a PII exposure vector. A user.email attribute stamps the email on every span. Tempo stores it for the retention window. The querier can retrieve it. The on-call engineer can search for it. A breach of the Tempo credentials is a breach of every user’s email.

The remediation is hashing. The application computes a deterministic hash of the identifier (HMAC-SHA256 with a service-scoped secret) and stamps the hash. The hash is unique per user but not reversible. A search for the hash returns the same spans as a search for the original value, but the value itself is never stored.

A second risk is around the data egress. Tempo’s search API returns the span content, including the attribute values. A service that queries Tempo for a user.id and returns the value to a client has inadvertently built a user-enumeration endpoint. The remediation is to expose only the hash to the search API and to keep the original identifier out of Tempo entirely.

Performance implications

The cost of high cardinality is paid in three places.

• Index memory. Tempo’s inverted index lives in memory at the querier. A cardinality-1 000 000 attribute adds roughly 50 MB to the index per block. A search over 24 hours at 5-minute blocks is 288 blocks. The memory cost is cumulative.
• Query latency. A TraceQL query on a high-cardinality attribute is a scan of the index column. The scan is O(distinct values), not O(matches). A search for a value that exists in 10 spans still scans the entire index column.
• Storage. Every distinct value of every indexed attribute is stored in the block. The block size is proportional to the cardinality sum across all indexed attributes.

The performance budget is roughly 10 000 distinct values per attribute across the search window. Attributes above that threshold should be hashed, truncated, or excluded from the index.

Production guidance

• Allow-list the indexed attributes. The collector should drop any attribute not on the list. The application discipline is the second line of defence.
• Hash the high-cardinality identifiers. user.id, order.id, session.id are useful for application-level debugging but useless for cross-cutting search. Hash them.
• Truncate the unbounded strings. SQL statements, error messages, URLs — every unbounded string should have a maximum length.
• Run stats by weekly. The cardinality table is a budget document. A growing cardinality on any attribute is a sign that the budget is being eroded.
• Alert on Tempo index size. A sudden growth in the block list size is a cardinality-explosion signal.

Verification

You should now be able to answer:

• What is the difference between a high-cardinality attribute and a high-volume attribute?
• Why does Tempo’s index size grow with attribute cardinality?
• What is the most common production cause of a cardinality explosion?

Quiz