Attribute Design

The team is investigating a latency spike. The on-call engineer opens Tempo, filters by service.name = checkout-svc, groups by name, and reads the slowest span. The slowest span is order.validate. The engineer wants to know which tenant or which region experienced the spike. The span has no tenant attribute and no region attribute. The engineer is stuck.

The fix is one attribute per relevant question. The lesson is about choosing those attributes deliberately, in advance, because every missing attribute is a question the engineer cannot answer later.

What it is

An attribute is a key-value pair on a span or resource. The attribute is the mechanism that lets an incident investigator narrow a trace down to a specific request, tenant, or operation. The OpenTelemetry specification defines two attribute classes:

• Span attributes — describe the work the span represents. http.request.method, db.system, order.id. They are scoped to the span and join with the trace ID.
• Resource attributes — describe the entity that emitted the span. service.name, service.version, deployment.environment, k8s.pod.name. They are scoped to the resource and appear on every span from the same process.

The naming convention is dotted lower-case with underscores (for example, http.request.method). The values are strings, booleans, integers, floats, or arrays of these. The decision about which values to use is the design the lesson is about.

+-------------------------------+   +-------------------------------+
| Span attributes               |   | Resource attributes           |
| (per-operation)               |   | (per-process)                 |
|                               |   |                               |
|  http.request.method=POST    |   |  service.name=checkout-svc  |
|  http.route=/api/v1/orders   |   |  service.version=1.42.0    |
|  http.status_code=200        |   |  deployment.environment=prod |
|  db.system=postgresql        |   |  k8s.pod.name=checkout-7d   |
|  order.id=ord-123           |   |  k8s.namespace.name=shop   |
|  order.amount=4995           |   |                               |
|  order.instrumentation.kind    |   |                               |
+-------------------------------+   +-------------------------------+

The two classes play different roles. Span attributes are the “questions an investigator will ask about this operation”. Resource attributes are the “questions an investigator will ask about this service or instance”.

Why a sysadmin cares

The cardinality of an attribute is the number of distinct values it can take. The cardinality of an attribute is the cost of the attribute. The cardinalities multiply: a span with http.method (5 values) and http.status_code (15 values) and service.name (50 values) has 5 * 15 * 50 = 3750 unique combinations in the index. A span with order.id (one per order) and service.name (50 values) has up to 50 * (number of orders) combinations, which is unbounded.

The production cost is paid on every dimension. Tempo’s index size is proportional to the number of unique attribute-value pairs. The query latency is proportional to the index size. The dashboard refresh is proportional to the query latency. The cost of one attribute being high-cardinality is a 10x regression in the cost of every other attribute on the same span.

The lesson is the discipline of bounding cardinality. The OTel semantic conventions are the team’s starting point because they codify the dimensions that are useful (http.method) and the dimensions that are dangerous (http.url with the query string).

How it works

The attribute design has three knobs and one hard rule.

+--------------------------+   +--------------------------+
| Knob 1: Type             |   | Knob 2: Cardinality       |
| (string, int, float,     |   | (bounded vs unbounded)    |
|  bool, array of these)   |   |                          |
+--------------------------+   +--------------------------+
              |                            |
              +-------------+--------------+
                            |
                            v
                +--------------------------+
                | Knob 3: Lifetime         |
                | (span vs resource)       |
                +--------------------------+
                            |
                            v
                +--------------------------+
                | Rule: index = type x      |
                | cardinality x count      |
                +--------------------------+

The hard rule is the index. The cost of an attribute is the product of its type width, its cardinality, and the number of spans with that attribute. The rule is the budget that a service can spend on attributes.

How to configure it

The configuration is twofold: the SDK code that sets the attribute, and the SDK-level filter that bounds it.

Setting attributes. The OTel semantic conventions are the first place to look. The convention names are the answer to “what is the canonical name for the HTTP method”, and using the convention means dashboards written by other teams work on this service’s traces.

from opentelemetry.semconv.trace import SpanAttributes
from opentelemetry import trace

tracer = trace.get_tracer("checkout-svc")

with tracer.start_as_current_span("order.validate") as span:
    # Convention keys
    span.set_attribute(SpanAttributes.HTTP_REQUEST_METHOD, "POST")
    span.set_attribute(SpanAttributes.HTTP_ROUTE, "/api/v1/orders")
    span.set_attribute(SpanAttributes.HTTP_RESPONSE_STATUS_CODE, 200)

    # Domain attributes
    span.set_attribute("order.id", order.id)
    span.set_attribute("order.amount", order.amount_cents)
    span.set_attribute("tenant.id", tenant.id)

The convention keys come from the opentelemetry-semantic-conventions package. The keys are static strings under the hood; the package is the canonical dictionary.

Bounding cardinality. The SDK ships a SpanProcessor that filters attributes at the boundary. The Collector has the same capability. The SDK is the right place for the hard limit because the application’s CPU cost of building the high-cardinality attribute is paid once.

from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.trace import Span, ReadableSpan

class CardinalityFilter:
    def __init__(self, max_distinct_values: int = 100):
        self.seen = {}
        self.max_distinct_values = max_distinct_values

    def on_end(self, span: ReadableSpan) -> None:
        attrs = dict(span.attributes)
        for key, value in attrs.items():
            seen = self.seen.setdefault(key, set())
            if len(seen) > self.max_distinct_values:
                # Attribute over budget -- drop it from the span
                span._attributes.pop(key, None)
            else:
                seen.add(value)

The discipline is to enumerate every attribute the service sets, name the maximum cardinality, and add a guard at the SDK. The guard is testable. The cost of not having the guard is unbounded.

Resource attributes. The configuration is at SDK initialisation, not per-span.

from opentelemetry.sdk.resources import Resource, SERVICE_NAME
from opentelemetry.semconv.resource import ResourceAttributes

resource = Resource.create({
    SERVICE_NAME: "checkout-svc",
    ResourceAttributes.SERVICE_VERSION: "1.42.0",
    ResourceAttributes.DEPLOYMENT_ENVIRONMENT: "prod",
})

The semantic conventions for resource attributes are the right answer because Grafana’s data-source configurations expect them. The dashboards that filter by service.name will work; the dashboards that filter by serviceName will not.

How to validate it

The validation is twofold: confirm the attributes are set as expected, and confirm the cardinality is within budget.

tempo-cli query '{ resource.service.name = "checkout-svc" } && span.http.request.method = "POST" } | limit 1'

tempo-cli metrics cardinality --service checkout-svc --since 24h | sort -k2 -nr | head -20

$ tempo-cli metrics cardinality --service checkout-svc --since 24h

order.id              1245023
order.payload           982345
order.amount              4638
tenant.id                  187
http.request.method          5
http.response.status_code   12
db.statement                43
service.name                 1
service.version              1
deployment.environment       1

curl -s http://otel-collector:8888/metrics | grep -E 'otelcol_processor_dropped|otelcol_exporter_dropped'

tempo-cli query '{ resource.deployment.environment = "prod" } && { span.order.amount > 1000 }'

How it can fail

Four failure shapes recur. The first two are design mistakes; the last two are operational regressions.

1. The attribute is unbounded. The developer adds span.set_attribute("user.full_request", json.dumps(request)) which captures the entire request payload. The distinct-value count grows with the unique requests. The index returns “yes” for every block. Symptom: a Bloom filter that flags every block, a query that’s slow regardless of the specific conditions.
2. The attribute is the wrong semantic convention. The developer uses request.method instead of http.request.method. The convention-following dashboards miss the span. Symptom: the service is missing from a dashboard that should include it.
3. The attribute stringifies non-string values. The developer sets span.set_attribute("order.amount", 49.95) but the dashboard expects a string. The query span.order.amount > 1000 returns nothing. Symptom: a numeric query that fails on a numeric attribute.
4. The high-cardinality attribute survives an upgrade. The auto-instrumentation library adds a new attribute that captures the bearer token. The trace contains credentials. The cost is paid on the index and on the audit. Symptom: a security incident.

How to troubleshoot it

The diagnostic order is “is the attribute set?”, “is the value the right type?”, “is the cardinality within budget?”, “is the query finding it?”.

tempo-cli trace <trace_id> | jq '.resourceAttributes, .span.attributes'

tempo-cli metrics cardinality --service checkout-svc --since 24h | sort -k2 -nr | head

tempo-cli query '{ span.order.amount > 1000 }' --explain

Security implications

The OTel semantic conventions are explicit about the attributes that must be scrubbed: http.url carries the query string (which can include bearer tokens), db.statement carries the SQL statement (which can include credentials), enduser.id should be a hashed identifier rather than an email address. The auto-instrumentation libraries may set these attributes by default.

The mitigation is the SDK-level scrubber. The Collector also has an attributes processor that can drop or hash key-value pairs. The right place is the SDK, because the application’s CPU spent building the attribute is paid once, and the scrubber prevents the value from appearing in the batch queue.

Performance implications

The cost of an attribute is the cost of the index entry plus the cost of the matching query. The cost is proportional to the cardinality of the attribute and the number of traces that carry it. High-cardinality attributes are the dominant cost of a Tempo cluster.

The SDK-level cost of setting an attribute is small — a single dictionary insertion — but the value has to be serialised into OTLP, which is a CPU cost. A span with 50 attributes is bulkier than a span with 5. The discipline is to set the attributes that the operator will search for and to drop the rest.

Production guidance

Verification

You should now be able to answer:

• What is the difference between a span attribute and a resource attribute?
• Why is cardinality the dominant cost of an attribute in Tempo?
• Name two OTel semantic conventions for HTTP request attributes and two for resource attributes.
• What is the right place to scrub a high-cardinality attribute, and why?
• Why does a high-cardinality attribute slow down queries for unrelated attributes on the same service?

Quiz