Span Attributes

A trace arrives in Tempo. You have one span, a duration, a service name. That is not enough to find the slow request from the user who reported it. Add ten well-chosen attributes — the HTTP method, the route template, the user id, the response status — and the same trace becomes searchable: “show me every trace for GET /orders/{id} that returned 500 for user u_8821 in the last ten minutes”. The first trace was an event. The second is an investigation.

Attributes are what turn a span from a datum into a queryable signal.

What it is

A span attribute is a typed key / value pair attached to a span. OpenTelemetry defines a set of standard attributes (the semantic conventions) and lets applications add custom attributes for anything not covered. Attributes are the search keys in TraceQL, the dimensions for group-by in Grafana panels, and the metadata every trace UI relies on to make spans readable.

Attributes are flat — they are not nested objects. Each attribute has a name (dot-separated, namespaced) and a value of one of four types: string, boolean, double-precision float, or signed 64-bit integer. Attribute names are case-sensitive.

Why a sysadmin cares

Attributes are the answer to every “show me traces that …” question. Without attributes, the trace UI shows you a flame graph and a duration. With attributes, it shows you the same flame graph sliced by HTTP route, by database, by response status, by customer tenant, by deployment version.

The trade-off is cost. Every distinct attribute value is a key in Tempo’s trace search index. An attribute with 10 000 distinct values per minute is 10 000 new entries in the search index per minute; the index grows, queries slow, retention shrinks. Attributes are the single biggest lever on the operational cost of tracing.

How it works

There are three categories of attribute to know about:

Resource attributes — set once on the TracerProvider, apply to every span it produces. Examples: service.name, service.version, deployment.environment, host.name, k8s.pod.name. Resource attributes are the foundation of every trace search: Tempo groups and filters by them.

Span attributes — set per-span, describe the unit of work. OpenTelemetry publishes semantic conventions for the common ones:

HTTP server span (kind = SERVER)
  http.request.method            = "GET"
  http.route                      = "/orders/{id}"
  http.response.status_code       = 200
  url.path                        = "/orders/8821"
  server.address                  = "orders.internal"
  user_agent.original             = "curl/8.5.0"

Database client span (kind = CLIENT)
  db.system                       = "postgresql"
  db.namespace                    = "shop"
  db.statement                    = "SELECT * FROM orders WHERE id=42"
  db.operation.name               = "SELECT"
  server.address                  = "db.internal"
  server.port                     = 5432

RPC client span (kind = CLIENT)
  rpc.system                      = "grpc"
  rpc.service                     = "payment.PaymentService"
  rpc.method                      = "Charge"

The semantic conventions are a contract between the application and the tracing backend. Auto-instrumentation sets them; queries assume them. If your hand-rolled span sets http_method = "GET" instead of http.request.method = "GET", TraceQL filters written by every other engineer will miss your spans.

Event attributes — set on individual span events (covered in the next lesson). They share the same type system but live underneath the event, not the span.

Attributes are read-only after the span ends. Once a span is exported, its attributes are immutable; changing them requires a new span.

Under the hood

How to configure it

The application side — Python SDK with auto-instrumentation plus one hand-rolled span with custom attributes:

from opentelemetry import trace

tracer = trace.get_tracer("checkout.charge")

with tracer.start_as_current_span("charge") as span:
    # semantic conventions -- these names are part of the contract
    span.set_attribute("rpc.system", "grpc")
    span.set_attribute("rpc.service", "payment.PaymentService")
    span.set_attribute("rpc.method", "Charge")

    # custom attributes -- application-specific
    span.set_attribute("app.payment.amount_minor", 4299)   # in minor units, not floats
    span.set_attribute("app.payment.currency", "GBP")
    span.set_attribute("app.cart.items", 3)

    # record the outcome using the same semantic conventions
    try:
        charge_card(...)
        span.set_attribute("rpc.response.status", "ok")
    except DeclineError as e:
        span.set_attribute("rpc.response.status", "declined")
        raise

A common operational mistake is to log rich context into the span as raw strings: span.set_attribute("request", json.dumps(ctx)). This destroys queryability — TraceQL cannot index into a JSON-encoded blob. Always set attributes as flat typed pairs.

The Collector side — strip known PII before the span reaches storage:

# /etc/otelcol/config.yaml
processors:
  # Drop attribute keys you never want indexed. Cardinality cap.
  attributes/remove_pii:
    actions:
      - key: http.request.header.authorization
        action: delete
      - key: enduser.id
        action: delete
      - key: db.statement
        action: hash
  batch:
    timeout: 5s

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [attributes/remove_pii, batch]
      exporters: [otlp/tempo]

The hash action replaces the value with a deterministic truncated hash, so the attribute is still searchable for joins without retaining the raw PII.

How to validate it

Confirm the attribute is on the span, then confirm Tempo’s search index picks it up.

# 1. Look at one trace by ID and print the span attributes.
TRACE=$(curl -s -X POST http://shop.internal/checkout \
  -H 'Content-Type: application/json' \
  -d '{"cart_id":42}' | jq -r .trace_id)

curl -s -u "$TEMPO_USER:$TEMPO_PASS" \
  "http://tempo.internal:3200/api/traces/$TRACE" \
  | jq '.batches[].scopeSpans[].spans[]
         | select(.name=="charge")
         | {name, attrs: (.attributes | map({(.key): .value}) | add)}'

# Expected:
# {
#   "name": "charge",
#   "attrs": {
#     "rpc.system": "grpc",
#     "rpc.service": "payment.PaymentService",
#     "rpc.method": "Charge",
#     "app.payment.amount_minor": 4299,
#     "app.payment.currency": "GBP",
#     "app.cart.items": 3
#   }
# }

# 2. Search Tempo by an attribute value.
curl -s -G -u "$TEMPO_USER:$TEMPO_PASS" \
  --data-urlencode 'q={ span.rpc.service = "payment.PaymentService" && span.app.payment.currency = "GBP" }' \
  --data-urlencode 'limit=10' \
  --data-urlencode 'since=15m' \
  http://tempo.internal:3200/api/search | jq '.traces | length'

# Expected: a non-zero count of matching trace IDs.

# 3. Search by an HTTP semantic-convention attribute.
curl -s -G -u "$TEMPO_USER:$TEMPO_PASS" \
  --data-urlencode 'q={ span.http.response.status_code = 500 && span.http.route = "/orders/{id}" }' \
  --data-urlencode 'limit=20' \
  http://tempo.internal:3200/api/search | jq '.traces[].traceID'

If step 1 returns the attribute and step 2 returns zero results, the attribute is on the span but not in the search index. The two most common causes are a Collector-side attributes filter stripping it, or the attribute being on a span that the search index does not sample (the index is built from a subset of spans).

How it can fail

1. High cardinality. An attribute like user.id or request.uuid produces one new value per request. After an hour the search index has millions of entries and queries become slow. Symptom: Tempo’s tempo_search_index_result_bytes grows faster than the trace volume would suggest.
2. PII leakage. A naïve developer adds enduser.email = req.user.email to every span. The value is retained for 14 days and exposed to everyone with Tempo read access. Symptom: a security review finds emails in the trace-search index.
3. Namespace drift. Hand-rolled spans set http_method instead of http.request.method. The trace UI shows the value but TraceQL filters written by every other engineer miss it. Symptom: dashboards built on the standard attribute return zero traces for the new service.
4. JSON-encoded blobs. span.set_attribute("ctx", json.dumps(payload)) — queryable as a single string, useless as a search dimension. Symptom: dashboards based on this attribute return “no data” despite every span carrying it.
5. Type confusion. set_attribute("count", "5") vs set_attribute("count", 5). The string version is not numerically queryable in TraceQL. Symptom: numeric range filters return nothing.
6. Redactor over-match. A Collector attributes/delete rule removes an attribute that the application depends on. Symptom: traces show in Tempo but dashboards keyed on the attribute return empty.

How to troubleshoot it

Security implications

Attributes are a long-retention storage class. Treat them as data you would be comfortable with an auditor pulling from a production backup. Strip authorisation headers, session cookies, JWTs, full SQL statements, raw email addresses, and raw IP addresses at the SDK or the Collector — never rely on “nobody queries this attribute” as a defence. The default Collector batch processor does not scrub attributes; the attributes processor does, but only for keys you name explicitly.

Performance implications

The cost of an attribute is dominated by the cardinality of its values. A low-cardinality attribute (http.response.status_code, six values) is essentially free. A high-cardinality attribute (user.id, millions of values) can dominate the trace search index. Concrete rules:

• Keep total attribute cardinality under roughly 1 million distinct values per attribute per day for a healthy Tempo deployment.
• Never put request-scoped identifiers (request.id, correlation.id, user.id) on every span without explicit justification. The temptation is real — they correlate beautifully — but the index will not survive.
• Prefer enums and small vocabularies over free-form strings.

Production guidance

• Standard attributes first. Custom attributes only for application-specific facts.
• Cardinality is a budget. Treat the trace search index like a Prometheus label set: bounded vocabulary, named owners, periodic review.
• Type every attribute at the SDK. Strings for IDs, integers for counts and money in minor units, booleans for flags.

Verification

You should now be able to answer:

• What is the difference between a resource attribute and a span attribute?
• Name three HTTP semantic-convention attributes and what values they take.
• Why is a 128-bit user identifier a poor attribute key?
• How do you write a TraceQL filter that selects every span where http.response.status_code is between 500 and 599?
• Where in the OpenTelemetry pipeline do you strip PII attributes, and why there?

Quiz