Trace Data Sensitivity

A developer adds an OpenTelemetry interceptor to a payments API to trace the slow path. The interceptor reads the incoming request, attaches it as a single span attribute called request.body, and exports the trace. The interceptor runs for ninety minutes before a code review flags it. In that window, every authenticated payment request, complete with bearer tokens and PAN-shaped payloads, has been flushed to the Tempo bucket. The data sits there for the configured retention period, queryable from Grafana by anyone with viewer access. The remediation is a hot-patch, a credential rotation across the affected user base, a legal disclosure, and a forensic preservation of the affected blocks.

This lesson is about preventing that incident. Traces can carry PII in attributes. The right redaction is at the source. The most common shape is the SDK auto-instrumentation that captures attributes by default and the application that adds a request.body field for debugging.

What data sensitivity in traces means

A Tempo trace is a set of OTLP spans. Each span carries:

• Resource attributes. Identity of the producing service (service.name, service.version, k8s.pod.name). Almost always operational metadata. Safe.
• Scope attributes. Identity of the producing library (otel.library.name, otel.library.version). Operational. Safe.
• Span attributes. Properties of the operation the span represents. This is the surface where PII hides.
• Span events. Named log lines attached to the span, each with its own attributes. Same risk profile as span attributes.
• Status and links. Safe.

The data-sensitivity surface is the union of span attributes and event attributes. Every field added to a span is a field that ends up in a Tempo block in object storage, queryable from Grafana, retained for block_retention. There is no field-level encryption. There is no per-attribute access control. The attribute exists or it does not.

   OTLP span (one "step" of a trace)
   +-------------------------------------------+
   | resource attrs (safe, operational)         |
   +-------------------------------------------+
   | scope attrs (safe, operational)            |
   +-------------------------------------------+
   | name + kind + start/end (safe)             |
   +-------------------------------------------+
   | attributes (THE sensitive surface)         |
   |   - http.url        <-- can carry PII      |
   |   - user.email      <-- direct PII         |
   |   - jwt.claim       <-- secret             |
   |   - db.statement    <-- secret via SQL      |
   +-------------------------------------------+
   | events (each event has its own attrs)      |
   +-------------------------------------------+

Why a sysadmin cares

Three operational pains converge here.

1. Compliance. GDPR Article 5(1)(c) requires that personal data be adequate, relevant, and limited to what is necessary. A user.email attribute on every span of every request is the opposite. PCI DSS Requirement 3.4 forbids storing PAN in unencrypted form anywhere in the cardholder data environment; a card.number attribute is a finding on the next audit.
2. Blast radius. Once a block is in the bucket, the block stays for the retention window. A leak at hour 1 is still a leak at hour 720 of a 30-day retention. Tempo has no in-place edit; you cannot retroactively scrub an attribute from a block.
3. Forensic value. A trace with a user.id attribute is investigation-grade. A trace with the same user.id next to an unhashed session_token is a token leak that breaks the chain of custody.

The lesson returns to the same data-protection rules that applied to logs. The only difference is that traces are structured by default. Logs are text. Traces have field names. The wrong assumption is “the field name is innocuous; the value is innocuous too.” The right assumption is “the field name is enough to flag the value as sensitive until proved otherwise.”

How it works — the sensitive surface in detail

The most common offenders in production traces, in roughly the order they appear during a code review:

• URLs. http.url, url.full. Path and query string of an authenticated request often carry session IDs, OAuth callback codes, password-reset tokens, or signed invitation tokens. Even a benign-looking ?id=... can be a key into an internal admin route.
• Headers as attributes. http.request.header.authorization, http.request.header.cookie. The OpenTelemetry HTTP semantic conventions explicitly say these should not be recorded by default — but a custom interceptor often does.
• Request body. request.body, http.request.body. The payload of a POST/PUT. Trivially the entire PII surface.
• Response body. response.body, http.response.body. Less obvious, but equally sensitive for endpoints that echo identifiers back into the trace.
• Database query text. db.statement. The literal SQL the service ran. SELECTs with literals, INSERTs with values, error messages with input. Connection strings (db.connection_string) and statement parameters (db.statement.parameters) are the same surface.
• JWT claims. jwt.claim.*, auth.claims.*. Decoded JWT payloads are not secrets but their sub, email, and name claims are direct PII; the raw token is a credential.
• GraphQL variables. graphql.variables. Often mirrors the request body.
• Custom business fields. customer.email, order.shipping_address, ticket.body, comment.author. These are the developer-added fields that look fine in code review but carry full payloads in production.

A trace with any of the above fields carries a copy of what the user submitted or what the application returned. The block sits in the bucket. The Grafana viewer sees it.

How to configure it

The defence-in-depth model: application, SDK, OTel Collector, Tempo, Grafana. Each layer is one part of the chain.

Layer 1 — application: emit a minimal span

// The application picks the fields; nothing else is emitted.
ctx, span := otel.Tracer("payments").Start(ctx, "POST /charge")
span.SetAttributes(
    attribute.String("http.route", "/charge"),
    attribute.String("customer.id_hash", hashUserID(req.UserID)),
    attribute.String("payment.method", req.Method),     // "card", not the PAN
    attribute.String("payment.outcome", "approved"),   // "approved"/"declined", not the response body
)
span.End()

customer.id_hash is the SHA-256 of the user identifier. It is investigation-grade (same input → same hash, correlable across lines) but not reversible. The PAN never enters the span; only the outcome does.

Layer 2 — OTel Collector: allowlist with attributes processor

The OTel Collector’s attributes processor runs on every span flowing through it. The allowlist approach keeps only the fields the team has reviewed; everything else is dropped.

# /etc/otelcol/config.yaml
processors:
  attributes/redact:
    actions:
      # Allowlist: only these span attribute keys survive.
      - key: http.route
        action: retain
      - key: http.method
        action: retain
      - key: http.status_code
        action: retain
      - key: customer.id_hash
        action: retain
      - key: payment.method
        action: retain
      - key: payment.outcome
        action: retain
      - key: service.name
        action: retain
      - key: service.version
        action: retain

      # Explicit deletes for known offenders (defence in depth).
      - key: user.email
        action: delete
      - key: request.body
        action: delete
      - key: response.body
        action: delete
      - key: db.statement
        action: delete
      - key: db.connection_string
        action: delete
      - key: http.url
        action: hash

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch, attributes/redact]
      exporters: [otlp/tempo]

The hash action on http.url replaces the value with its SHA-256 hex digest; the path-without-query is lost but the same URL still correlates.

Layer 3 — Tempo: lock the bucket and the querier

# /etc/tempo/tempo.yaml (only the security-relevant excerpt)
auth_enabled: true

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 127.0.0.1:4317   # only the collector reaches this
          tls:
            cert_file: /etc/tempo/tls/tempo.crt
            key_file:  /etc/tempo/tls/tempo.key

storage:
  trace:
    backend: s3
    s3:
      bucket: tempo-traces-prod
      # ... region, SSE-KMS, forcepathstyle as required

Loopback-bind the OTLP receiver: only the OTel Collector (on the same network) can reach it. mTLS (client_ca_file) is added if the collector identity is service-mesh-issued.

Layer 4 — Grafana: lock the data source

In Grafana, the Tempo data source uses a service account token, not a viewer token. Viewer access to Grafana then proxies to Tempo with that service account; viewer access cannot directly talk to Tempo. The Tempo API is reachable only from the Grafana pod subnet.

How to validate it

Severity: READ-ONLY.

# 1. Inspect the collector's effective configuration.
otelcol-contrib components   --exceptions=false | grep -i attributes

# 2. Confirm the egress payload no longer carries the offender.
# Send a synthetic span that includes the bad field, then check.
otel-cli span export --endpoint otelcol:4317 \
  --service validate-redaction --name "POST /audit" \
  --attrs user.email=alice@example.com,request.body=secret
# Response includes a trace_id.

trace_id=...
# 3. Retrieve the trace back and confirm the field is gone or hashed.
curl -s "http://tempo:3200/api/traces/${trace_id}" | \
  jq '.resourceSpans[].scopeSpans[].spans[].attributes[] | select(.key | test("user.email|request.body|http.url"))'
# The `update this field` block is empty. `http.url` is hashed.

# 4. Run a TraceQL search for the offender pattern to confirm
# it does not match.
curl -sG http://tempo:3200/api/search \
  --data-urlencode 'q={ span.http.url =~ "alice@example.com" }' \
  --data-urlencode 'limit=10' | jq '.traces | length'
# 0

# 5. Audit a sample of recent blocks for sensitive data.
# This grep runs against a Tempo block on disk or via the
# trace export API.
for trace_id in $(curl -sG http://tempo:3200/api/search \
                 --data-urlencode 'q={ resource.service.name = "validate-redaction" }' \
                 --data-urlencode 'limit=20' | jq -r '.traces[].traceID'); do
  curl -s "http://tempo:3200/api/traces/${trace_id}" \
    | jq -r '.resourceSpans[].scopeSpans[].spans[].attributes[].key' \
    | grep -E '^user\.email$|^request\.body$|^db\.statement$|^http\.url$'
done
# (no output)

A clean validation: synthetic spans that include a known offender come out of the collector without the offender field, Tempo records the span without it, and a TraceQL search for the offender content matches zero traces.

How it can fail

Six recurring shapes from real audits and incident reviews.

1. Default request-body capture in auto-instrumentation. A middleware framework ships with a hook that attaches the incoming request body to the span. Symptom: every authenticated request payload sits in request.body. Fix: disable the hook at the SDK; add an explicit attributes processor step to delete request.body.
2. http.url carries the session ID. A link such as https://app.example.com/orders?session=... is captured as http.url. Symptom: URL traces are searchable by the session token. Fix: action: hash on http.url in the collector; or strip the query string before trace creation.
3. Database driver emits SQL with parameters inlined. The db.statement is configured to use the formatted query, not the parameterised one. Symptom: SELECTs with literal PII appear as span attributes. Fix: switch the driver to parameterised statements and disable query-text capture (db.statement).
4. JWT payload decoded into a span attribute. A custom middleware parses the bearer token and attaches the claims (jwt.claim.email, jwt.claim.sub). Symptom: every authenticated request has a jwt.claim.email field. Fix: do not parse the token inside the trace boundary; use request-scoped correlation IDs instead.
5. Trace exporter retries write spans to local WAL with raw data. The Tempo distributor’s WAL or an OTel Collector file exporter stores the unfiltered span before re-transmission. Symptom: an audit of the WAL directory finds the offender. Fix: set the file exporter buffer to drop, not retry; run the audit on the WAL path regularly.
6. A third-party SDK exports customer.email by default. A SaaS service library adds the configured customer’s email as a resource attribute. Symptom: every span in the affected service has the offender field. Fix: add the attribute to the collector delete list and override the resource attribute with a random identifier.

How to troubleshoot it

The diagnostic order for “we may have a leak”:

1. Is the field set by the SDK or by the application? Read the SDK configuration first. Auto-instrumentation libraries have explicit toggle lists for request-body, response-body, and SQL capture.
2. Is the field reaching the collector? Tap the receiver port with tcpdump or enable the collector’s debug log for the OTLP receiver. A field on the wire is a SDK / application issue. A field absent on the wire and present in Tempo is a collector issue.
3. Is the field reaching Tempo from the collector? Use the collector’s logging exporter in a non-production pipeline to dump a known-bad span. Compare the dumped attributes to the collector’s attributes/redact allowlist.
4. Is the field in a block already? Run a TraceQL search for a sentinel value, e.g. a fabricated email address, that the synthetic test sent in the last hour. If it matches, blocks in the bucket contain the offender; the audit surfaces the leak window.
5. What is the leak window? Compare the timestamp of the last scrubber patch to the timestamp of the earliest leaked block. The window is the difference.
6. Who needs to be told? Legal and security own the disclosure decision. Engineering can limit the window; legal decides the disclosure.

Security implications

• Pre-ingestion is the only edit window. Tempo does not redact after the block is written. Every redaction rule lives upstream of the distributor.
• The collector is the right boundary for a fleet. The application SDK cannot be trusted uniformly across first-party and third-party libraries. The collector is a single, auditable point.
• Allowlist over denylist. A denylist misses new fields. An allowlist drops any field the team has not reviewed.
• hash over delete for forensic correlation. Where the team needs to group traces by an identifier, hash it; deleting loses the correlation.

Performance implications

• The attributes processor is roughly 50 ns per span attribute at the collector. A trace with 200 attributes incurs about 10 microseconds of overhead. This is negligible against the cost of OTLP gRPC and against the cost of writing the block to S3.
• Allowlisting is cheaper than denylisting. The processor walks the allowlist per attribute and skips the field name match; for 1000 spans/sec with 50 attributes each, this is ~5 ms of CPU per second on the collector.
• Hashing adds SHA-256 cost per match. ~1 microsecond per hash on a modern x86 core. At 1000 hashes/sec this is 1 ms of CPU. Negligible.

Production guidance

• Allowlist at the OTel Collector. Do not rely on the SDK or the application. The collector is the one place every service has to pass through. The allowlist is auditable.
• Hash for forensic correlation. Where the team needs to group by user, use hash; do not delete.
• Audit on a schedule. A weekly grep against the egress span attributes catches new offenders before they reach the bucket.
• Pair with retention. A short block_retention bounds the blast radius of a leak that was missed. The audit grep finds the offender faster than the leak is forgotten.

Verification

You should now be able to answer:

• Which OTLP span surface carries PII — resource attributes, scope attributes, span attributes, events, or status?
• Why cannot a Tempo operator redact a span attribute after the block is written?
• Which OTel Collector processor is the canonical place to scrub span attributes?
• What is the trade-off between delete and hash on a span attribute named http.url?
• Why is allowlisting at the collector preferable to denylisting at the SDK?

Quiz