Attribute Redaction

A service writes a custom http.route.title attribute by parsing the URL path and storing the human-readable title: "checkout-success". The OTel Collector’s attributes processor is configured with an allowlist that retains http.route, http.method, http.status_code. The title attribute is not in the list, so the collector drops it. Sounds right. The next morning, the security team reports a trace that contains user.email=alice@example.com. The collector is dropping attributes by allowlist, but the synthetic attribute user.email had been added by a configuration that the team thought was upstream, not on the service. The attribute travels through the collector with the allowlist of the service layer too permissive, and lands in the bucket.

The OTel Collector is the right boundary. The lesson is about how to make that boundary actually close.

What attribute redaction means in an OTel pipeline

An OpenTelemetry Collector pipeline runs four stages over each span:

1. Receivers — accept OTLP / Jaeger / Zipkin traffic.
2. Processors — transform, filter, redact, batch.
3. Exporters — send the result to Tempo, Loki, Prometheus, a logging backend.

The redaction processor sits in the processors stage. The canonical processor is attributes. Other relevant processors:

• transform (OTTL) for arbitrary attribute-level policy.
• filter for dropping an entire span, not just an attribute.
• redaction (preview in newer releases) for centrally-curated PII patterns.

The right boundary is the collector. Three reasons:

• Coverage. Every service sends through one (or a few) collectors. The collectors cover first-party and third-party SDKs uniformly.
• Auditability. One config file to review, one config file to test in CI, one config file to diff.
• Disposability. Upgrading the application’s instrumentation is a code change; upgrading the collector is a config reload.

The application SDK still has a responsibility: emit a minimal span. The collector is defence in depth. The lesson above stays the same: the application is the primary defence; the collector catches what the application forgot.

Why a sysadmin cares

Three reasons to invest in collector-level redaction rather than relying on the SDK:

1. Third-party SDKs. A vendor’s auto-instrumentation may emit attributes the team cannot influence. The collector is the only control surface.
2. Fleet uniformity. One attributes/redact processor covers every service the collector accepts. The SDK would need to be modified per-service, per-language.
3. Audit reproduction. The collector config is committed alongside the application config; the redactor can be tested in CI with a synthetic span and a known offender.

How it works — the processor model

   OTLP receiver (otlp)
         |
         |  SpanStream
         v
   +-------------------+
   |  attributes/redact|  Allowlist. Drop or hash fields.
   +---------+---------+
             |
             v
   +-------------------+
   |  transform/policy |  OTTL rules for the
   |                   |  edge cases the allowlist misses.
   +---------+---------+
             |
             v
   +-------------------+
   |  filter/spandrop  |  Drop the entire span if it
   |                   |  carries a forbidden attribute.
   +---------+---------+
             |
             v
   +-------------------+
   |  batch            |  Group for transmission.
   +---------+---------+
             |
             v
   +-------------------+
   |  otlp/tempo       |  Send to Tempo.
   +-------------------+

The processor order matters:

• attributes/redact first. Drop or hash the offenders before any other processor sees them. If transform runs first and copies the attribute, the copy survives the redact step.
• transform/policy next. Handle the edge cases with OTTL. For example, “if db.statement contains an email address, drop the statement attribute.”
• filter/spandrop after. If a span carries a forbidden attribute that the previous processors missed, drop the whole span.
• batch last. A span that has been redacted is the payload; batching at the end groups redacted spans for transmission.

How to configure it

A production-ready redaction pipeline:

# /etc/otelcol/config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  # 1. Allowlist at the attribute level.
  attributes/redact:
    actions:
      - key:           http.route
        action:        retain
      - key:           http.method
        action:        retain
      - key:           http.status_code
        action:        retain
      - key:           service.name
        action:        retain
      - key:           service.version
        action:        retain
      - key:           customer.id_hash
        action:        retain
      - key:           payment.method
        action:        retain
      - key:           payment.outcome
        action:        retain
      # Drop sensitive fields (defence in depth).
      - key:           user.email
        action:        delete
      - key:           request.body
        action:        delete
      - key:           response.body
        action:        delete
      - key:           db.statement
        action:        delete
      - key:           db.connection_string
        action:        delete
      - key:           jwt.claim.sub
        action:        delete
      # Hash where correlation matters.
      - key:           http.url
        action:        hash

  # 2. OTTL rules for the edge cases.
  transform/policy:
    trace_statements:
      - context: span
        statements:
          - replace_all_patterns(span.attributes, "value", "redacted", "[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Za-z]{2,}")
          - delete_key(span.attributes, "customer.name") where span.attributes["customer.name"] != nil

  # 3. Drop entire spans with forbidden attributes.
  filter/spandrop:
    error_mode: ignore
    traces:
      span:
        - 'attributes["request.body"] != nil'
        - 'attributes["db.statement"] != nil and IsMatch(attributes["db.statement"], "(?i)(select|insert|update|delete)\\s+.*\\bfrom\\b")'

  # 4. Batch for transmission.
  batch:
    send_batch_size: 1024
    timeout: 5s

exporters:
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true     # Tempo is on the internal network only

service:
  pipelines:
    traces:
      receivers:  [otlp]
      processors: [attributes/redact, transform/policy, filter/spandrop, batch]
      exporters:  [otlp/tempo]

Severity: CONFIGURATION. Reload the collector with systemctl reload otel-collectord; no service restart required.

The order in the pipeline list matches the order in the configuration. attributes/redact is the first processor; filter/spandrop is third; batch is last. Reversing this order means the redact step runs after batching, which is fine for in-memory batching but breaks when the batch is written to disk for retries (e.g., a file exporter).

Application-side reduction: emit a minimal span

The application’s responsibility is to emit a span that contains only what the operation needs:

ctx, span := otel.Tracer("payments").Start(ctx, "POST /charge",
    trace.WithAttributes(
        attribute.String("http.route",    "/charge"),
        attribute.String("payment.method", req.Method),     // "card", "wallet", not PAN
        attribute.String("payment.outcome", "approved"),   // "approved"/"declined", not response body
        attribute.String("customer.id_hash", hashUserID(req.UserID)),
    ),
)
defer span.End()

customer.id_hash is the SHA-256 of the user ID; same input maps to the same hash, correlable across spans without reversible PII.

How to validate it

Severity: READ-ONLY.

# 1. Send a known-bad span.
otel-cli span export --endpoint otelcol:4317 \
  --service validate-redaction --name "POST /audit" \
  --attrs 'user.email=alice@example.com,request.body=secret-value,http.url=https://app.example.com/orders?session=abcd1234'
# Response includes a trace_id.

# 2. Read the trace back from Tempo and confirm the offender is
# gone or hashed.
trace_id=...
curl -s "http://tempo:3200/api/traces/${trace_id}" \
  | jq '.resourceSpans[].scopeSpans[].spans[].attributes[] | {key: .key}'
# The output should NOT include `user.email`, `request.body`,
# `db.statement`. `http.url` is a hex SHA-256.

# 3. Tap the wire between the collector and Tempo.
# A `tcpdump` on the OTLP port and a search for the email
# confirms the field never left the collector.
tcpdump -A -i any port 4317 2>/dev/null | grep -c 'alice@example.com'
# 0

# 4. The collector's processor counters confirm activity.
curl -s http://otelcol:8889/metrics | \
  grep -E 'otelcol_processor_(attributes|transform|filter)_'
# otelcol_processor_accepted_spans{processor="attributes/redact"}  128173
# otelcol_processor_dropped_spans{processor="attributes/redact"}    42
# otelcol_processor_accepted_spans{processor="filter/spandrop"}    42

# 5. End-to-end: a TraceQL search for the sender address returns
# nothing.
curl -sG http://tempo:3200/api/search \
  --data-urlencode 'q={ span.http.url =~ "abcd1234" }' \
  --data-urlencode 'limit=10' | jq '.traces | length'
# 0

A clean validation: the wire dump shows no offender, the trace record from Tempo has no offender, the collector metrics show attributes/redact dropping 42 expected fields, and the TraceQL search for the sensitive value returns zero traces.

How it can fail

Five recurring shapes.

1. Allowlist order is wrong. attributes/redact runs after batch because of a pipeline reorder. Symptom: the wire dump to Tempo shows the raw attribute. Fix: set processors: [attributes/redact, batch] in the pipeline block; reload the collector.
2. A new offender field is added but the allowlist is not updated. A service starts emitting a new field by default. The allowlist retains only known-safe fields; the new field is dropped. Symptom: the field is silently missing from every trace. Fix: the allowlist is working as intended; the fix is operational — surface the missing field as a finding on the next code review.
3. hash on http.url is sensitive to URL case. The hash function is case-sensitive. Symptom: the same URL with different case produces different hashes. Fix: canonicalise the URL before hashing (lowercase the host, lowercase the scheme, preserve the path).
4. The transform processor fails silently with a bad OTTL expression. A syntax error makes OTTL skip the statement. Symptom: a regex transform that was supposed to redact emails does not fire; the test grep against the wire finds the pattern. Fix: set error_mode: report on the transform processor; surfaces the error in the collector log.
5. The filter processor drops an entire trace by accident. A condition matches a benign attribute. Symptom: the service’s data appears in the trace UI as “missing”. Fix: run the filter in error_mode: report mode for a week; tighten the condition based on what the report surfaces.

How to troubleshoot it

The diagnostic order for “data is appearing in traces that should not be there”:

1. Is the pipeline order correct? otelcol-contrib components lists the active config; the pipeline block is the authoritative source. The order in [attributes/redact, ...] is the order the processors run.
2. Is the field reaching the collector? Wire-dump the receiver port. If the wire has the offender, the application is the source; if not, the collector dropped it.
3. Did the transform processor log an error? The collector’s stdout with error_mode: report surfaces OTTL syntax errors and unsupported expressions.
4. Is the field reaching Tempo? A TraceQL search for a fabricated value confirms whether Tempo has the offender.
5. Is the metrics-generator deriving a new metric that re- exposes the value? The metrics-generator redacts with the same discipline; a derived metric that includes the raw value is a metric leak that is invisible to the trace audit.

Security implications

• The collector is the right boundary. Application SDKs are heterogeneous; the collector is uniform. The application is the primary defence; the collector is the fleet-wide backstop.
• Allowlist over denylist. The collector’s retain action explicitly names what survives. A denylist misses fields the team did not anticipate.
• Pipeline order matters. attributes/redact first, batch last. Reversing this leaves a window during which a bug exposes raw data.
• Hashing over deletion for forensic correlation. When the team needs to group traces by user, hash the identifier. Deleting loses the correlation.

Performance implications

• attributes/redact cost. ~50 ns per attribute. A trace with 200 attributes is ~10 microseconds of overhead. Negligible against the OTLP frame cost.
• hash cost. SHA-256 on a 256-byte input is ~1 microsecond per match. Negligible at production span volumes.
• transform/policy cost. OTTL is interpreted; a regex match is ~10-100 microseconds depending on the pattern. At 1000 spans/sec this is 10-100 ms of CPU. Watch the otelcol_processor_batch_send_size queue for back- pressure.
• filter/spandrop cost. The condition evaluation is one comparison per span. Negligible.

Production guidance

• Allowlist at the collector. The application’s SDK is defence in depth, but the collector is the one place every service passes through.
• attributes/redact first, batch last. Pipeline order is policy.
• Hash for forensic correlation. Delete for true removal. The two are not interchangeable.
• error_mode: report on transform and filter. A silent skip is a leak that the audit cannot find.
• Audit on a schedule. A weekly grep against the collector-to-Tempo wire (or against a sampled set of blocks) catches the offenders before they reach the bucket.

Verification

You should now be able to answer:

• Which OTel Collector processor is the canonical redaction primitive, and how does it differ from the transform processor?
• Why is the collector the right boundary for fleet-wide redaction rather than the application SDK?
• Why does pipeline order matter — what happens if attributes/redact runs after batch?
• When should hash be preferred to delete for a span attribute like http.url?
• What is OTTL, and when does the transform processor beat the attributes processor?

Quiz