Redaction Tools

An engineer ships a new OTel Collector configuration. The configuration includes a transform processor that is supposed to strip a sensitive label from a metric, and the statement calls a function the engineer remembered rather than looked up. The collector parses its OTTL statements when it builds the pipeline, not when the first record arrives, so it never starts: it exits with invalid config for "transform" processor. The deploy rolls back to the previous configuration. The previous configuration has no redaction at all. Nobody notices, because the rollback is reported as a success. The audit grep finds the label a week later.

This lesson is the operational reference for the tools that do the work: the OpenTelemetry Collector attributes, transform, and redaction processors at the OTLP boundary, and the Grafana Alloy loki.process stages at the log line boundary. It also states plainly what the backend stores can and cannot do, because that is where most redaction designs quietly fail.

What redaction tools means

A redaction tool is the operational component that replaces or removes a sensitive value at a specific boundary. The boundary determines the tool:

• OpenTelemetry Collector processors — agent-layer redaction for OTLP metrics, logs, and traces. The attributes processor matches on attribute keys; the transform processor evaluates OTTL statements against paths and values; the redaction processor keeps an allowlist of keys and masks values that match a blocked pattern.
• Grafana Alloy loki.process stages — agent-layer redaction for log lines. stage.replace rewrites the substrings its own expression captures; stage.regex extracts named groups for later stages to act on; stage.luhn masks numbers that pass the Luhn checksum; stage.label_drop removes labels from the stream.
• The backend stores — Loki, Tempo, and Prometheus store what they are given. None of them redacts at query time. Once a value is in a block, the only remedies are access control, retention, and — for Loki alone — a delete request.

The first two boundaries are where redaction happens. The third is where you find out that it did not.

Why a sysadmin cares

Three operational reasons drive the tool choice.

1. Coverage. A field emitted as an OTLP span attribute reaches Tempo. A field emitted in a log line reaches Loki. The tool that catches the field depends on the signal and on how it was shipped, not on where you would prefer to fix it.
2. Failure shape. The attributes and redaction processors are declarative and fail loudly on a bad config. The transform processor runs a language: a statement that does not parse stops the collector from starting, and a statement that parses but never matches does nothing at all and says nothing about it.
3. Irreversibility. The agent is the last boundary that can change the record. Everything after it is retention policy and access control. Getting the agent rule right is cheaper than every downstream remedy combined.

How it works

The mental model. The record crosses two boundaries where it can still be edited, and then it is fixed.

   OpenTelemetry Collector (OTLP metrics, logs, traces)
       |
       |  otlp receiver
       |    -> attributes processor  match on key, replace the value
       |    -> transform processor   OTTL statements over paths and values
       |    -> redaction processor   allowlist of keys, mask blocked values
       |    -> exporters
       v
   Grafana Alloy (log lines)
       |
       |  loki.source.* -> loki.process -> loki.write
       |    stage.regex        capture a named group
       |    stage.replace      rewrite the captured group
       |    stage.luhn         mask numbers passing the Luhn checksum
       |    stage.label_drop   remove labels from the stream
       v
   Backend store (Loki / Tempo / Prometheus)
       |
       |  no redaction here: the store keeps what it was given
       v
   Query (Grafana)

The two agent pipelines are independent. A span attribute is only ever seen by the collector; a log line tailed from a file is only ever seen by Alloy. A field that escapes both is in the store.

How to configure it

The tools, with a verified configuration for each.

Tool 1 — OpenTelemetry Collector attributes processor

The simplest tool. Matches on the attribute key; updates the value in place.

# /etc/otelcol/config.yaml
processors:
  attributes/redact:
    actions:
      - key: authorization
        action: update
        value: "[REDACTED]"
      - key: password
        action: update
        value: "[REDACTED]"
      - key: pan
        action: update
        value: "[REDACTED]"
      - key: api_key
        action: update
        value: "[REDACTED]"
      - key: ssn
        action: update
        value: "[REDACTED]"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [attributes/redact, batch]
      exporters: [otlp/tempo]
    logs:
      receivers: [otlp]
      processors: [attributes/redact, batch]
      exporters: [otlphttp/loki]

The actions list is evaluated in order. The valid actions are insert, update, upsert, delete, hash, extract, and convert — there is no allowlist action, so a processor that must keep only known-safe keys is a keep_keys statement in the transform processor or an allowed_keys list in the redaction processor instead.

update only rewrites a key that already exists, which is the behaviour you want: a service that never emits ssn does not acquire an ssn attribute reading [REDACTED].

Tool 2 — OpenTelemetry Collector transform processor

The more flexible tool. Matches on paths and values through OTTL.

processors:
  transform/redact:
    error_mode: ignore
    trace_statements:
      - context: span
        statements:
          # Bearer tokens in the authorization header attribute.
          - replace_pattern(attributes["http.request.header.authorization"], "Bearer\\s+[A-Za-z0-9._\\-/+=]+", "Bearer [REDACTED]")
          # Card-shaped digit runs anywhere in a free-text attribute.
          - replace_pattern(attributes["db.statement"], "\\b[0-9]{4}[ -]?[0-9]{4}[ -]?[0-9]{4}[ -]?[0-9]{4}\\b", "[REDACTED]")
          # Drop whole attributes by exact key and by pattern.
          - delete_key(attributes, "session_id")
          - delete_matching_keys(attributes, "(?i).*(password|api_key|secret|token).*")
    metric_statements:
      # Metric labels are attributes on data points, not on the metric.
      - context: datapoint
        statements:
          - delete_matching_keys(attributes, "(?i).*(password|api_key|secret|token).*")
    log_statements:
      - context: log
        statements:
          # `body` is a log record field. Spans do not have one.
          - replace_pattern(body, "Bearer\\s+[A-Za-z0-9._\\-/+=]+", "Bearer [REDACTED]")

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [transform/redact, batch]
      exporters: [otlp/tempo]

Two placement rules are easy to get wrong and expensive to debug:

• body belongs to the log context. Writing replace_pattern(body, ...) under context: span is a parse failure, because the span context has no body path. The collector refuses to start.
• A metric’s labels are attributes on its data points. Under context: metric you can reach name, description, and unit, but not the label set. Use context: datapoint.

Tool 3 — OpenTelemetry Collector redaction processor

The fail-closed backstop. Where the other two processors name what to remove, this one names what to keep.

processors:
  redaction/allowlist:
    # An empty allowed_keys list with allow_all_keys false
    # removes every attribute. The default is to fail closed.
    allow_all_keys: false
    allowed_keys:
      - service.name
      - service.version
      - http.route
      - http.request.method
      - http.response.status_code
      - db.system
    # Allowed unconditionally, and never value-checked.
    ignored_keys:
      - deployment.environment
    # Values of allowed keys matching these are masked.
    blocked_values:
      - '4[0-9]{12}(?:[0-9]{3})?'   # Visa
      - '(5[1-5][0-9]{14})'         # MasterCard
    # debug lists redacted key names; info gives counts only;
    # silent adds nothing. A key name can itself be sensitive.
    summary: info

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [redaction/allowlist, batch]
      exporters: [otlp/tempo]

The processor removes every attribute that is not on allowed_keys or ignored_keys, then masks the parts of the surviving values that match blocked_values with a fixed-length run of asterisks. With summary set to info or debug it adds redaction.redacted.count, redaction.masked.count, and redaction.ignored.count attributes to each record, which is what the validation ladder below reads.

The trade-off is the trade-off of every allowlist: a new attribute that the platform team has not added to allowed_keys disappears without warning. That is the correct default for a security control and a nuisance for everything else, so run it on the pipelines that carry regulated data and not on the whole fleet by reflex.

Tool 4 — Grafana Alloy loki.process stages

The log-line equivalent. Rewrites substrings and drops labels.

// /etc/alloy/config.alloy
loki.process "redact" {
  // 1. Mask numbers that pass the Luhn checksum. This is the
  //    right tool for card numbers: it checks the digits
  //    rather than trusting a length-and-shape regex.
  stage.luhn {
    min_length  = 13
    replacement = "[REDACTED]"
  }

  // 2. Replace the captured group after "Bearer ".
  //    `expression` is required; the capture group is what
  //    `replace` substitutes.
  stage.replace {
    expression = "Bearer\\s+([A-Za-z0-9._\\-/+=]+)"
    replace    = "[REDACTED]"
  }

  // 3. Drop labels before the entry is written to Loki.
  stage.label_drop {
    values = ["password", "token", "api_key", "session_id"]
  }

  forward_to = [loki.write.default.receiver]
}

The block is stage.label_drop, with an underscore — there is no stage.labeldrop in Alloy. The stage.replace argument is replace, not replacement; stage.luhn is the one stage here whose argument really is called replacement. The stages run in the order they are written, and a later stage sees the line the earlier stages produced.

Tool 5 — Promtail pipeline stages

Promtail reached end of life on 2 March 2026 and receives no further updates. Treat this section as a reading aid for configurations you have inherited, and migrate them with alloy convert --source-format=promtail.

# /etc/promtail/config.yaml
scrape_configs:
  - job_name: application
    static_configs:
      - targets: [localhost]
        labels:
          job: application
    pipeline_stages:
      - replace:
          expression: 'Bearer\s+([A-Za-z0-9._\-/+=]+)'
          replace: '[REDACTED]'
      - labeldrop:
          - password
          - token
          - api_key

The stage names are not the same as Alloy’s. Promtail’s label-dropping stage is labeldrop, one word; Alloy’s block is stage.label_drop. Promtail’s replace stage requires expression and takes an optional source and an optional replace; there is no replacement key. Assuming the two configuration formats are interchangeable is how a migrated pipeline silently stops redacting.

There is no backend tool

Loki, Tempo, and Prometheus do not redact at query time. There is no data source setting, no grafana.ini section, and no provisioning endpoint that masks values on their way to a panel. A design that assumes one exists is a design with no last line of defence.

What the backend can actually do, once a value has landed:

• Restrict who can query it. Grafana data source permissions and Loki’s per-tenant X-Scope-OrgID separation limit the audience. They do not remove the value.

• Shorten how long it lives. A stream-scoped retention period bounds the exposure window.

• Delete it, in Loki only. With TSDB as the index store, retention_enabled: true on the compactor, a delete_request_store configured, and deletion_mode set to filter-only or filter-and-delete, the compactor accepts delete requests:

  curl -X POST -G 'http://loki:3100/loki/api/v1/delete' \
    -H "X-Scope-OrgID: prod" \
    --data-urlencode 'query={job="application"} |= "Bearer "' \
    --data-urlencode 'start=2026-08-17T00:00:00Z' \
    --data-urlencode 'end=2026-08-18T00:00:00Z'
  # 204 No Content on success. The request is applied after
  # the cancellation period, 24h by default.

  Tempo has no equivalent. A span attribute that reaches Tempo stays there until the block ages out.

How to validate it

The validation ladder for “the agent-layer rules are live”.

# 1. The collector started at all. An OTTL statement that does
#    not parse prevents this, so a running collector is the
#    first piece of evidence that the transform config is
#    syntactically valid.
journalctl -u otelcol --since '5 min ago' | grep -i 'invalid config'
# (no output)

# 2. The processors are seeing records.
curl -s http://otelcol:8888/metrics | grep otelcol_processor_accepted_spans
# otelcol_processor_accepted_spans{processor="transform/redact"} 42

# 3. Emit a known-bad span and read it back from Tempo.
otel-cli span --service test --name redaction-check \
  --endpoint otel-collector:4317 \
  --attrs 'http.request.header.authorization=Bearer eyJhbGciOi...,session_id=abcd1234'
# Look the trace up in Tempo. The authorization attribute
# should read "Bearer [REDACTED]" and session_id should be
# absent entirely.

# 4. The redaction processor reports what it removed.
#    With summary: info, every record carries the counters.
curl -s "http://tempo:3200/api/traces/${trace_id}" \
  | jq -r '.. | .attributes? // empty | .[] | select(.key|startswith("redaction."))'
# redaction.redacted.count = 2

# 5. Push a known-bad line through Alloy, not directly into
#    Loki. A direct push to the Loki API bypasses loki.process
#    entirely and proves nothing about the pipeline.
echo 'auth header: Bearer eyJhbGciOi... card 4111111111111111' \
  >> /var/log/application/test.log
logcli query --since=5m '{job="application"}' --output=raw
# auth header: Bearer [REDACTED] card [REDACTED]

# 6. The dropped labels are absent from the stream.
logcli series --since=5m '{job="application"}'
# (no api_key, password, token or session_id label)

Step 5 is the step teams skip. Pushing a test line straight to /loki/api/v1/push exercises Loki, not the redaction rule, and a green result from that test is meaningless.

How it can fail

Five recurring failure modes. Each maps to a recognisable symptom.

1. An OTTL statement does not parse. A function that does not exist, or a path the context does not have — body under context: span is the classic one. Symptom: the collector will not start and logs invalid config for "transform" processor. Risk: a deploy pipeline that rolls back on a failed start restores a configuration with no redaction, and reports the rollback as a success.
2. A statement parses but never matches. The attribute key in the config is not the key the service emits. Symptom: nothing in the logs, nothing in the metrics, and the value in the store. Fix: compare the config against a real record, not against the semantic conventions.
3. The regex matches too broadly. A card regex that matches any 13-to-19-digit run catches epoch timestamps in microseconds. Symptom: timestamps are replaced with [REDACTED] and engineers cannot read the logs. Fix: anchor to the field, or use stage.luhn, which validates the checksum rather than the shape.
4. The Alloy stages are in the wrong order. stage.label_drop runs before the stage that copies the label value into the line. Symptom: the line is missing the context the stage was supposed to preserve. Fix: the order.
5. The processor is on the wrong pipeline. attributes/redact is listed under traces but not under logs. Symptom: traces are clean, logs are dirty. Fix: add the processor to every pipeline that carries the signal, and check the service.pipelines block rather than the processors block, because defining a processor and never referencing it is silent.

How to troubleshoot it

The diagnostic order for “the redaction is not working”:

1. Which tool is supposed to handle this signal? An OTLP attribute is the collector; a tailed log line is Alloy. Nothing after the agent can help.
2. Did the collector start? An unparseable OTTL statement is a startup failure, and a rolled-back deploy looks like a healthy one.
3. Is the rule matching? Enable the collector’s debug logging via service.telemetry.logs.level: debug; the OTTL troubleshooting output prints the statement and the record it ran against.
4. Is the rule on the pipeline? Check service.pipelines, not the processor definitions.
5. Where does the value first appear? Check the wire, then the agent’s output, then the store. The first boundary that has it is the boundary that failed.

Security implications

The implementation details that make the tooling auditable:

• The collector configuration is in version control, and a PR that changes a processor rule is approved by the platform team. A processor that is defined but not referenced in service.pipelines is a review finding.
• The Alloy configuration is generated from a template, and the template is reviewed. The generated file is loaded in CI before it is deployed.
• error_mode is set explicitly on every transform processor. silent is not used in production.
• The summary setting on the redaction processor is info rather than debug on pipelines carrying regulated data, because the list of redacted key names is itself a disclosure.
• The audit grep runs against every layer: the wire payload, the collector’s debug output, and the content of the stores.

Performance implications

The tools have different cost profiles, and all of them are cheap relative to the cost of the leak they prevent.

• The attributes processor is a map lookup per configured key per record. It does not scale with record size.
• The transform processor costs one regex evaluation per replace_pattern statement per record. Regex cost dominates and is a property of your pattern, not of the collector: an anchored pattern is cheap, an unanchored alternation over a large body is not.
• delete_matching_keys evaluates the pattern against every key in the map, so it is proportional to attribute count. Prefer delete_key where the key is known exactly.
• stage.luhn scans the line for digit runs and checksums them, which is cheaper than an unanchored card regex and materially more accurate.

Measure with the collector’s own telemetry rather than estimating. The expensive failure shape is not CPU: it is a pattern that matches too broadly, leaving a log line where every field reads [REDACTED].

Verification

You should now be able to answer:

• What is the difference between the OTel Collector attributes processor and the transform processor?
• Which OTTL context holds a metric’s label set, and which editor removes a key from it?
• Why is stage.luhn a better tool for card numbers than a digit-shaped regex?
• What can a backend store do about a sensitive value that has already been ingested, and what can it not do?

Quiz