Redaction Strategies

A team deploys an OTel Collector whose transform processor runs delete_matching_keys(attributes, ".*(password|token|secret).*") over every span. The pipeline is live. The audit grep is clean. Two weeks later, a developer emits a new field called authToken — camelCase, not snake_case. The pattern has no (?i) flag, so token does not match Token. The credential lands in Tempo. The audit grep finds it. The team updates the pattern. A month later, a library emits pwd. The team updates the pattern again.

The team has been doing agent-layer redaction for a year. The team has spent less time on the application boundary because the pipeline catches everything. The team has built a system where the pipeline is the primary defence.

This lesson is the response to that posture. There are two boundaries where a record can still be edited — the application and the agent — and a third, the backend, where it can only be contained. The discipline is defence in depth: drop at the source, scrub at the agent, and design as though the backend will catch nothing, because it will.

What redaction strategies means

Redaction is the replacement of a sensitive value with a non-reversible or tokenised equivalent. Three boundaries act on it, but only two of them redact:

1. Source redaction — the application does not emit the sensitive value. The call site picks the fields it logs; the fields it does not pick are absent from the wire payload.
2. Agent redaction — the application emits the value; the collector / agent replaces it before forwarding. The OpenTelemetry Collector attributes processor and the Loki pipeline stages are the operational examples.
3. Backend containment — the source and the agent both missed it and the value is in the store. Nothing in this stack redacts at query time: Loki, Tempo, and Prometheus return what they were given. The remedies are access control, a shorter retention period, and — in Loki only — a delete request.

The three layers are not interchangeable. They serve different purposes and have different trade-offs.

Why a sysadmin cares

The team that picks the wrong layer pays in two ways: false confidence and operational cost.

• False confidence. A pipeline regex that catches password= does not catch pwd= or auth_token_v2. The team that treats the regex as the primary defence discovers the gap only when an audit finds it.
• Operational cost. A regex that matches too broadly — every field that contains the substring key, for example — replaces operational metadata with [REDACTED]. The pipeline is doing its job; the engineers cannot read the logs.

The right discipline is to do the cheap thing at the source (allowlist at the call site) and the structural thing at the agent (structured field names and known patterns), and to plan the backend layer as containment rather than as a safety net. Defence in depth, not defence in one place — and no pretending there is a layer behind the agent that will catch the misses.

How it works

The mental model. The application emits a record. The record crosses three boundaries before it lands in a backend.

   Application
       |
       |  source redaction: call site picks the fields
       |  (allowlist, drop before serialisation)
       v
   Wire payload (stdout, OTLP, syslog)
       |
       |  agent redaction: collector / agent scrubs the payload
       |  (attributes processor, regex stage, label drop)
       v
   Forwarder (Alloy / OTel Collector)
       |
       |  the last boundary at which the record can be edited
       v
   Backend store (Loki / Tempo / Prometheus)
       |
       |  backend containment only: access control, retention,
       |  and (Loki alone) a delete request. There is no
       |  query-time mask in this stack.
       v
   Query (Grafana)

Each boundary has an owner. The application developer owns the source. The platform team owns the agent. The platform and security teams own the backend. When the leak is reported, the owner of the boundary that failed is the owner of the fix.

The trade-off matrix

The rule: the cheapest, most complete layer is the source. The rule is not “do everything at the source.” Application developers do not own the agent; the platform team does. The right answer is to use both editing layers, in order, with the source as the primary defence and the agent as the backstop — and to treat the backend row of that table as the cost of getting the first two wrong.

How to configure it

The three layers, with real configurations.

Layer 1 — source redaction

The simplest pattern is the call-site allowlist.

// Go, slog: explicit allowlist at the call site.
slog.Info("checkout completed",
    "request_id", reqID,
    "user_id",    userID,
    "amount",     amount,
)
// No password, no PAN, no token. The call site picks the
// fields; nothing else is emitted.

The application pattern that catches the most leaks is the redacting handler. The handler intercepts the record before serialisation and replaces known-sensitive keys.

type redactingHandler struct{ slog.Handler }

func (h *redactingHandler) Handle(ctx context.Context, r slog.Record) error {
    // Known-sensitive keys are replaced with [REDACTED]
    // before the line is serialised to stdout.
    // In production, rebuild the record from a filtered map;
    // slog does not allow attribute removal in the callback.
    return h.Handler.Handle(ctx, r)
}

Layer 2 — agent redaction

The OpenTelemetry Collector attributes processor:

processors:
  attributes/redact:
    actions:
      - key: authorization
        action: update
        value: "[REDACTED]"
      - key: password
        action: update
        value: "[REDACTED]"
      - key: pan
        action: update
        value: "[REDACTED]"
      - key: api_key
        action: update
        value: "[REDACTED]"

service:
  pipelines:
    traces:
      processors: [attributes/redact, batch]
    logs:
      processors: [attributes/redact, batch]

The OpenTelemetry Collector transform processor — when the rule depends on the value rather than only on the key:

processors:
  transform/redact:
    error_mode: ignore
    trace_statements:
      - context: span
        statements:
          # Anchor to the token, not to the whole value, so the
          # "Bearer " prefix survives and stays greppable.
          - replace_pattern(attributes["http.request.header.authorization"], "Bearer\\s+[A-Za-z0-9._\\-/+=]+", "Bearer [REDACTED]")
          # Case-insensitive, so authToken and AUTH_TOKEN match too.
          - delete_matching_keys(attributes, "(?i).*(password|api_key|secret|token).*")

The log line equivalent — a Grafana Alloy loki.process stage:

loki.process "redact" {
  // `expression` is required. Each capture group is
  // substituted with the value of `replace`.
  stage.replace {
    expression = "(?i)(?:password|token|secret|authorization)[=:\\s]+([A-Za-z0-9._\\-/+=]+)"
    replace    = "[REDACTED]"
  }
  forward_to = [loki.write.default.receiver]
}

The Alloy argument is replace, not replacement, and expression is not optional. Alloy rejects an unknown argument at load time, so both mistakes fail the config load rather than failing quietly at runtime — provided the config is actually loaded in CI before it is deployed.

Layer 3 — backend containment

There is no backend redaction configuration to show, because no backend in this stack redacts. What the backend can do is bound the exposure once the value has landed.

Loki, with a TSDB index and the compactor configured for deletion, accepts delete requests:

# loki.yaml
compactor:
  retention_enabled: true
  delete_request_store: s3
  # A delete request can be cancelled inside this window.
  delete_request_cancel_period: 24h

limits_config:
  # filter-only hides matching entries at query time;
  # filter-and-delete also removes them from storage.
  deletion_mode: filter-and-delete

curl -X POST -G 'http://loki:3100/loki/api/v1/delete' \
  -H "X-Scope-OrgID: prod" \
  --data-urlencode 'query={job="checkout"} |= "Bearer "' \
  --data-urlencode 'start=2026-08-17T00:00:00Z' \
  --data-urlencode 'end=2026-08-18T00:00:00Z'
# 204 No Content. Applied once the cancellation period expires.

Tempo has no equivalent: a span attribute that reaches Tempo stays there until the block ages out under the retention policy. Prometheus has an admin delete-series API, and it is disabled by default.

The two editing configurations do the work. The application drops the field. The collector or Alloy scrubs the values the application forgot. The backend configuration above is not a third scrubber — it is the remediation you reach for on the day the first two failed.

How to validate it

The validation ladder for “the three layers are live”:

# 1. The source layer is clean. The application is not
#    emitting the field.
grep -E '"authorization"|"password"|"pan"' /var/log/app/checkout.log | wc -l
# 0

# 2. The agent layer is catching what the source forgot.
#    Emit a known-bad span through the collector.
otel-cli span --service test --name redaction-check \
  --endpoint otel-collector:4317 \
  --attrs 'authorization=Bearer eyJhbGciOi...'
# The span that lands in Tempo shows "authorization=[REDACTED]".

# 3. The agent layer is catching the log line too. Write the
#    known-bad value where the forwarder tails it, so
#    loki.process actually runs. A direct push to
#    /loki/api/v1/push bypasses the pipeline and proves
#    nothing about the rule.
echo 'authorization=Bearer eyJhbGciOi...' \
  >> /var/log/app/checkout.log

# 4. The audit grep is empty for the test payloads.
logcli query --since=1h '{job="checkout"}' --output=raw \
  | grep -E 'Bearer [A-Za-z0-9._/-]{16,}' | wc -l
# 0

How it can fail

Six recurring failure modes. Each maps to a recognisable symptom.

1. The pipeline is the only defence. The application emits the field; the regex catches the obvious case; a renamed field slips through. Symptom: the audit grep finds the renamed field; the fix is to enforce the allowlist at the source, not to widen the regex.
2. The regex matches too broadly. A PAN regex that matches any 13-to-19-digit number catches ISO 8601 timestamps. Symptom: timestamps are replaced with [REDACTED]; engineers cannot read the logs. The fix is to anchor the regex to the field name.
3. The agent processor is bypassed. The application writes to stdout, but a sidecar tails the file directly. The sidecar is not in the collector pipeline. Symptom: Loki has two streams for the same service, only one of which is scrubbed.
4. The transform processor statement does not parse. OTTL statements are parsed when the pipeline is built, not when a record arrives. An unknown function, or a path the context does not have, is a configuration error. Symptom: the collector will not start and logs invalid config for "transform" processor. The dangerous variant is a deploy pipeline that rolls back on a failed start: what ends up running is the previous configuration, without the rule, reported as a successful rollback.
5. The design expects a backstop that does not exist. The rule is written on the assumption that a query-time mask will catch what the agent missed. No such control exists in Grafana, Loki, Tempo, or Prometheus. Symptom: the audit finds the raw value in the store, and the only remaining options are deletion, retention, and disclosure.
6. The allowlist is not enforced by review. The application adds a field; the review does not check the allowlist. Symptom: the new field is in the wire payload; the pipeline catches it because the regex is broad; the next audit finds a field the regex does not cover.

How to troubleshoot it

The diagnostic order for “we may be leaking”:

1. Which layer is supposed to handle this field? Source, agent, or backend. The answer is in the tier list.
2. Is the layer live? Run the validation ladder. The answer is in the metrics from the processor (dropped attributes, regex matches).
3. Where is the value landing? Run the audit grep at each boundary — the wire, the collector, the backend. The first boundary where the value appears is the boundary that failed.
4. What is the field’s actual key in the wire payload? Compare with the allowlist or regex. The mismatch is the bug.
5. What is the remediation owner? The owner of the boundary that failed is the owner of the fix.

Security implications

The defence-in-depth model has three layers and three owners. The implementation details:

• The application allowlist is reviewed on every code change. A PR that adds a log line without an allowlist check is rejected.
• The agent processor is monitored. A spike in dropped attributes means the application added a field the processor was not configured for; the team widens the processor rule.
• Query access to the affected data source is reviewed on a schedule. With no backend mask to fall back on, access control and retention are the only post-ingest controls, and both are audited as security controls rather than as tuning knobs.
• The audit grep runs against every layer. The wire payload, the collector debug output, the Loki content, and the Grafana rendering are all checked.

Performance implications

Source redaction is essentially free — the application does not emit the field, so there is nothing to process. Agent redaction costs a map lookup per configured key for the attributes processor, and one regex evaluation per replace_pattern statement for the transform processor; the regex, not the collector, is what determines the bill. Measure it with the collector’s own telemetry rather than estimating. The backend has no redaction cost because it does no redaction — what it costs, once a value has landed, is the compaction work of a delete request and the time of the incident that produced it.

The expensive failure shape is the regex that matches too broadly. The cost is not CPU; it is operator readability. A log line where every field is [REDACTED] is a log line nobody can use.

Verification

You should now be able to answer:

• Which two boundaries can still edit a record, and which one is the cheapest and most complete?
• What can the backend actually do about a value it has already ingested, and what can it not do?
• Why is a regex that matches too broadly worse than a regex that matches too narrowly?
• Which layer is the appropriate place for a field whose ownership is unclear?

Quiz