Runbook: Respond to a Telemetry Data Leak

A detector has fired, or an engineer has noticed something in a panel, and a value that should never have been in telemetry is in telemetry. A bearer token in a log line. A card number in a span attribute. A customer email promoted to a Loki label.

The instinct is to make it disappear. Work out how to delete it, get it out of the store, close the ticket. That instinct produces the wrong ordering, and telemetry punishes the wrong ordering harder than a code repository does.

Rotate first. The store comes last, and two of the three stores cannot be cleaned at all.

Telemetry stores are append-only by design. There is no query-time mask in Grafana, Loki, Tempo or Prometheus: no data source setting, no grafana.ini section, no provisioning endpoint that hides a value on its way to a panel. Once a value lands, it is in the chunk or the block, in every replica, and in every backup taken since.

That leaves three distinct realities:

Loki can delete, but only with a TSDB index, retention_enabled on the compactor, a delete_request_store, and deletion_mode set to filter-only or filter-and-delete - and the deletion is applied by the compactor after a cancellation period, 24 hours by default.
Tempo cannot. A span attribute that reached Tempo stays until the block ages out under retention.
Prometheus stores a leaked label value as series, and deleting series needs an admin API that is disabled by default.

So the exposure does not end when the store is cleaned. If the value is a credential, the exposure ends at revocation, and that is a step you can take in the first hour regardless of what the stores can do.

When to use this runbook

A secret scanner or the periodic audit flags a value in Loki, Tempo or Prometheus.
An engineer finds a credential, a card number or personal data in a panel, a log line or a trace.
A schema change ships a new field and the classification review is retrospective.
A tenant reports seeing another tenant’s data.

It does not apply when:

The finding is a detector false positive. A documentation example, a test fixture, or a placeholder that matches the pattern. Confirm by field name and location first - the confirmation is cheap and the incident is not.
The value is only in an application log file on the host and has not reached the pipeline. That is a host-level problem with a much smaller blast radius and a completely different remedy.
The concern is a high-cardinality label with no sensitivity. That is a cardinality incident, not a leak. It has its own runbook and a different urgency.
A leaked credential lives in a Git repository rather than in telemetry. Same rotation discipline, different hunt: the copies are in clones and CI caches rather than in chunks and blocks.

Blast radius

Two blast radii run on different clocks, and conflating them is the commonest mistake in this class of incident.

The credential’s blast radius is everything the value grants access to, for the whole interval from first emission to revocation. It is bounded by an action you control and can complete quickly.

The store’s blast radius is everyone who can read the affected tenant, for as long as the data exists there. That is bounded by retention, by the delete-request machinery if it is configured, and by backups that nobody has inventoried. It is often measured in weeks.

The remediation carries its own blast radius, which is why the ordering below puts evidence first. A delete request removes log lines the incident’s own timeline may need. A retention change removes them for every tenant in the window, not just the affected stream. An over-broad redaction rule creates a permanent observability gap that nobody notices until the next outage.

The clocks

Phase	Clock	Owner	What it changes
Contain	1 hour	Security on-call	Who can read it, and whether the credential still works
Notify	4 hours	Legal	The disclosure decision, on a statutory window
Rotate	24 hours	Engineering and security	Ends the exposure for a credential
Redact	7 days	Engineering	Stops new values reaching the store
Document	30 days	Incident commander	The detector that catches the next one

The phases run in parallel where they can. Containment does not wait for legal, and legal does not wait for engineering.

Step 1: Confirm, without spreading the value

Read-only / Safescope the finding by name, not by value

# Which streams carry the field name at all, and since when.
logcli series --since=24h '{job="checkout"}'

# Is the field a LABEL (in the index) or content (in the chunk)?
# The answer changes the cardinality picture and the remediation.
logcli labels --since=24h

# Count, do not print. The count is the finding; the value is the leak.
logcli query --since=24h --quiet \
'count_over_time({job="checkout"} |= "authToken" [24h])'

Record four facts before moving: the store and tenant, the stream or service, the field name, and the earliest timestamp it appears. Those four are what every later phase is scoped by, and the fourth is one end of the exposure window.

Step 2: Contain

Containment is two actions with very different value. Restricting the audience limits who else can read it. Revoking the credential ends the exposure. Do the first while you organise the second; do not mistake the first for remediation.

Service impact possiblebound the audience, then preserve the evidence

# 1. Restrict read access to the affected tenant to the incident team.
#    Then invalidate existing sessions: a permission change does not
#    evict a session that is already open.

# 2. Preserve the affected chunks before anything destructive happens.
#    The preservation target must not be the store you are about to modify.
rclone sync /loki/chunks/ preservation:leak-incident-2026-08-19/ \
--transfers=4 --checkers=4 --log-file=/var/log/preserve.log

Step 3: Rotate, if the value is a credential

This is the step that ends the exposure, and it is the only step in this runbook whose completion is unambiguous.

The order inside it is fixed: enumerate every consumer, issue the replacement, distribute it and confirm each consumer works, then revoke. Rotating without the inventory turns a security incident into an outage; revoking before the inventory is confirmed does the same thing faster.

Read-only / Safethe check that must be able to fail

# After revocation, use the old credential deliberately and require failure.
curl -s -o /dev/null -w '%{http_code}\n' \
-H "Authorization: Bearer $LEAKED_TOKEN" https://api.example.com/v1/user
# Expect 401. A 200 here means the exposure is still open.

# Then confirm at the provider's own audit log that the status is
# "revoked", with a timestamp - not "replaced".

Illustrative output

“Superseded” is not “revoked”. A token that was replaced but never invalidated is a live credential sitting in a log store, and a rotation that stopped at issuing a new value has closed nothing.

Record the revocation timestamp. It is the end of the exposure window, and it is the number the incident is measured by.

Step 4: Stop the emission - source first, agent second

The application is the primary defence and the agent is the safety net. Fixing only the agent leaves the field being emitted into every pipeline the agent does not cover, and the next audit finds it in a different store.

The source fix is the field removed from the call site and added to the allowlist as dropped, with a regression test that fails if it comes back.

Then the agent. For OTLP attributes, the collector’s attributes processor is the simplest tool and the one least able to fail silently:

Configuration changecollector: attributes processor

processors:
attributes/redact:
  actions:
    - key: authToken
      action: update
      value: "[REDACTED]"

service:
pipelines:
  traces:
    receivers: [otlp]
    processors: [attributes/redact, batch]
    exporters: [otlp/tempo]

update rewrites a key only if it already exists, which is the behaviour you want: a service that never emits the field does not acquire an attribute reading [REDACTED]. Note the service.pipelines block - a processor that is defined but never listed in a pipeline is a perfectly valid no-op, and it is the most common reason a redaction rule “does not work”.

For patterns rather than exact keys, the transform processor evaluates OTTL statements; for a fail-closed allowlist, the redaction processor names what to keep instead of what to remove. Both are stronger tools and both are easier to get wrong under pressure.

For log lines, the equivalent is Alloy’s loki.process:

Configuration changealloy: redact the line, drop the label

loki.process "redact" {
stage.replace {
  expression = "Bearer\\s+([A-Za-z0-9._\\-/+=]+)"
  replace    = "[REDACTED]"
}

stage.label_drop {
  values = ["authToken", "auth_token", "token"]
}

forward_to = [loki.write.default.receiver]
}

Then verify through the agent, not around it:

Read-only / Safeprove the rule is live

# The collector came back, and its journal has no config error.
systemctl is-active otelcol
journalctl -u otelcol --since '5 min ago' | grep -i 'invalid config'

# The processor is actually seeing records.
curl -s http://otelcol:8888/metrics | grep otelcol_processor_accepted_spans

# Push a known-bad line through the AGENT and read it back from Loki.
echo 'auth header: Bearer eyJhbGciOi... test line' >> /var/log/application/test.log
logcli query --since=5m '{job="application"}' --output=raw

The last step is the one teams skip. Pushing a test line straight to /loki/api/v1/push exercises Loki, not the redaction rule, and a green result from it proves nothing about the pipeline.

Step 5: What is already stored

Three stores, three different answers. Write the answer down for each one, including the ones where the answer is “we cannot”.

Store	Can the stored value be removed?	What you actually do
Loki	Yes, with preconditions	File a delete request scoped by stream selector and time range; the compactor applies it after the cancellation period
Tempo	No	Record the date the affected blocks age out under retention, and treat that date as the end of the exposure for that store
Prometheus	Only via the admin API	A leaked label value is series; series deletion needs an API that is off by default and is a separate, deliberate change
Grafana	Not applicable	Grafana stores no telemetry, but its query history holds any search you ran that contained the value

Destructiveloki: file the delete request, then record its id

# Scope it as narrowly as the finding allows. This deletes log lines.
curl -X POST -G 'http://loki:3100/loki/api/v1/delete' \
-H "X-Scope-OrgID: prod" \
--data-urlencode 'query={job="checkout"} |= "authToken"' \
--data-urlencode 'start=2026-08-17T00:00:00Z' \
--data-urlencode 'end=2026-08-19T09:00:00Z'
# 204 No Content on acceptance.

# List the requests for the tenant and record the id.
curl -s -G 'http://loki:3100/loki/api/v1/delete' -H "X-Scope-OrgID: prod"

# Cancel, while still inside the cancellation window:
REQ_ID=replace-with-the-id-from-the-listing
curl -X DELETE -G 'http://loki:3100/loki/api/v1/delete' \
-H "X-Scope-OrgID: prod" --data-urlencode "request_id=$REQ_ID"

The cancellation window - 24 hours by default - is the reversibility that makes this step safe to take. It is also why the store is not clean when the request returns 204, and why “deleted” is not a claim to put in the ticket until the compactor has swept.

Step 6: Close the loop

Re-run the detection over a fresh sample, searching for the field name. A clean result proves the source fix landed; it says nothing about the historical data, and the two must be reported separately.

Then the record: first emission, detection, containment, revocation, source fix merged, delete request filed, and the date each store stops holding the value. Two intervals matter and they have different fixes. Time from first emission to detection is a detector problem. Time from detection to revocation is a rehearsal problem.

Common patterns

Symptom	Likely cause	Resolution
Redaction merged, value still in new lines	Processor defined but not listed in `service.pipelines`	Add it to the pipeline; verify the accepted-records counter moves
Collector healthy, no redaction happening	Deploy rolled back to the previous config after an OTTL parse error	Check the journal for `invalid config`; fix forward
`email` label gone, `user_email` still there	`stage.label_drop` takes literal names, not patterns	Name every spelling, or use a relabel rule with a regex
Test line arrives redacted, production does not	Test pushed directly to the Loki API, bypassing the agent	Re-test through the agent
Delete request returned 204, value still queryable	Cancellation window has not elapsed; compactor has not swept	Wait, then re-verify; do not file a second request
Delete request rejected	Compactor preconditions not met	Establish `deletion_mode` and `retention_enabled` before relying on deletion
Value gone from Loki, still in Tempo	Tempo has no delete path	Record the block-retention expiry date as the exposure end for that store
Old credential still works after “rotation”	Superseded rather than revoked	Revoke at the source; verify with a deliberate attempt
The same field reappears next quarter	Source never fixed; only the scrubber was	Fix the call site and add the regression test

Escalation

Escalate when:

The finding is confirmed. Immediately, to security. This is an incident, not a configuration ticket.
The classification is tier 0 or tier 1 - legal has a statutory clock running from confirmation.
Rotation needs a credential owner who is not you.
Deletion or a retention change is on the table: destroying telemetry is the data owner’s decision.
The exposure window cannot be bounded, or the value reached a backup, a replica, or a downstream export outside your control.
A consumer cannot be updated before revocation.

Why this matters

Every instinct in this incident points at the store, and the store is the part you have the least power over.

A telemetry pipeline is built to be append-only and to fan out: the value is in the chunk, in the index, in the replicas, in the object store, and in every backup taken since it arrived. No backend redacts at query time, so there is no configuration change that makes it invisible. Loki will delete under conditions somebody had to set up months ago; Tempo will not; and the honest answer for Tempo is a date, not an action.

That is exactly why the ordering matters. Revocation is available in the first hour and it is the only step that ends the exposure outright. The source fix is the only step that is permanent. The agent rule is a safety net for what the source forgot, and the delete request is housekeeping that arrives a day late by design.

A team that starts at the store spends the first hours on the least effective action, with the credential still live and the application still emitting. A team that starts at revocation has closed the incident before it has finished discussing what can be deleted - and it can then take the deletion decision calmly, which matters, because that decision destroys evidence and cannot be taken twice.

1 · Prerequisites

2 · Pre-checks

3 · Procedure

4 · Verification

5 · Rollback

6 · Escalation