Break/Fix: Telemetry Contains Sensitive Customer Data · Break/Fix · Observability

Reported symptoms

The nightly secret scan is a boring job that produces nothing. This morning it produces 1,900 findings, all in Loki, all from the checkout service, all carrying an Authorization bearer value and a sixteen-digit number in the message body. The oldest is 11 days old.

The platform team’s first response is that the scanner must be wrong, and they have good grounds for saying so:

Redaction is configured. The collector configuration contains an attributes/redact processor and a transform/redact processor, both reviewed, both committed.
Redaction is running. The collector is healthy and has not restarted.
Redaction is tested. CI runs a synthetic offender through the configuration on every change and asserts the token is replaced. It passed this morning.
Redaction is demonstrably effective. Run the same scan across Tempo: clean. Spans from the checkout service show Bearer [REDACTED] exactly where the token should have been masked.
Alloy has not changed. Its stage.luhn and stage.replace blocks are still there, still running, untouched for five months.
The audit dashboard agrees. Zero tier-0 findings for 30 days.

Three other things happened 11 days ago and were filed separately by three different people. A Grafana panel stopped returning data and was raised as a dashboard bug. Loki ingestion for checkout rose about 30 percent and was attributed to a marketing campaign. And in an unrelated thread, the data protection officer reports that a customer erasure request has turned up that customer’s email address in the observability platform, which is not on the inventory of systems that hold personal data.

Nobody has connected any of these. There is no incident yet, and there are 1,900 live bearer tokens in a log store.

Evidence provided

Read-only / Safethe first offending line, and the exact minute it starts

$ logcli query --limit 3 --from 2026-08-02T00:00:00Z --forward '{service_name="checkout"} |= "Bearer"'

2026-08-02T14:07:11Z {service_name="checkout"} POST /api/v1/charge authorization="Bearer eyJhbGciOi..." pan="4111 1111 1111 1111" amount=49.99 trace_id=8f7c3a1b2e9d4c50
2026-08-02T14:07:12Z {service_name="checkout"} POST /api/v1/charge authorization="Bearer eyJhbGciOi..." pan="4111 1111 1111 1111" amount=12.00 trace_id=1d4e77a90b3c1f22
2026-08-02T14:07:12Z {service_name="checkout"} POST /api/v1/charge authorization="Bearer eyJhbGciOi..." pan="4111 1111 1111 1111" amount=8.75  trace_id=c02b9f6e13a7d481

Illustrative output

The same request, in the other store:

Read-only / Safeone trace ID, two records, two outcomes

$ curl -s -H 'Accept: application/json' 'http://tempo.obs.example.com:3200/api/traces/8f7c3a1b2e9d4c50' | jq -r '.. | .attributes? // empty | .[] | select(.key | test("authorization|pan")) | [.key, .value.stringValue] | @tsv'

http.request.header.authorization   Bearer [REDACTED]

Illustrative output

The stream itself changed shape at the same minute:

# Two selectors, one service. Compare the line counts either side of 14:07.
for SEL in '{job="checkout"}' '{service_name="checkout"}'; do
  printf '%s ' "$SEL"
  logcli query --quiet --limit 0 --from 2026-08-02T14:00:00Z \
    --to 2026-08-02T15:00:00Z --output raw "$SEL" | wc -l
done

The old selector goes to zero at 14:07. The new one starts at 14:07.

Read-only / Safethe processors exist; the question is which pipelines list them

$ yq '.service.pipelines | to_entries | map({signal: .key, processors: .value.processors}) ' /etc/otelcol/config.yaml

- signal: traces
processors: [memory_limiter, attributes/redact, transform/redact, batch]
- signal: logs
processors: [memory_limiter, batch]
- signal: metrics
processors: [memory_limiter, batch]

Illustrative output

On a checkout host, the file Alloy tails:

Read-only / Safenothing has been written to it for 11 days

$ ssh checkout-3.app.example.com 'ls -lt /var/log/checkout/ | head -3'

total 1841204
-rw-r--r-- 1 checkout checkout 214748364 Aug  2 14:07 app.log
-rw-r--r-- 1 checkout checkout 214748364 Aug  2 09:31 app.log.1

Illustrative output

And the change that everybody has already dismissed:

Read-only / Safeone commit in twelve days, and it does not touch logging content

$ git -C /srv/checkout log --oneline --since=12.days -- deploy/ internal/telemetry/

a91f4c2 checkout: switch log export from file to OTLP (no functional change)

Illustrative output

Work the evidence before reading on

Three controls say the redaction works. One store says it does not. Both are telling the truth.

The same trace ID appears on a masked span and an unmasked log line. Two records, one request, two outcomes. What is different about the two records, and where does that difference first appear?
Alloy has not changed and is still running. What is it processing?
The CI test passes and the audit detector is clean. Write down, in one sentence each, exactly what each of them proves - and then write down what each of them does not cover.
Three tickets were filed on 2 August by three people: a broken dashboard, a 30 percent ingestion rise, and nothing at all about the third. What single event produces all three?

Before continuing: you have found the configuration line that explains everything and you can fix it in thirty seconds. What are you leaking while you do that, who owns the decision, and what does fixing the configuration first cost you?

Root cause

1. Two agent pipelines, and a record can only be in one

A log line tailed from a file is seen by Alloy and never by the collector. A record shipped over OTLP is seen by the collector and never by Alloy. This is not a subtlety of the deployment; it is the shape of the pipeline, and it means redaction rules do not follow a record - the record chooses which rules apply by choosing which agent carries it.

The checkout service used to write app.log. Alloy tailed it and ran stage.luhn, which masks numbers that pass the Luhn checksum, and stage.replace, which rewrote the bearer token. Card numbers and tokens were being caught, correctly, on every line, by a component nobody had touched in five months.

At 14:07 on 2 August the service stopped writing that file.

2. The gap was already there and had never mattered

The collector configuration lists the redaction processors under service.pipelines.traces.processors. It does not list them under service.pipelines.logs.processors. That has been true since the configuration was written.

It had no consequence, because no log record had ever entered the collector. The logs pipeline existed, was syntactically valid, was reviewed, and carried nothing. A dormant gap is invisible in review precisely because it is not wrong yet: a reviewer looking at that file sees redaction configured and a logs pipeline that is not in use.

The deploy did not create the defect. It made the defect load-bearing, from a change whose description - and whose diff - contains nothing about redaction, logging content, or security.

3. Every control was scoped like the gap

This is the part worth sitting with, because the leak was survivable and the eleven days were not.

The CI test constructs a span, runs it through the collector configuration, and asserts the token is replaced. It is a correct test of the traces pipeline. It has never contained a log record.
The audit detector queries the trace store. It reported zero findings because there were zero findings in Tempo.
The team’s own spot check - run the scan against Tempo, see it clean - was the same test a third time.

Three independent controls, all honest, all answering a question about traces. The signal that was leaking was the one none of them looked at, and the reason is the same in all three cases: each was built at the moment the traces pipeline was the thing being secured, and each inherited that scope without ever stating it.

4. The other two tickets are the same event

Loki derives stream labels for an OTLP record from its resource attributes, so the checkout stream stopped being selected by the old label and started being selected by one derived from service.name. That is the broken dashboard panel. An OTLP log record also carries structure the plain file line did not, which is most of the 30 percent ingestion rise.

Three tickets, three teams, three plausible independent explanations, one deploy. The estate had all the evidence it needed on day one and no mechanism for putting three tickets on the same table.

Under the hood

Two backend facts decide what is still possible on day 11, and both are worth knowing before the incident rather than during it.

Nothing in this stack redacts at query time. Loki, Tempo and Prometheus return what they were given. There is no view, no mask, no column permission that hides a stored value from a query that selects it. The agent boundary is the last point at which a record can be edited; after that the only levers are access control, retention, and - in Loki alone - a delete request.

The Loki delete request is a real remedy with real preconditions. It needs a TSDB index, retention_enabled on the compactor, and deletion_mode set to filter-and-delete. It takes a LogQL selector with a line filter plus a start and end time, and it does not take effect immediately: there is a cancellation period, 24 hours by default, during which the request can be withdrawn. That delay is usually described as a safety feature and it genuinely is one - a delete request with a selector one character wider than intended removes evidence you needed - but it also means the values remain readable for another day, which changes what you tell legal.

Tempo has no equivalent. A span attribute that reached the store stays until the block ages out under retention. In this incident that asymmetry happens to be in your favour, because the traces were the records that were masked. Reverse the two signals and the same leak would have had no removal path at all.

Fixing the configuration first feels like progress and is not

The gap is one line. You can see it, you can fix it, and the fix is obviously right. Every instinct says do it now.

What the fix accomplishes is that no new tokens enter Loki. It does nothing about the ones already there, and those are not a compliance abstraction: they are live bearer credentials for real users, sitting in a store that a large number of engineers can read, for as long as retention keeps them.

The order that limits blast radius is contain, notify, rotate, redact, document, and the configuration change is in the fourth phase for a reason. Containment is what stops further reads. Notification is what starts the legal clock inside the window rather than after it. Rotation is what makes the leaked values worthless, and it is the phase with real customer cost - here it means invalidating every authenticated session in an eleven-day window on a payment service, which is a decision somebody senior has to make and own.

There is a second reason not to lead with the config change. A transform processor with a bad OTTL statement does not fail at the first record; it fails when the pipeline is built, and the collector does not start. The deploy then rolls back to the previous configuration - the one with no log redaction at all - and reports itself successful. That is how a redaction change becomes a redaction regression with a green deploy behind it.

Resolution

Declare an incident and put the three tickets on the same table. The dashboard bug, the ingestion rise and the erasure request are the same event; until somebody says so, three teams keep working three problems.
Contain. Restrict read access to the affected Loki tenant, and invalidate existing Grafana sessions - a role change does not evict a session that is already open, and the audit log will show reads after the restriction if you skip this.
Preserve the evidence before you remove anything. Snapshot the chunk store to a preservation bucket. You are about to submit a delete request, and the post-mortem and any regulatory response both need the record you are deleting.
Notify legal inside the four-hour window, and tell them the exposure window is eleven days rather than one morning. The disclosure clock is driven by the window and by the moment of confirmation, not by how long you have known.
Rotate every credential that appeared in the window. On a checkout service that is every authenticated session over eleven days, which is a user-visible mass logout. Name the owner of that decision; it is not the platform team.
Now fix the pipeline. Add attributes/redact and transform/redact to service.pipelines.logs.processors. Roll it to one collector first, confirm the process started, and confirm log records are still being accepted before touching the rest of the fleet.
Submit the Loki delete request for the affected stream and window. Check the preconditions first - TSDB index, retention_enabled on the compactor, deletion_mode: filter-and-delete - and use the cancellation period deliberately: spend it confirming the selector matches only what you intend, rather than treating it as an unwanted delay.
Decide explicitly what to do about checkout log ingestion in the meantime. Switching it off removes the leak and leaves a payment service unobservable during a live security incident. If you choose it, call it a hold, give it an owner and an end time, and say what would change the decision.
Close the loop on the deploy record rather than only on the collector configuration. The defect is that a transport migration was reviewed as though it had no security surface, and the next service to migrate will be reviewed by whoever reads that record.

Verification

Test the path, not the file. Send a synthetic offender - a distinctive marker value that is not a real credential - through the same OTLP log path checkout uses, and require it to arrive in Loki masked.
Prove the test can fail. Run the same synthetic against the previous configuration and require it to arrive unmasked. A redaction check that has never produced a failure has never been tested, and it is the single cheapest thing to get wrong here.
Confirm the collector is running and accepting log records after the change. A redaction fix that stops the pipeline is an outage stacked on a leak, and it will be reported as a successful deploy.
Extend the CI test to construct a log record as well as a span, and watch the new case fail against the old configuration before it passes against the new one. A test added green proves nothing.
Re-run the store scan across both signals, without binding the query to a stream selector. The scan that missed eleven days was scoped by a label convention, and a label rename is exactly what such a scan cannot see.
Verify the rotation by attempting authentication with an old token and requiring it to be refused. A new credential existing is not the same as the old one being dead.
Verify the deletion after the cancellation period has elapsed, by re-querying the exact window and requiring no matches. Before that point the request is accepted and the data is still readable.
Confirm the observability platform is now on the inventory of systems holding personal data, and that the next erasure request will reach it without a data protection officer having to discover it.

Prevention

Treat a change of transport as a change of control surface. Moving a service from file tailing to OTLP export moves every record from one agent to another, and the redaction rules do not travel with it. The deploy will honestly be described as having no functional change, because from the application’s side there is none.
Make the reviewable unit of a collector configuration the service.pipelines block, not the processors block. A processor that exists is not a processor that runs, and the per-signal processor list is the only place that distinction is visible.
Test every control against the signal it protects. A redaction test built from spans certifies traces and is silent about logs and metrics. Write down the scope of each control next to the control, because scope is inherited silently from the day it was written.
Scope audit detectors to the store and to content rather than to a stream selector. A detector bound to a label convention stops seeing a stream the moment the convention changes, and reports zero findings while doing so.
Alert when a telemetry source goes quiet. Alloy stopped shipping checkout logs at 14:07 and nothing said so. A no-lines-for-N-minutes alert on a known stream would have named the date and the minute within the hour, and would have turned an eleven-day exposure into a same-day one.
Keep classification tiers attached to fields rather than to pipelines, and hold an unclassified field at the strictest tier until security and legal classify it. The tier travels with the field through whichever agent carries it.

What if the traces had been the unredacted signal instead?

Everything up to the fix would have run the same way and the ending would have been considerably worse.

Loki has a delete request. Tempo does not. A span attribute that reached the store stays until the block ages out under whatever retention the tenant is configured with, and the only levers left are access control and waiting. If the retention is 30 days, the answer to “when will the credentials be out of the store” is thirty days, and there is no engineering action that shortens it.

That asymmetry should shape the design rather than the response. When you are deciding which signals carry which fields, ask what the removal path looks like for each store before anything goes wrong, and let the store with no removal path be the one that carries the least. Retention that felt generous when it was chosen for query convenience is the same number that defines your exposure window when a leak lands in it.

It also reframes what looks like luck in this incident. The traces were masked because the traces pipeline was the one somebody secured. Nothing about the design made the recoverable store the leaking one; that was an accident, and accidents run both ways.

Why this matters

Every component in this incident did exactly what it was configured to do. The application emitted the fields it had always emitted. Alloy redacted every line it was given. The collector ran every processor it was told to run, on the pipelines it was told to run them on. The CI test tested what it was written to test. The audit detector queried the store it was pointed at. There is no misconfiguration in this story and no mistake in any single file.

The leak came from a change of route. A record moved from one path to another, and the controls were attached to paths rather than to the record. That is the characteristic failure of defence in depth done by location: each layer is correct about the traffic it sees, and nobody owns the question of whether the traffic still goes there.

The operational habit worth taking from this is to ask, for every sensitive field, which agent will carry it and what happens if that answer changes. Not “is redaction configured” - it was - but “what is the complete set of paths this record can take, and is every one of them covered”. A control that protects a route protects nothing the day the route moves, and the deploy that moves it will not mention security at all.

Reported symptoms

Evidence

Root cause

Remediation

Verification

Prevention

Reported symptoms

Evidence provided

Work the evidence before reading on

Root cause

1. Two agent pipelines, and a record can only be in one

2. The gap was already there and had never mattered

3. Every control was scoped like the gap

4. The other two tickets are the same event

Resolution

Verification

Prevention