Telemetry Security Review

Run this before the platform takes production telemetry, and re-run it quarterly and after any change to the network topology, the identity model, or the set of teams pushing data into it.

It overlaps the Grafana readiness gate deliberately and asks a different question. That gate covers who can log into the dashboard and what they can see once they are in. This review covers everything behind it: what each component listens on, who can call it directly without going through Grafana at all, and what the stored data would be worth to somebody who got there.

The two questions underneath

The first is exposure: the telemetry platform is a system with a network listener on every host in the estate and a copy of what those hosts are doing. Its defaults are permissive because upstream optimises for a first install that works, and every one of those defaults is a decision somebody has to make again for production.

The second is content. Telemetry is a data store, and it is the one data store nobody classified — the logs, spans and labels arrive from a hundred call sites written by people who were thinking about a bug, not about a retention policy. The first question decides how hard the platform is to reach. The second decides what the reach is worth.

Where the numbers come from

Most items are one shell command on a component host, one call to a config or metrics endpoint, or one object-store API call. Substitute your own addresses and bucket names; the ones below are the shapes used throughout this course, not an estate that exists.

Four items are attested rather than measured: the data classification, the source-redaction discipline, the leak runbook and the cross-tenant policy. Those are the four an auditor will ask about, and a command cannot answer any of them.

Access this needs

Shell access on the component hosts for the bind and firewall items, read access to each component’s HTTP API, read-only credentials for the object store, and read access to the repository holding the configuration and the runbooks. The scan items need a host on the network you are scanning from. Nothing on this list writes.

The debug print that ran for six hours

A developer adds a print to investigate a slow request. It serialises the whole incoming payload, which includes the authorization header and a cardholder field. It ships. It runs for six hours before anybody notices.

Every part of the platform behaves perfectly. The agent forwards what it is given, the ingester stores it, retention keeps it for exactly as long as the policy says, and the dashboard renders it to anyone with viewer access. There is no query-time mask to turn on and no field-level access control to apply, so removing the print stops new values arriving and changes nothing about the ones already stored.

What follows is a credential rotation across the affected users, a disclosure decision on a regulatory clock, and a forensic hold on the chunks that would otherwise have aged out. The engineering fix is one line. The incident is three months long, and none of it is recoverable by a platform change.

Sign-off

Reviewer: ________________ Date: ___________
Platform owner: ___________ Date: ___________
Security owner: ___________ Date: ___________

Why this matters

The security question about an observability platform is not whether the components are secure. It is that a working platform is, by construction, a listener on every host in the estate feeding a single store that holds a description of everything those hosts did — and, whenever the discipline at the source slipped, some of what they were told.

That makes the store worth more to an attacker than most of the systems it watches. A snapshot of the metrics database carries the scrape configuration that produced it, which is a map of credentials. A month of logs carries whatever the applications logged, which is whatever the developers found useful at the time. Traces carry request attributes, which is the request.

None of that is an argument for collecting less. It is an argument for treating the platform as the production data store it already is, and for reviewing it on the cadence you would review any other system that holds customer data.

The two questions underneath

Where the numbers come from

Access this needs

Sign-off

Critical20 items

Warning3 items

Info1 item