Run this quarterly, and additionally after any month where the telemetry invoice moved more than the forecast said it would, or after any incident where an investigation found the data already deleted.
It overlaps the Loki and Prometheus readiness gates on purpose, and asks a different question of the same settings. The readiness gates ask whether retention is configured correctly before the platform takes production traffic. This review asks whether the numbers still describe the platform a year later: whether the window matches what the business now promises, whether the compactor that enforces it is still alive, and whether each capacity tier has enough space left to survive the next spike.
What the review reconciles
Three artefacts that drift apart quietly. The policy document says what the platform keeps and why. The engine state is what the retention flags and compactors are actually doing. The field reality is which tenants and streams still exist and who owns them. Any two of the three can agree while the third has moved, and the direction of the drift decides which incident you get: data kept that should not have been, or data deleted that somebody needed.
Where the numbers come from
Almost every item is one Prometheus query, one call to a component’s config or metrics endpoint, or one object-store API call. Substitute your own addresses, bucket names and the path to your policy file; the ones below are the shapes used throughout this course, not an estate that exists.
The arithmetic behind the capacity items is the same arithmetic in every capacity review, and it is small. What separates a review that prevents outages from one that records them is running it against live metrics rather than against last year’s spreadsheet.
Access this needs
Read access to the Prometheus HTTP API, the Loki and Tempo config, metrics and runtime-config endpoints, and read-only credentials for the object store. Two items — the retention policy document and the estimate behind a newly onboarded workload — are read from the repository. Nothing here writes.
Every remediation this produces is a separate reviewed change. A retention reduction in particular is irreversible and belongs in its own ticket with the prior value recorded, not in the hour the review was booked for.
Sign-off
- Reviewer: ________________ Date: ___________
- Platform owner: ___________ Date: ___________
- Budget owner: _____________ Date: ___________