Run this before the platform carries production load, and again after any change to retention, to the object-store lifecycle rules, to the accounts the buckets live in, or to the set of components in the stack.
The three failures, and why one copy cannot cover them
Operator error, host loss and hostile action need different copies. A recent snapshot on a different host answers the first. A copy on different hardware answers the second. Only a copy outside the reach of the primary credential answers the third. An estate with a single well-managed backup usually has the first covered and believes it has all three.
That is why the items about versioning, lifecycle and account boundaries are
critical while the ones about mechanics are not. Getting the snapshot command
right is a morning of work; discovering that every copy sat inside the blast
radius is not recoverable at all.
Configuration is not state
Most of this stack is configuration, and configuration belongs in Git, where the history is already the backup. What actually needs copying is short: the Prometheus TSDB, the Grafana database, the Alertmanager silences, and the durability configuration of the object stores that Loki and Tempo treat as their source of truth.
Backing up more than that is the commonest mistake, and it is expensive in a
way that hides the real gap. The silences are the usual casualty: they do live
on disk, in the silences/ directory under the Alertmanager storage path, but
that path is not where a tar job written around /etc and /var/lib looks. A
nightly archive of the configuration tree looks thorough and copies none of the
state that actually needs a copy.
Access this needs
Read access to the component APIs, to the configuration tree, and to the object stores; a credential that can call the Prometheus admin API; and the ability to read the drill marker. Where an item names a repository path, a backup directory, a bucket or a component address, substitute the ones your estate uses; the values shown are the shapes, not your inventory.
The snapshot item creates a snapshot directory on the Prometheus host — delete it afterwards through the same admin API rather than by hand, and remember the snapshot uses hard links, so it costs little space until the source blocks are compacted away.
Six items — the scope document, the RPO/RTO targets, the off-host drill, the offline runbook, the escape plan, and the secret-rotation policy — are attested by a person. They are also the six that decide whether the recovery works.
Sign-off
- Reviewer: ________________ Date: ___________
- Platform owner: ___________ Date: ___________
- Recovery owner: ___________ Date: ___________