How to use
Quarterly, and one item needs a cluster you are willing to destroy.
Keep three things apart for the whole review, because every failure in this subject comes from collapsing them: the desired state in version control, the cluster state the API server holds, and the application data inside the PersistentVolumes. Only the first is in the repository. Each has a different owner, a different failure mode and a different restore, and a plan covering two of the three fails quietly rather than loudly.
Work the items in order. The snapshot and inventory items are answerable from a terminal in about twenty minutes. The rebuild item is not: it needs an isolated cluster and an afternoon, and it is the only item here that produces evidence instead of an opinion.
Where the numbers come from
Snapshot age comes from the newest file at the off-node destination, listed — not from the schedule in the job definition, which is a statement of intent. Its integrity evidence comes from the status output recorded at backup time: on etcd 3.7.1 that is a hash, a revision and a key count, and it is the only integrity claim the file carries. Record all three when the snapshot is taken; they are what you compare against when the file is finally used.
The recovery point is the older of two ages — the newest usable control-plane snapshot, and the newest volume backup — and it is reported as reverted objects rather than as minutes. A restore returns the cluster to the moment the snapshot was taken and discards every decision made since, without an error and without a log line.
The claim inventory comes from the live cluster, one row per PersistentVolumeClaim across every namespace, never from the repository; the repository is the thing being checked. Rebuild timings come from the isolated-cluster exercise, measured from “no cluster exists” to “a business transaction succeeded”, including the time to provision the cluster — routinely omitted, and routinely the largest part.
Access this needs
Read access to the manifest repository and its history. A kubeconfig with cluster-wide read across every namespace, covering claims, StorageClasses and Secret metadata — the review establishes which Secrets exist and where they come from, never what they contain. Read access to the snapshot destination, and shell access on a control-plane node to see what is written locally.
For the rebuild item: an isolated cluster sharing no credentials, no storage and no DNS with production, plus the authority to destroy it.
Nothing here requires write access to production. A reviewer who needs it to finish an item has found a defect in the cluster’s RBAC, and that is a finding in its own right.
What the review produces
A dated record naming the reviewer and carrying four artefacts: the claim inventory, one row per claim, with a named protection and the only-copy rows marked; the snapshot inventory with age, destination and recorded integrity evidence; the diff between the live cluster and the manifest repository; and the rebuild log with its list of objects that came back present and empty.
That last list is the headline. It is the honest scope of the separate data backup, and it goes to the service owner whether or not it makes comfortable reading. Anything missing from it was assumed rather than checked.
Sign-off
- Reviewer: ____________________ Date: ____________
- Platform owner: ______________ Date: ____________
- Service owner: _______________ Date: ____________
Every critical item must pass. A failing critical item is a blocker, not a note for next quarter: record the disposition of every item that did not pass, and the name of whoever accepted the residual risk.