Snapshot validation — verify before you trust

A snapshot whose integrity is unverified is a snapshot that cannot be trusted in an incident. Verification is two-fold: the snapshot file is well-formed and the restored cluster behaves as expected. This lesson walks both: status checks, hash verification, restore-to-temp, and the production discipline of continuous validation.

Verification at write time

After every etcdctl snapshot save, run:

etcdutl snapshot status /backup/etcd-snapshot.db --write-out=table

+----------+----------+------------+----------------+
|   HASH   | REVISION |  TOTAL KEY |   TOTAL SIZE   |
+----------+----------+------------+----------------+
| a1b2c3d4 | 41289312 |       4123 |   82419000     |
+----------+----------+------------+----------------+

A successful snapshot has:

• A non-zero HASH (the SHA-256 of the bbolt file’s content).
• A non-zero REVISION (the last applied log index in the snapshot).
• A TOTAL KEY count equal to the cluster’s object count.
• A TOTAL SIZE consistent with the bbolt DB size.

etcdutl snapshot status /backup/etcd-snapshot.db --write-out=json

{
  "hash": 2735966328,
  "revision": 41289312,
  "totalKey": 4123,
  "totalSize": 82419000
}

The JSON output is suitable for programmatic checks in a verification script.

Verification at upload time

After upload to object storage, the SHA-256 hash of the uploaded file should match the source:

# Local hash
sha256sum /backup/etcd-snapshot.db

# Remote hash (S3)
aws s3 cp s3://prod-etcd-backups/etcd/snapshot.db /tmp/verify.db
sha256sum /tmp/verify.db

A mismatch indicates a partial upload or an object-store transmission error. The disciplined pipeline fails the upload and re-takes the snapshot.

sequenceDiagram
    autonumber
    participant Op as Cronjob
    participant Loc as Local /backup
    participant S3 as Object storage
    Op->>Loc: etcdctl snapshot save
    Op->>Loc: sha256sum
    Op->>Loc: etcdutl snapshot status
    Op->>S3: aws s3 cp
    Op->>S3: aws s3api get-object-attributes (etag)
    Op->>Op: compare local sha256 vs S3 ETag
    Op->>Op: record success / failure in monitoring

Restore-to-temp validation

The most reliable verification is to actually restore the snapshot and run it:

1. Spin up a temporary etcd cluster from the snapshot file on a sandbox host.
2. Run kubectl against it to verify the API responds.
3. Inspect the contents for known objects.
4. Tear down the temp cluster.

The restored cluster is independent of the production cluster; it lives on a different IP, with its own certificates, and is reached by a kubeconfig context that points at it.

# On a sandbox host with /backup mounted read-only
etcdutl snapshot restore /backup/etcd-snapshot.db \
  --data-dir=/tmp/etcd-restore \
  --name=sandbox \
  --initial-cluster=sandbox=https://127.0.0.1:2380 \
  --initial-advertise-peer-urls=https://127.0.0.1:2380

# Start an etcd process pointing at the restored data
etcd --data-dir=/tmp/etcd-restore \
  --listen-client-urls=https://127.0.0.1:12379 \
  --advertise-client-urls=https://127.0.0.1:12379 \
  --listen-peer-urls=https://127.0.0.1:12380 \
  --initial-advertise-peer-urls=https://127.0.0.1:12380 \
  --initial-cluster=sandbox=https://127.0.0.1:12380

# Verify it works
etcdctl --endpoints=https://127.0.0.1:12379 member list
etcdctl --endpoints=https://127.0.0.1:12379 get --prefix /registry/ --keys-only | head

$ etcdctl --endpoints=https://127.0.0.1:12379 get --prefix /registry/secrets/ --keys-only | wc -l

76

The expected count of Secrets in the snapshot should match the cluster’s known Secret count. If the cluster reported 76 Secrets before the snapshot, the restored cluster should have 76.

Schema sanity checks

Beyond hash and key count, schema sanity checks verify the snapshot’s contents match expectations:

# Total objects in the snapshot
kubectl --kubeconfig /tmp/snapshot.kubeconfig get all -A --no-headers | wc -l
# Expected: matches the production count approximately

# Specific high-value objects
kubectl --kubeconfig /tmp/snapshot.kubeconfig get namespace prod
kubectl --kubeconfig /tmp/snapshot.kubeconfig get clusterrolebinding cluster-admin
kubectl --kubeconfig /tmp/snapshot.kubeconfig get clusterrole cluster-admin -o yaml | grep -c '  - '

A schema sanity check that matches the production numbers is a high-confidence signal that the snapshot is internally consistent and matches the cluster’s intent.

flowchart TB
    SNAP[Snapshot file] --> STATUS[Status check]
    STATUS -->|ok| HASH[Hash check]
    HASH -->|ok| RESTORE[Restore to sandbox]
    RESTORE -->|ok| CONTENT[Schema sanity check]
    CONTENT -->|ok| PRODUCTION[Trusted]

The restore drill

The discipline that keeps the team ready is the restore drill. A drill:

• Picks a recent snapshot.
• Restores it to a sandbox cluster.
• Spins up the rest of the Kubernetes stack against the sandbox (API server, scheduler, controller manager).
• Runs a workload against the sandbox.
• Records the time taken and any procedural issues.
• Updates the runbook with findings.

A typical drill schedule:

What catches snapshot corruption

The verification suite covers all of these. The most common escape is the encryption-key loss case: a snapshot taken under one key cannot be restored under a different key.

Alerting

A small set of high-signal alerts for snapshot integrity:

- alert: EtcdSnapshotHashMismatch
  expr: etcd_snapshot_upload_hash_mismatch > 0
  for: 5m
  labels:
    severity: warning

- alert: EtcdSnapshotStatusFailed
  expr: etcd_snapshot_status_failed > 0
  for: 5m
  labels:
    severity: critical

- alert: EtcdSnapshotAgeTooOld
  expr: time() - etcd_snapshot_last_success_time > 7200
  for: 5m
  labels:
    severity: critical

- alert: EtcdSnapshotKeyCountDrift
  expr: abs(etcd_snapshot_key_count - etcd_cluster_key_count) > 5000
  for: 1h
  labels:
    severity: warning

The metrics:

• etcd_snapshot_upload_hash_mismatch is incremented when the local vs remote hash differs.
• etcd_snapshot_status_failed is incremented when snapshot status reports a non-zero exit.
• etcd_snapshot_last_success_time is the unix timestamp of the last successful snapshot.
• etcd_snapshot_key_count and etcd_cluster_key_count capture drift between the snapshot and the live cluster.

The “we don’t have backups” false signal

Teams sometimes report “we have backups” because a snapshot was taken at one point. The reality check:

• When was the most recent successful snapshot?
• Was the snapshot’s hash verified?
• Was the snapshot’s contents validated?
• Has the snapshot been restored in a drill?

A positive answer to all four makes the snapshot trusted. Anything less leaves the snapshot as “uncategorised data that may or may not be usable”.

Quiz

Production discipline

• Verify at write time. snapshot status + hash immediately after snapshot save.
• Verify at upload time. Local SHA-256 vs S3 ETag.
• Restore drill quarterly. A snapshot whose restore has never been exercised is a snapshot whose restore will fail in the incident.
• Alert on snapshot age. A snapshot more than 2 hours old is a backup chain break.
• Capture schema content. Compare snapshot’s key count and named objects to the live cluster.

Validation is the discipline of converting “we have backups” from a hopeful statement into a verified statement. Every snapshot is verified; every drill exercises a real restore.