Skip to main content
RunBook Academy

← All break/fix scenarios in Backup & DR

advancedbdr-volume-snapshot~50 min

The volume snapshot restored and the application found its data half-written

Reported symptoms

  • ●A schema change to the rbdr-ledger service goes wrong at 03:40, and the team restores the rbdr-ledger-data claim from the 03:00 CSI VolumeSnapshot into a fresh claim
  • ●The restore finishes in four minutes, the restored pod reaches Ready in eleven seconds, the restore test records PASS, and the service returns to traffic at 04:12
  • ●At 09:20 the settlement reconciliation reports that the export files on the volume total 418000 minor units more than the ledger rows they are supposed to summarise
  • ●The difference is exactly one batch, rbdr-2026-08-28-0179: its export file is present on the volume and complete, and its thirty-four ledger rows are not in the database
  • ●Nothing is missing in a way anybody can point at - the claim is the same size as the source, every file the source held is present, and no restore step logged an error
  • ●The restored pod logged PostgreSQL crash recovery at startup, which nobody read at the time because the pod went Ready and the test said PASS
  • ●The monthly restore rehearsal has recorded PASS for nineteen consecutive months, and its only assertion is that the pod reaches Ready within five minutes

Evidence

  • · The hourly snapshot CronJob does exactly one thing: it creates a VolumeSnapshot object naming the claim. There is no exec into the pod before it, no filesystem freeze, no call into the application and no step after it, so the workload is never told a copy is being taken
  • · The VolumeSnapshot object itself reports readyToUse true with a restoreSize equal to the claim, so the storage layer did precisely what it was asked to do at the moment it was asked
  • · The restored PostgreSQL log records that the database system was not properly shut down and that automatic recovery was in progress, then redo starting, then an invalid record length where the WAL ran out, then redo done, then ready to accept connections
  • · The settlement register recorded a row count and a sum(amount) for 03:00 before the snapshot was requested; the exports half of the restored volume matches that register and the ledger half does not
  • · The restored claim is 64Gi against a 64Gi source, and a recursive inventory of the mounted volume returns the same 8214 files and the same total byte count as the inventory taken from the source at 03:00
  • · Every export file including the one for the disputed batch is present and complete, because the export directory was not mid-write at the instant the snapshot fired and the database was
  • · The restore test job runs kubectl wait --for=condition=Ready and then echoes PASS; it reads no row, no file and no checksum out of the restored volume
  • · The course capture shows that same assertion passing against a volume holding no application data at all: pod Ready after 6s, followed by cat being unable to open the orders file
Diagnosis and resolutionclick to reveal

Root cause

Two defects. The first put wrong data on the volume; the second is why the wrong data was carried back into production with a PASS beside it. **The immediate cause: the snapshot was taken at an instant of the storage's choosing, not one the application agreed to.** A CSI VolumeSnapshot is atomic at the block layer, so what it captured is a coherent image of the volume at 03:00:00 - exactly the state the volume would have held if power had been cut at that moment. That is crash consistency, and it is not the same property as an image the application would recognise. `rbdr-ledger` settles a batch in two movements: it commits the batch rows into the database, and it writes the batch export file the downstream billing system consumes. The snapshot instant fell between them. Crash recovery correctly discarded the transaction that had not committed and correctly kept the export file that had been written and fsynced, and the volume that came back therefore has two halves that are each internally consistent and that describe different worlds. Nothing in the pipeline could have moved that instant, because nothing in the pipeline talks to the workload. The CronJob creates one object. The CSI VolumeSnapshot API has no hook field that was left blank - the coordination has to be supplied by whatever creates the snapshot, and here nothing supplies it. **The control that should have caught it: the restore test asserted that the pod reached Ready.** A readiness condition reports that a container's probe returned success. The ledger's probe answers once PostgreSQL accepts connections, which it did, eleven seconds after the claim was mounted, having just told the log it had performed crash recovery. The test has passed nineteen times because starting is the one thing that was never in question. It has no independently recorded property of the business data to disagree with, so there is no volume state in the plausible range that it would have reported as a failure.

Remediation

Recover the missing batch first, then fix the two defects in the order that matters. Batch rbdr-2026-08-28-0179 exists in full in its export file on the restored volume, so the thirty-four rows are reconstructable; replay them into the ledger inside one transaction and re-run the reconciliation before touching anything else. Where a batch is not reconstructable from either half, the recovery point is the last instant at which the register and the volume agreed, and that number is what the incident report is written around. Then give the snapshot an instant the application chooses. The pre-snapshot step drains the settlement writer so that no batch is part-way between the ledger and the export directory, and holds the database's own backup bracket open for the duration of the snapshot: ```bash set -eu NS=rbdr-ledger SNAP=rbdr-ledger-0300 POD=$(kubectl -n "$NS" get pod -l app=rbdr-ledger \ -o jsonpath='{.items[0].metadata.name}') kubectl -n "$NS" exec "$POD" -c settlement -- rbdr-quiesce --begin kubectl -n "$NS" apply -f rbdr-ledger-snapshot.yaml kubectl -n "$NS" wait --for=jsonpath='{.status.readyToUse}'=true \ "volumesnapshot/$SNAP" --timeout=600s kubectl -n "$NS" exec "$POD" -c settlement -- rbdr-quiesce --end ``` `rbdr-quiesce` is a single in-pod helper rather than a sequence of `kubectl exec` calls because PostgreSQL's non-exclusive backup bracket has to be held open by one session: `pg_backup_start` and `pg_backup_stop` issued from two separate connections abort the bracket rather than establishing it. Then rewrite the restore test so that it can fail. It must read an independently recorded property of the business data out of the restored volume and compare it against the register written before the snapshot, and it must compare the ledger half against the exports half. A test that cannot distinguish this volume from a correct one is not evidence about either.

Verification

Verification is a comparison against something recorded before the snapshot, never a reading taken from the restored artefact alone. For each rehearsal: restore the snapshot into an isolated namespace, wait for the pod, and then compute the ledger row count, the ledger sum(amount) and the exports total from the restored volume. All three must equal the values the settlement register recorded at the snapshot instant, and the ledger sum and the exports total must equal each other. Record the three measured numbers in the rehearsal result, not the word PASS. Prove the new pre-snapshot step actually does something by running a rehearsal under load: drive settlement at production rate, take a coordinated snapshot, and confirm the register and both halves of the restored volume agree. Then repeat with the coordination disabled and confirm the test reports a mismatch. A control that has never been observed failing has not been tested. The pattern the capture demonstrates is the standard: after a point-in-time recovery it reported `rows recovered : 50000 (expected 50000)` and `sum(amount) : 825025000 (expected 825025000)`, and stated plainly that what was validated was not that the server started but that the data matched an independently recorded property of the business data from before the incident.

Prevention

**Name the consistency level of every recovery point.** Crash-consistent and application-consistent are different properties, and only one of them is what a bare volume snapshot gives you. A schedule that has never stated which one it produces has been claiming the stronger one by default. **Coordination is a step before the snapshot, not a setting inside it.** There is no field on a VolumeSnapshot that makes it application-aware. If nothing execs into the workload, freezes the filesystem or drives the application's own quiesce entry point, then the instant is whichever microsecond the scheduler fired at. **Find the invariants that span more than one file.** This workload's correctness lived across a database and a directory, so no single-file check could see the break. Write those invariants down, record their value alongside every recovery point, and make them the thing the restore test compares. **A readiness probe is not a data assertion.** Ready means a process answered. The capture shows a pod reaching Ready in six seconds on a volume that held none of the application's data, which is the same result the ledger's rehearsal produced nineteen times. **Ask of every green control what state would have turned it red.** If the answer cannot be written as a concrete volume state, the control is reporting on something other than the thing being protected.

Reported symptoms

At 03:40 a schema change to rbdr-ledger goes wrong. The team restores the rbdr-ledger-data claim from the 03:00 CSI VolumeSnapshot. The pod reaches Ready eleven seconds after the mount, the restore test records PASS, and the service is back in traffic at 04:12.

At 09:20 the settlement reconciliation reports that the export files on the volume total 418000 minor units more than the ledger rows they summarise. The whole difference is one batch, rbdr-2026-08-28-0179: its export file is on the volume and complete, its thirty-four ledger rows are not in the database.

Nothing is missing in a way anybody can point at. The claim is the same size as the source, every file the source held is present, and no restore step logged an error.

Evidence provided

First, what takes the snapshot.

Read-only / Safethe entire snapshot pipeline
$ kubectl -n rbdr-ledger get cronjob rbdr-snapshot-hourly -o yaml
spec:
schedule: "0 * * * *"
jobTemplate:
  spec:
    template:
      spec:
        serviceAccountName: rbdr-snapshotter
        restartPolicy: OnFailure
        containers:
        - name: snapshot
          image: registry.rbdr.example/rbdr-kubectl:1.36
          args:
          - snapshot
          - --namespace=rbdr-ledger
          - --claim=rbdr-ledger-data
          - --class=rbdr-csi-ssd
          - --name-prefix=rbdr-ledger

Illustrative output

That is the whole job, and the object it creates is equally plain:

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: rbdr-ledger-0300
  namespace: rbdr-ledger
spec:
  volumeSnapshotClassName: rbdr-csi-ssd
  source:
    persistentVolumeClaimName: rbdr-ledger-data

No exec into the pod before it, no filesystem freeze, no call into the application, no step after it. The snapshot reported readyToUse: true with a restoreSize equal to the claim.

Second, the restored application’s own account of what it found. The pod’s log records the sequence below, quoted from the course capture of an uncoordinated copy of a running cluster.

Read-only / Safethe application repairs what it can and says so
$ pg_ctl -D /work/naive-copy start
  2026-08-28 13:34:37.611 UTC [94] LOG:  database system was interrupted; last known up at 2026-08-28 13:34:36 UTC
2026-08-28 13:34:37.613 UTC [94] LOG:  database system was not properly shut down; automatic recovery in progress
2026-08-28 13:34:37.613 UTC [94] LOG:  redo starts at 0/17615F8
2026-08-28 13:34:37.634 UTC [94] LOG:  invalid record length at 0/256A8D8: expected at least 24, got 0
2026-08-28 13:34:37.634 UTC [94] LOG:  redo done at 0/256A8B0 system usage: CPU: user: 0.01 s, system: 0.00 s, elapsed: 0.02 s
2026-08-28 13:34:37.639 UTC [88] LOG:  database system is ready to accept connections

Third, the invariant recorded before the snapshot, against what came back.

Read-only / Safetwo halves of one volume, disagreeing by one batch
$ rbdr-verify-restore --register /var/lib/rbdr/settlement-register.json
recorded 2026-08-28T03:00:00Z, before the snapshot was requested
ledger rows   : 1284416
sum(amount)   : 918447320
exports total : 918447320

read from the restored claim rbdr-ledger-data-restore
ledger rows   : 1284382     (-34)
sum(amount)   : 918029320   (-418000)
exports total : 918447320   (matches the register)

MISMATCH - batch rbdr-2026-08-28-0179 is in the exports and not in the ledger

Illustrative output

Fourth, the state of the restored volume itself.

Read-only / Safesame capacity, same file set, same bytes
$ kubectl -n rbdr-restore get pvc rbdr-ledger-data-restore; rbdr-inventory /mnt/rbdr-restore
NAME                       STATUS   CAPACITY   ACCESS MODES   STORAGECLASS
rbdr-ledger-data-restore   Bound    64Gi       RWO            rbdr-csi-ssd

                      files   bytes         inventory digest
source claim at 03:00    8214   61203847168   sha256:4c1f9ab2
restored claim           8214   61203847168   sha256:4c1f9ab2

Illustrative output

Fifth, the control. The rehearsal job is three lines:

- kubectl -n rbdr-restore apply -f rbdr-ledger-from-snapshot.yaml
- kubectl -n rbdr-restore wait --for=condition=Ready pod -l app=rbdr-ledger --timeout=300s
- echo PASS

The course capture shows how far that assertion sits from the data it is taken to be about.

Read-only / SafeReady in six seconds, on a volume with nothing in it
$ kubectl apply -f rbdr-shop.yaml; kubectl -n rbdr-shop exec rbdr-orders-writer -- cat /data/orders.csv
  namespace/rbdr-shop created
persistentvolumeclaim/rbdr-orders created
configmap/rbdr-config created
pod/rbdr-orders-writer created
>>> exit code: 0
pod Ready after 6s

--- every object is back ---
persistentvolumeclaim/rbdr-orders   Bound   pvc-a29538af-09d4-4bb8-82bb-b2c222b9db93   64Mi   RWO   local-path   <unset>   6s
configmap/kube-root-ca.crt   1     6s
configmap/rbdr-config        1     6s
pod/rbdr-orders-writer   1/1   Running   0     6s

--- and the business data? ---
total 8
drwxrwxrwx    2 root     root          4096 Aug 28 14:34 .
drwxr-xr-x    1 root     root          4096 Aug 28 14:34 ..
cat: can't open '/data/orders.csv': No such file or directory
command terminated with exit code 1

Work the evidence before reading on

  1. The snapshot reported readyToUse and a restoreSize equal to the claim. What does that cover, and what does it say nothing about?
  2. The inventory and the capacity match the source exactly. Which hypothesis does that eliminate, and which does it leave standing?
  3. PostgreSQL logged crash recovery and then started. A fault in the restore, or the expected behaviour of the thing captured?
  4. Nineteen PASS results. Name one volume state that would have made the rehearsal fail.
  5. The exports half matches the register and the ledger half does not. What does that asymmetry say about the moment the snapshot fired?

Root cause

The snapshot was taken at an instant of the storage’s choosing

A CSI VolumeSnapshot is atomic at the block layer, so what came back is a coherent image of the volume at 03:00:00 — the state it would have held if power had been cut then. That is crash consistency, and the restored cluster showed it working: redo replayed, stopped where the WAL ran out, and the server opened.

What it does not supply is an instant the application would have chosen. rbdr-ledger settles a batch in two movements — it commits the batch rows into the database, and it writes the batch’s export file for the downstream billing system. 03:00:00 landed between them. Recovery discarded the uncommitted transaction and kept the fsynced export file, both correct. The volume that came back has two halves that are each internally consistent and that describe different worlds.

Nothing in the pipeline could have moved that instant, because nothing in it speaks to the workload. No field on the VolumeSnapshot was left blank: the API has no hook mechanism, so coordination must come from whatever creates the object, and the CronJob creates it and exits.

The restore test asserted readiness, not correctness

The rehearsal waits for condition=Ready and echoes PASS. A readiness condition reports that a container’s probe returned success; the ledger’s probe answers as soon as PostgreSQL accepts connections, which it did eleven seconds after the mount, having just written a crash-recovery sequence to its log.

It recorded PASS nineteen times because starting was never in question. It reads nothing out of the volume and holds no independently recorded property of the business data to disagree with, so no plausible volume state would have been reported as a failure. The capture above is the same assertion passing on a volume holding none of the application’s data.

Resolution

Recover the batch first. rbdr-2026-08-28-0179 exists in full in its export file, so the thirty-four rows are reconstructable; replay them into the ledger inside one transaction and re-run the reconciliation before anything else is touched. Where a batch survives in neither half, the recovery point is the last instant at which the register and the volume agreed.

Then give the snapshot an instant the application chooses. The pre-snapshot step drains the settlement writer so no batch is part-way between the ledger and the export directory, and holds the database’s backup bracket open for the capture.

set -eu
NS=rbdr-ledger
SNAP=rbdr-ledger-0300
POD=$(kubectl -n "$NS" get pod -l app=rbdr-ledger \
  -o jsonpath='{.items[0].metadata.name}')

kubectl -n "$NS" exec "$POD" -c settlement -- rbdr-quiesce --begin
kubectl -n "$NS" apply -f rbdr-ledger-snapshot.yaml
kubectl -n "$NS" wait --for=jsonpath='{.status.readyToUse}'=true \
  "volumesnapshot/$SNAP" --timeout=600s
kubectl -n "$NS" exec "$POD" -c settlement -- rbdr-quiesce --end

rbdr-quiesce is one in-pod helper rather than a pair of kubectl exec calls because PostgreSQL’s non-exclusive backup bracket must be held by a single session: pg_backup_start and pg_backup_stop issued over two connections abort the bracket instead of establishing it.

Finally, rewrite the rehearsal so it is capable of failing.

Verification

Verification compares the restore against something recorded before the snapshot, never against a reading taken from the restored artefact alone.

Each rehearsal restores into an isolated namespace, waits for the pod, then computes the ledger row count, the ledger sum(amount) and the exports total from the restored volume. All three must equal what the settlement register recorded at the snapshot instant, and the ledger sum and the exports total must equal each other. The result records those three numbers, not the word PASS.

Then prove the pre-snapshot step does something: drive settlement at production rate, take a coordinated snapshot, confirm all three numbers agree, and repeat with the coordination disabled to confirm the rehearsal reports a mismatch. A control never observed failing has not been tested.

Read-only / Safethe standard the rehearsal has to meet
$ psql -p 5434 -Atc 'SELECT count(*), sum(amount) FROM orders'
  rows recovered  : 50000   (expected 50000)
sum(amount)     : 825025000   (expected 825025000)
RECOVERED - row count and business checksum both match the pre-DELETE state

Note what was validated: not that the server started, but that the
data it now holds matches an independently recorded property of the
business data from before the incident.

Prevention

Name the consistency level of every recovery point. Crash-consistent and application-consistent are different properties, and a bare volume snapshot supplies only the first. A schedule that never states which it produces has been claiming the stronger one by default.

Coordination is a step before the snapshot, not a setting inside it. No field on a VolumeSnapshot makes it application-aware. If nothing execs into the workload, freezes the filesystem or drives its quiesce entry point, the instant is whichever microsecond the scheduler fired at. Velero documents pre- and post-backup hooks for this reason; a hand-rolled CronJob supplies the equivalent itself or does without.

Find the invariants that span more than one file. This workload’s correctness lived across a database and a directory, so no single-file check could see the break. Record their value beside every recovery point.

A readiness probe is not a data assertion. Ready means a process answered.

Ask of every green control what state would have turned it red. If that cannot be written down as a concrete volume state, the control reports on something other than the thing being protected.