Application-consistent backups — database quiescence and ordering

A CSI volume snapshot is crash-consistent: it captures whatever was in flight at the instant the snapshot was requested. For a transactional database — Postgres, MySQL, MongoDB, etcd — a crash-consistent snapshot can land mid-transaction, mid-write-ahead-log record, mid-fsync. Restoring such a snapshot often requires the database to replay its WAL to recover, and replay can fail if the snapshot is not actually consistent. Application-consistent backups quiesce the database before the snapshot, then release it after. This lesson covers the quiesce pattern and its operational discipline.

Crash-consistent vs application-consistent

The two consistency models:

flowchart LR
    A[Database writes] --> B[Buffer cache]
    B --> C[Write-ahead log]
    C --> D[Data files]
    E[CSI snapshot] -->|captures| D
    F[Time] --> G[t0]
    G --> H["t1: snapshot"]
    H --> I["t2: write completes"]
    E -.-> H

At t1 the snapshot captures the data files mid-write. The WAL record for the transaction is in the WAL but not yet in the data file. On restore, the database sees a half-applied transaction. Postgres will detect this and refuse to start; MySQL with InnoDB will crash-recover, but may lose recent committed transactions; etcd will fail to start because its WAL is inconsistent with its snapshot.

flowchart LR
    A[PRE hook] -->|flush, lock| B[Database quiesced]
    B --> C[CSI snapshot]
    C --> D[POST hook]
    D -->|unlock| E[Database running]

An application-consistent backup coordinates the quiesce and the snapshot so the data files and the WAL are in a consistent state at snapshot time. This is the only safe way to back up a transactional database.

The quiesce hook pattern

The hook pattern uses annotations on the workload that the backup tool reads and acts on. Velero, for example, supports pre.hook.backup.velero.io and post.hook.backup.velero.io annotations, plus a container-image hook and an exec hook. The pattern:

1. PRE hook: the database is told to flush its buffers, take a checkpoint, and either freeze writes or accept them but mark them with the snapshot’s LSN.
2. CSI snapshot: the storage backend captures the volume atomically.
3. POST hook: the database is told to resume normal operation, log the snapshot’s identity, and (in some engines) garbage-collect the WAL before the snapshot.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
  annotations:
    pre.hook.backup.velero.io/container: db
    pre.hook.backup.velero.io/command: '["/scripts/quiesce.sh", "freeze"]'
    post.hook.backup.velero.io/container: db
    post.hook.backup.velero.io/command: '["/scripts/quiesce.sh", "thaw"]'

The ordering problem

The hardest part of application-consistent backups is ordering. Three things must happen in order, and any out-of-order execution corrupts the backup:

If step 1 and step 2 race, the snapshot is inconsistent. If step 2 and step 3 race, the database is frozen longer than necessary (writes pile up, applications see stalls). If step 1 or step 2 fails, the POST hook must not run — the database would be released into a state where the backup is missing.

The database-specific mechanics

Each database has its own quiesce semantics:

• Postgres: pg_start_backup('label') writes a backup label and forces a checkpoint, then pg_stop_backup() writes the WAL stop record and releases. The snapshot is taken between the two calls.
• MySQL (InnoDB): FLUSH TABLES WITH READ LOCK plus SET GLOBAL innodb_backup_adaptive_flush=ON and a checkpoint. The lock blocks writes; the snapshot must happen during the lock window, then UNLOCK TABLES.
• etcd: etcdctl snapshot save is itself application-consistent — etcd’s MVCC store supports point-in-time snapshots without quiesce. Backing up etcd by snapshotting its volume is incorrect; use etcdctl snapshot save.
• MongoDB: db.fsyncLock() blocks writes, the snapshot is taken, then db.fsyncUnlock(). The lock window must be shorter than the oplog retention or secondary replicas will fall behind.

A Postgres quiesce script:

#!/bin/bash
case "$1" in
  freeze)
    psql -U postgres -c "SELECT pg_start_backup('velero-snapshot');"
    ;;
  thaw)
    psql -U postgres -c "SELECT pg_stop_backup();"
    ;;
esac

The operational failure modes

Application-consistent backups fail in production for predictable reasons:

• Hook hangs. The PRE hook runs pg_start_backup but the database is locked by another session; the hook blocks indefinitely and the snapshot never starts. Add timeouts and idempotency.
• Snapshot races the hook. The schedule fires the CSI snapshot before the hook completes; the backup is crash-consistent. Add a wait-for-hook-completion barrier.
• Hook runs but the snapshot fails. The PRE hook fired, the database is frozen, the snapshot request returns an error, and the POST hook never runs because the backup tool does not chain the failure path. The database stays frozen.
• POST hook fails. The snapshot succeeded but pg_stop_backup failed. The next backup’s PRE hook will block because the previous backup was never closed.
• WAL is not in the snapshot. The data directory is on PVC A and the WAL is on PVC B; only PVC A is snapshotted; restore is incomplete.

Quiz

Production discipline

Application-consistent backups in production rest on four non-negotiable elements:

• Quiesce every transactional workload. Databases, message brokers, search indexes — anything with on-disk state that must be coherent across snapshot.
• Verify the hook chain. The hook must complete before the snapshot starts; the snapshot must complete before the hook releases. The toolchain must enforce the ordering; ad-hoc scripts do not.
• Test restore, not just backup. A snapshot of an apparently-quiesced database that on restore fails WAL replay is not a backup. Quarterly restore tests in a sandbox cluster are the proof.
• Document the WAL and data layout. If the WAL lives on a different PVC than the data, the backup program must snapshot both and combine them on restore. Documenting this is part of the runbook.

Application-consistent backups are a coordinated choreography. Skipping any step turns the backup into a crash-consistent snapshot, which for transactional workloads is worse than no backup at all — because the operator believes the data is safe when it is not.