Skip to main content
RunBook Academy

KubernetesLIV · Stateful WorkloadsStateful workloads

Application-consistency vs crash-consistency — what backups actually mean

Advanced⏱ ~16 minkubectl

What you'll learn

  • Define application-consistency and crash-consistency
  • Explain why crash-consistent snapshots can lose data
  • Identify the techniques for application-consistency (quiesce, freeze, fsync)
  • Apply the production discipline for choosing the right backup type

Prerequisites

Verified against Kubernetes 1.34.x · kubeadm 1.34.x · kubectl 1.34.x · etcd 3.6.x · CoreDNS 1.11.x · containerd 1.7.x / 2.x · 2026-08-16

Not yet marked complete on this device.

The most important distinction in stateful workload backups is between application-consistent and crash-consistent. This lesson establishes the difference, why it matters, and the production discipline for achieving the right level of consistency.

What crash-consistent means

A crash-consistent snapshot captures the volume at one moment, as if the system had crashed:

flowchart LR
    A[Disk write 1: txid=100] --> B[Disk write 2: txid=101]
    B --> C[Snapshot]
    C --> D[Disk write 3: txid=102]

The snapshot captures writes 1 and 2 but not 3. The volume is consistent in the sense that the on-disk structures are not torn (a write is either fully captured or not), but the application may have been in the middle of a transaction.

For a database, this means:

  • The data files contain committed and uncommitted transactions.
  • The WAL may have entries that have not been applied to the data files.
  • On restore, the database must replay the WAL to reach a consistent state.

The restore procedure:

# Crash-consistent restore
# 1. Restore the data files from the snapshot
# 2. The database replays the WAL to reach a consistent state
# 3. Some transactions may be lost (those that were
#    committed in the WAL but not applied to the data files
#    before the snapshot)

What application-consistent means

An application-consistent snapshot is captured while the application is in a known quiescent state:

flowchart LR
    A[Application quiesces] --> B[fsync / flush all data]
    B --> C[Snapshot]
    C --> D[Application resumes]

The application is paused (or its writes are flushed); the snapshot captures a state that the application considers valid; the application resumes.

For a database:

  • All committed transactions are on disk.
  • The WAL is consistent with the data files.
  • The restore procedure does not need WAL replay; the snapshot is directly usable.

The production guarantee:

  • All committed transactions are preserved.
  • No uncommitted transactions are present.
  • The restore is direct; no recovery procedure is needed.

The techniques

The techniques for achieving application-consistency:

Quiesce

The application is told to pause writes:

# PostgreSQL: pg_backup starts a backup mode
psql -c "SELECT pg_start_backup('snapshot-2026-08-16');"
# Application is in backup mode; writes are still allowed
# but the WAL is marked for the backup

Freeze (filesystem-level)

The filesystem is told to flush and quiesce:

# Linux: fsfreeze
fsfreeze -f /var/lib/postgresql/data
# Filesystem is frozen; no new writes
# Snapshot can now be taken
fsfreeze -u /var/lib/postgresql/data
# Filesystem is unfrozen

fsync

The application flushes all in-memory data to disk:

# PostgreSQL: CHECKPOINT forces all dirty pages to disk
psql -c "CHECKPOINT;"

Application hooks

The application has built-in hooks for snapshots:

# MySQL: FLUSH TABLES WITH READ LOCK
mysql -e "FLUSH TABLES WITH READ LOCK;"
# Snapshot can now be taken
mysql -e "UNLOCK TABLES;"

What can be lost with crash-consistent

A crash-consistent snapshot of a busy database can lose:

  • Uncommitted transactions in the WAL: the transactions were committed in memory but not written to the data files.
  • Pages in the buffer pool: the dirty pages in memory are lost.
  • Torn writes: a write that was in progress at the snapshot moment is partially captured.

The PostgreSQL documentation quantifies the risk:

“It is not possible to make a fully consistent backup of a running database without coordination with the database server. The only safe way is to use pg_basebackup or pg_dump.”

In Kubernetes, the equivalent is to use a database operator (e.g., Zalando’s postgres-operator, the Cloud Native PG operator) that knows how to coordinate snapshots with the database.

The production pattern

The production pattern for stateful database backups:

sequenceDiagram
    participant O as Operator
    participant DB as Database
    participant BE as Backend
    O->>DB: quiesce (e.g., pg_start_backup)
    O->>DB: CHECKPOINT (flush dirty pages)
    O->>BE: create snapshot
    BE-->>O: snapshot created
    O->>DB: resume (e.g., pg_stop_backup)
    Note over O,DB: snapshot is application-consistent

The operator:

  1. Tells the database to quiesce.
  2. The database flushes dirty pages and marks the WAL.
  3. The operator triggers the snapshot.
  4. The snapshot is captured while the database is in a quiescent state.
  5. The operator tells the database to resume.

The result is an application-consistent snapshot that can be restored directly, without WAL replay.

Quiz

Knowledge check · 4 questions

  1. Q1. What is the difference between application-consistent and crash-consistent snapshots?

  2. Q2. Most CSI drivers produce application-consistent snapshots by default.

  3. Q3. Your team uses CSI snapshots for PostgreSQL backups. The restore fails because the database is corrupted. Walk through the diagnosis.

    Team runs PostgreSQL on a StorageClass with CSI snapshots. The snapshot schedule runs nightly. On restore, the database reports corruption: "WAL ends before consistent state." The team uses the snapshot as-is.

  4. Q4. Explain the techniques for achieving application-consistency in a snapshot.

Passing score: 75%. Answers are checked in this browser.

Production discipline

  • Crash-consistent is the default; application-consistent is the production requirement. Coordinate with the application.
  • Use a database operator. Cloud Native PG, Zalando’s postgres-operator, and similar operators know how to coordinate snapshots.
  • Test the restore. A snapshot that is never restored is not a backup. Test with WAL replay; verify the restored database is consistent.
  • Archive the WAL. For crash-consistent restore, WAL replay requires the WAL. Archive it.
  • Document the consistency level. Each backup procedure has a documented consistency level; the restore procedure matches.