KubernetesLIV · Stateful WorkloadsStateful workloads
Application-consistency vs crash-consistency — what backups actually mean
What you'll learn
- Define application-consistency and crash-consistency
- Explain why crash-consistent snapshots can lose data
- Identify the techniques for application-consistency (quiesce, freeze, fsync)
- Apply the production discipline for choosing the right backup type
Prerequisites
Verified against Kubernetes 1.34.x · kubeadm 1.34.x · kubectl 1.34.x · etcd 3.6.x · CoreDNS 1.11.x · containerd 1.7.x / 2.x · 2026-08-16
The most important distinction in stateful workload backups is between application-consistent and crash-consistent. This lesson establishes the difference, why it matters, and the production discipline for achieving the right level of consistency.
What crash-consistent means
A crash-consistent snapshot captures the volume at one moment, as if the system had crashed:
flowchart LR
A[Disk write 1: txid=100] --> B[Disk write 2: txid=101]
B --> C[Snapshot]
C --> D[Disk write 3: txid=102]
The snapshot captures writes 1 and 2 but not 3. The volume is consistent in the sense that the on-disk structures are not torn (a write is either fully captured or not), but the application may have been in the middle of a transaction.
For a database, this means:
- The data files contain committed and uncommitted transactions.
- The WAL may have entries that have not been applied to the data files.
- On restore, the database must replay the WAL to reach a consistent state.
The restore procedure:
# Crash-consistent restore
# 1. Restore the data files from the snapshot
# 2. The database replays the WAL to reach a consistent state
# 3. Some transactions may be lost (those that were
# committed in the WAL but not applied to the data files
# before the snapshot)
What application-consistent means
An application-consistent snapshot is captured while the application is in a known quiescent state:
flowchart LR
A[Application quiesces] --> B[fsync / flush all data]
B --> C[Snapshot]
C --> D[Application resumes]
The application is paused (or its writes are flushed); the snapshot captures a state that the application considers valid; the application resumes.
For a database:
- All committed transactions are on disk.
- The WAL is consistent with the data files.
- The restore procedure does not need WAL replay; the snapshot is directly usable.
The production guarantee:
- All committed transactions are preserved.
- No uncommitted transactions are present.
- The restore is direct; no recovery procedure is needed.
The techniques
The techniques for achieving application-consistency:
Quiesce
The application is told to pause writes:
# PostgreSQL: pg_backup starts a backup mode
psql -c "SELECT pg_start_backup('snapshot-2026-08-16');"
# Application is in backup mode; writes are still allowed
# but the WAL is marked for the backup
Freeze (filesystem-level)
The filesystem is told to flush and quiesce:
# Linux: fsfreeze
fsfreeze -f /var/lib/postgresql/data
# Filesystem is frozen; no new writes
# Snapshot can now be taken
fsfreeze -u /var/lib/postgresql/data
# Filesystem is unfrozen
fsync
The application flushes all in-memory data to disk:
# PostgreSQL: CHECKPOINT forces all dirty pages to disk
psql -c "CHECKPOINT;"
Application hooks
The application has built-in hooks for snapshots:
# MySQL: FLUSH TABLES WITH READ LOCK
mysql -e "FLUSH TABLES WITH READ LOCK;"
# Snapshot can now be taken
mysql -e "UNLOCK TABLES;"
What can be lost with crash-consistent
A crash-consistent snapshot of a busy database can lose:
- Uncommitted transactions in the WAL: the transactions were committed in memory but not written to the data files.
- Pages in the buffer pool: the dirty pages in memory are lost.
- Torn writes: a write that was in progress at the snapshot moment is partially captured.
The PostgreSQL documentation quantifies the risk:
“It is not possible to make a fully consistent backup of a running database without coordination with the database server. The only safe way is to use pg_basebackup or pg_dump.”
In Kubernetes, the equivalent is to use a database operator (e.g., Zalando’s postgres-operator, the Cloud Native PG operator) that knows how to coordinate snapshots with the database.
The production pattern
The production pattern for stateful database backups:
sequenceDiagram
participant O as Operator
participant DB as Database
participant BE as Backend
O->>DB: quiesce (e.g., pg_start_backup)
O->>DB: CHECKPOINT (flush dirty pages)
O->>BE: create snapshot
BE-->>O: snapshot created
O->>DB: resume (e.g., pg_stop_backup)
Note over O,DB: snapshot is application-consistent
The operator:
- Tells the database to quiesce.
- The database flushes dirty pages and marks the WAL.
- The operator triggers the snapshot.
- The snapshot is captured while the database is in a quiescent state.
- The operator tells the database to resume.
The result is an application-consistent snapshot that can be restored directly, without WAL replay.
Quiz
Knowledge check · 4 questions
Q1. What is the difference between application-consistent and crash-consistent snapshots?
Q2. Most CSI drivers produce application-consistent snapshots by default.
Q3. Your team uses CSI snapshots for PostgreSQL backups. The restore fails because the database is corrupted. Walk through the diagnosis.
Team runs PostgreSQL on a StorageClass with CSI snapshots. The snapshot schedule runs nightly. On restore, the database reports corruption: "WAL ends before consistent state." The team uses the snapshot as-is.
Q4. Explain the techniques for achieving application-consistency in a snapshot.
Passing score: 75%. Answers are checked in this browser.
Production discipline
- Crash-consistent is the default; application-consistent is the production requirement. Coordinate with the application.
- Use a database operator. Cloud Native PG, Zalando’s postgres-operator, and similar operators know how to coordinate snapshots.
- Test the restore. A snapshot that is never restored is not a backup. Test with WAL replay; verify the restored database is consistent.
- Archive the WAL. For crash-consistent restore, WAL replay requires the WAL. Archive it.
- Document the consistency level. Each backup procedure has a documented consistency level; the restore procedure matches.