Backup cadence and retention — schedule, RPO, RTO, compliance

The cadence and retention of snapshots are decisions that bind RPO, RTO, compliance, and cost. Cadence is “how often we take a snapshot”; retention is “how long we keep each”. Each has a target shaped by the workload’s criticality and the regulatory regime. This lesson walks the choices, the trade-offs, and the production discipline that makes cadence a deliberate design decision.

The two axes

flowchart LR
    C[Snapshot cadence] -->|frequency| CAD[How often snapshots]
    R[Snapshot retention] -->|how long| RET[How long kept]

• Cadence is the period between snapshots. Hourly is the production minimum; daily is the typical base; weekly / monthly are tiered longer-retention snapshots.
• Retention is how long each snapshot is kept before deletion or lifecycle-tier transition.

The two axes combine into a tiered policy:

The RPO/RTO framing

• RPO (Recovery Point Objective): the maximum amount of data loss the business can tolerate. Hourly snapshots give 1-hour RPO. Daily gives 24-hour RPO. No snapshots would be unbounded data loss.
• RTO (Recovery Time Objective): the maximum time to recover. From snapshot, typical restore is 1-2 hours for control-plane state. Workload data is separate.

The cadence is the RPO. The retention is “how many snapshots we keep to meet the RPO”. A 1-hour RPO with 24-hour retention means “we have snapshots for the last 24 hours; in a worst case we lose at most 1 hour”.

Hourly cadence

The production minimum:

# /etc/systemd/system/etcd-snapshot.timer
[Timer]
OnCalendar=*:0  # top of every hour

Or a Kubernetes CronJob (schedule: "0 * * * *").

The operational discipline:

• Snapshot at the top of every hour.
• Verify (snapshot status, hash).
• Upload to tier 3 storage.
• Retain the most recent 24 (last 24 hours).
• Delete older snapshots.

A 24-hour retention of hourly snapshots covers the typical day; an incident within the day can be reverted to the previous hour.

Daily cadence

A daily snapshot is the typical snapshot that satisfies the 24-hour-RPO:

[Timer]
OnCalendar=*-*-* 02:00:00

Daily snapshots typically run at 02:00 (low-traffic window). The retention is 30-90 days. Daily snapshots are the operational workhorse.

$ ls /backup/daily/ | head -10

etcd-snapshot-20260815-0200.db
etcd-snapshot-20260814-0200.db
etcd-snapshot-20260813-0200.db
etcd-snapshot-20260812-0200.db

Weekly cadence

A weekly snapshot captures a longer-timescale steady state. Useful for:

• Compliance requirements with a “weekly retention” clause.
• Investigating regressions introduced in the past week.
• Disaster recovery scenarios where the latest daily is corrupt.

[Timer]
OnCalendar=Sun *-*-* 03:00:00

Retention: 12-26 weeks (depending on compliance).

Monthly and annual

Long-retention snapshots satisfy compliance regimes requiring years of cluster state:

[Timer]
OnCalendar=*-01 04:00:00

Retention: 7 years (some SOX and HIPAA regimes).

These snapshots are expensive in storage but rarely read. They live in cold storage and are the absolute last-resort recovery.

The costs

For a 2 GiB snapshot file (typical for a busy cluster), the hourlies are 48 GB; dailies 60 GB; weekly 24 GB; monthly 168 GB. Multiply by the off-cluster cost per GiB (typically $0.01-0.03/GiB-month on cold storage).

A production cluster running all four tiers spends ~$50-200/month on snapshot storage.

The verification load

Verification at each cadence costs operator time and compute. The trade-off:

The discipline of “verify at the cadence the snapshot is taken” prevents verification debt. A snapshot whose hash and status are not checked at write time is a snapshot whose defects surface only at restore.

Compliance framing

PCI-DSS requires daily backups retained for 12 months, plus 3 months immediately accessible. HIPAA requires six-year retention on relevant backups. SOC 2 looks for restoration testing.

The snapshot cadence that satisfies these regimes:

A bank’s cluster might run daily + weekly + monthly snapshots with multi-year retention; a small SaaS startup may run hourly + daily with 30-day retention. The decision follows the compliance regime and the RPO.

The reconciliation

gantt
    title etcd snapshot cadence (full)
    dateFormat HH:mm
    axisFormat %H:%M
    section Hourly
    Hourly snapshot :a1, 00:00, 2m
    section Daily
    Daily snapshot :b1, 02:00, 5m
    section Weekly
    Weekly snapshot :c1, Sun 03:00, 30m
    section Monthly
    Monthly snapshot :d1, 1st 04:00, 60m
    section Verification
    Hash verify :e1, 00:05, 1m
    Status check :e2, 02:10, 1m

The plan harmonises:

• 00:00: hourly snapshot.
• 02:00: daily snapshot (covers the previous day’s worth).
• 03:00 on Sunday: weekly snapshot.
• 04:00 on the 1st: monthly snapshot.
• Verification at write time for each.

Failure modes to plan around

• Cron fails for a day. The cluster’s RPO is breached for the gap; the alert fires. Restore plan relies on the daily snapshot.
• Cron fails for an hour. Hourly RPO is breached; daily snapshot is fresh enough.
• Snapshot is taken but not uploaded. Local snapshot covers host failure; off-cluster upload is the cross-host redundancy.
• Off-cluster upload fails for a week. Cluster is surviving on local snapshots; cross-host is the fallback.

Each failure is bounded. The cadence and retention combine to cover every realistic failure mode.

Quiz

Production discipline

• Hourly snapshots are the minimum. RPO under 1 hour requires hourly cadence.
• Daily + weekly + monthly for compliance. Long-tail retention covers HIPAA, PCI-DSS, SOC 2.
• Tier the storage. Recent snapshots in standard storage, long-retention in cold storage, to bound cost.
• Verify at every cadence. Snapshot status + hash at every write; restore drill quarterly.
• Test RTO. A 4-hour RTO without validation is not a 4-hour RTO; it is a guess. Drills validate it.

Cadence and retention are deliberate choices bound to RPO, RTO, and compliance. The discipline is to design explicitly, then operate to the design.