Backup storage strategy — local, off-cluster, object storage

A snapshot that lives on the same host as the etcd member is a snapshot that dies with the host. Off-cluster storage is the production answer, with object storage the standard target because of its versioning, lifecycle, and multi-region capabilities. This lesson walks the storage tiers, the configuration patterns, and the operational discipline.

The storage tiers

flowchart LR
    L1[Tier 1: local fast volume] -->|async upload| L2[Tier 2: cross-host NFS]
    L2 -->|async upload| L3[Tier 3: object storage primary region]
    L3 -->|cross-region replication| L4[Tier 4: object storage DR region]

The tiers:

• Tier 1 (local fast volume): writes from the snapshot operation complete here in seconds. Lost with the host.
• Tier 2 (cross-host NFS): a shared volume across hosts; survives single-host failure.
• Tier 3 (object storage primary region): survives zone and most region-disrupting events.
• Tier 4 (object storage DR region): survives regional disaster; the longest-tail recovery primitive.

A small cluster may compress this to two tiers; a production-grade cluster runs all four. The cost of each tier is dominated by storage and egress; tier 4 typically costs an order of magnitude more than tier 1.

Tier 1 — local fast volume

The typical implementation:

# A dedicated NVMe-backed volume mounted at /backup
# Owned by root with restricted permissions
chmod 700 /backup
ls -la /backup
# drwx------  2 root root  4096 Aug 16 12:00 .

The local volume is sized for a small number of snapshots (e.g., last 24 hours of hourly = 24 files at ~1 GiB each = 24 GiB). The pattern:

• Snapshot lands at /backup/etcd-snapshot-YYYYMMDD-HHMM.db.
• After successful upload to tier 3, the local file is retained for 24 hours and deleted by a cleanup job.

The volume’s I/O profile: small writes when the snapshot is written (~1 GiB in seconds), no reads except by the verification job. The volume can be the same physical device as the etcd data dir if needed; the operational risk is that an etcd member’s local failure takes the recent snapshot too.

$ ls -la /backup/

total 8
-rw------- 1 root root 82419000 Aug 16 02:00 etcd-snapshot-20260816-0200.db
-rw------- 1 root root 82419000 Aug 16 03:00 etcd-snapshot-20260816-0300.db
-rw------- 1 root root 82419000 Aug 16 04:00 etcd-snapshot-20260816-0400.db

Tier 2 — cross-host storage

A shared filesystem across hosts is the most common tier 2:

• NFS export on a separate host.
• CephFS gluster or similar — co-located with other tier 2 / tier 3 storage in production.
• DRBD mirror in HA configurations.

The trade-off: tier 2 protects against a single host failure but not a network partition that isolates the NFS server. The etdctl snapshot to NFS is straightforward:

etcdctl snapshot save /mnt/nfs/etcd-snapshot.db

The NFS mount should be HA (multiple NFS server addresses or a replicated backing store) to survive the failure of the NFS host.

flowchart LR
    E1[etcd host 1] -->|nfs mount| N[NFS server HA pair]
    E2[etcd host 2] -->|nfs mount| N
    E3[etcd host 3] -->|nfs mount| N
    N -->|sync to| S3[object storage tier 3]

Tier 3 — object storage primary

Object storage (S3, GCS, Azure Blob, MinIO, Swift) is the standard tier 3:

• Multi-AZ redundancy by default.
• Versioning (every object keeps its history).
• Lifecycle policy (move old snapshots to cold storage, delete after retention).
• Server-side encryption.

The upload pattern:

etcdctl snapshot save /tmp/etcd-snapshot.db
aws s3 cp /tmp/etcd-snapshot.db \
  s3://prod-etcd-backups/etcd/snapshot-$(date +%Y%m%d-%H%M).db \
  --sse aws:kms \
  --storage-class STANDARD_IA

# Verify upload
aws s3 ls s3://prod-etcd-backups/etcd/ | tail -10

Object storage is the canonical answer for production etcd snapshots.

Versioning and lifecycle

Object storage versioning keeps multiple versions of a snapshot file (the same name with different timestamps is not versioning; versioning keeps prior versions when an object is overwritten).

{
  "Rules": [
    {
      "ID": "MoveOldToIA",
      "Status": "Enabled",
      "Transitions": [
        {
          "Days": 7,
          "StorageClass": "GLACIER"
        }
      ]
    },
    {
      "ID": "ExpireAfter90Days",
      "Status": "Enabled",
      "Expiration": {
        "Days": 90
      }
    }
  ]
}

The lifecycle policy:

• 0-7 days: standard storage (fast restore).
• 7-90 days: cold storage (cheap; minutes-to-hours restore).
• 90+ days: expired.

A daily snapshot is typically retained for 30-90 days; hourly snapshots are retained for 24-48 hours; weekly monthlies are retained for years (compliance-driven).

Tier 4 — cross-region DR

flowchart LR
    P[Primary region S3] -->|cross-region replication| D[DR region S3]
    D -->|periodic restore drill| T[Test restore]

Cross-region replication:

• AWS S3 cross-region replication (CRR).
• GCS multi-region bucket.
• Azure blob geo-redundancy.

Tier 4 is enabled for production DR. The cost is the egress and the storage duplication; the benefit is recovery survives a regional disaster.

# AWS CLI cross-region replication is a bucket-level configuration
aws s3api put-bucket-replication \
  --bucket prod-etcd-backups \
  --replication-configuration file://replication.json

The replication has its own latency (eventual consistency, minutes to hours for new snapshots). The DR region’s freshness is “up to ~1 hour behind the primary”.

The retention plan

A common retention mix:

The numbers depend on:

• Recovery RTO/RPO. Hourly snapshots give 1-hour RPO. Daily gives 24-hour RPO.
• Compliance. PCI, HIPAA, SOC 2 may require 7 years of retention.
• Cost. Hourly at 1 hour × 24 = 24 files; daily at 1 per day × 30 = 30 files. Storage scales with retention.

Integrity and immutability

The objective is that the snapshot cannot be tampered with:

• Object lock on S3 (compliance mode) prevents deletion within the retention window.
• Versioning keeps prior versions even after a deletion request.
• KMS-managed encryption keys with role-based access.

flowchart LR
    S[Snapshot file] -->|upload| L[S3 with object lock]
    L -->|immutable for 30 days| V[Tamper-evidence]
    L -->|after 30 days| D[Lifecycle expiry]

Object lock is the answer for compliance-grade immutability. Object lock’s compliance mode means even the root account cannot delete the object during the lock window.

The verification loop

Each snapshot should be verified within minutes of being written. The verification:

• Hash check (the snapshot’s hash is consistent with the source bbolt DB).
• etcdutl snapshot status reports non-zero revision and key count.
• Optional: a small-scale restore on a sandbox host.

The verification result feeds into the alerting system: a failed snapshot verification is the cluster’s “backup chain broken” alert.

Quiz

Production discipline

• Tier 1 is not a backup. A snapshot on the etcd host is a copy, not a backup. Push past tier 1 within minutes.
• Tier 3 is the production minimum. Object storage with versioning, lifecycle, and SSE.
• Tier 4 is the DR goal. Cross-region replication covers regional failure; test it quarterly.
• Retention is policy-driven. Compliance and RPO/RTO frame the policy; cost frames the optimisation.
• Verification at write-time is mandatory. A snapshot written but not verified is a snapshot that may not restore.

Backup storage is the discipline of “what survives what failure”. The four tiers map to four failure modes: host, site, region, and corruption.