etcdctl snapshot save — flags, options, and integration

The etcdctl snapshot save command is the primary operator-driven backup primitive. It runs against a live member, reads the bbolt file coherently, and writes a self-contained snapshot file. This lesson walks the arguments, the runtime considerations, the typical scheduling, and the integration into a kubeadm-managed cluster.

The command in one line

ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /backup/etcd-snapshot.db

The arguments:

The destination path can be a local file, a mounted NFS, or a writer that writes through to object storage.

Running against a leader vs follower

The snapshot operation reads the member’s bbolt file. The read is a point-in-time consistent view (the file is opened with mmap and the read is bounded). The operation cost:

flowchart LR
    R[read bbolt file] -->|mmap copy| S[snapshot file]
    R -->|page in from disk if cold| DISK[disk read]
    R -->|page cache hit| MEM[RAM]

For a 500 MB bbolt file with a hot page cache, the operation takes seconds. For a cold page cache or a 2 GiB DB, it can take 30 seconds to several minutes.

The runtime considerations:

• Follower is preferable. A snapshot of a follower does not interrupt the leader’s commit path. Schedule the snapshot against a follower if there are multiple members.
• Leader is acceptable but watch latency. A snapshot of the leader holds the bbolt file open for the duration of the read; commits continue, but the operation competes with the WAL fsync path for the disk. Schedule during low traffic.
• Network is irrelevant. The snapshot reads from local disk; the network is not on the path.

$ ETCDCTL_API=3 etcdctl --endpoints=https://10.0.1.10:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key snapshot save /backup/etcd-snapshot-$(date +%Y%m%d-%H%M).db

"saved snapshot to /backup/etcd-snapshot-20260816-1200.db"

The arguments in depth

--endpoints

--endpoints=https://10.0.1.11:2379

Selects the target member. Multiple endpoints separated by commas are tried in order. The first reachable member gets the snapshot. In practice, point at a known follower to avoid the leader.

--cacert, --cert, --key

--cacert=/etc/kubernetes/pki/etcd/ca.crt
--cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt
--key=/etc/kubernetes/pki/etcd/healthcheck-client.key

The ca.crt is the kubeadm-managed CA. The client cert is any client cert from that CA; healthcheck-client is a kubeadm default that has read access.

snapshot save <path>

The destination. May be a local path, a writable NFS, or a writer that uploads to object storage. Standard file write semantics apply: the operation is atomic at the filesystem level only if the filesystem supports it.

Other useful arguments

• --wal-dir <path>: for cases where WAL files are on a separate volume.
• --data-dir <path>: source bbolt dir; default is the member’s data dir.
• --dial-timeout <seconds>: timeout for the gRPC dial; default is fine in production.

The destination

# Local path
etcdctl snapshot save /backup/etcd-snapshot.db

# NFS mount
etcdctl snapshot save /mnt/nfs-backup/etcd-snapshot.db

# Remote via s3 (using a wrapper)
( etcdctl snapshot save /tmp/etcd-snapshot.db && aws s3 cp /tmp/etcd-snapshot.db s3://bucket/etcd/ ) &

The destination pattern in production:

• Write to a local fast volume first.
• Asynchronously upload to object storage.
• Never write directly to slow storage as the primary destination (the snapshot operation must complete in seconds for the schedule to keep up).

Scheduling the snapshot

The most common production patterns:

Systemd timer

# /etc/systemd/system/etcd-snapshot.service
[Unit]
Description=etcd snapshot save

[Service]
Type=oneshot
Environment="ETCDCTL_API=3"
ExecStart=/usr/local/bin/etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
  --key=/etc/kubernetes/pki/etcd/healthcheck-client.key \
  snapshot save /backup/etcd-snapshot-$(date +%Y%m%d-%H%M).db

# /etc/systemd/system/etcd-snapshot.timer
[Unit]
Description=Run etcd snapshot hourly

[Timer]
OnCalendar=hourly
Persistent=true

[Install]
WantedBy=timers.target

Kubernetes CronJob (against a pod with host paths)

apiVersion: batch/v1
kind: CronJob
metadata:
  name: etcd-snapshot
  namespace: kube-system
spec:
  schedule: "0 * * * *"
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          hostNetwork: true
          nodeSelector:
            node-role.kubernetes.io/control-plane: ""
          containers:
          - name: etcdctl
            image: registry.k8s.io/etcd:3.5.x-0
            command:
            - /bin/sh
            - -c
            - |
              ETCDCTL_API=3 etcdctl \
                --endpoints=https://127.0.0.1:2379 \
                --cacert=/etc/kubernetes/pki/etcd/ca.crt \
                --cert=/etc/kubernetes/pki/etcd/server.crt \
                --key=/etc/kubernetes/pki/etcd/server.key \
                snapshot save /backup/etcd-snapshot-$(date -u +%Y%m%d-%H%M).db
            volumeMounts:
            - name: etcd-certs
              mountPath: /etc/kubernetes/pki/etcd
              readOnly: true
            - name: backup
              mountPath: /backup
          volumes:
          - name: etcd-certs
            hostPath:
              path: /etc/kubernetes/pki/etcd
          - name: backup
            hostPath:
              path: /backup

The CronJob runs hourly; the hostPaths must match the control-plane node. The nodeSelector ensures the pod lands on a control-plane node. The certs are mounted read-only.

The runtime impact on the live cluster

Snapshot save involves disk reads and minimal CPU. The read I/O contends with:

• WAL fsyncs. The leader’s WAL writes compete with the snapshot read I/O. On a slow disk, this can cause momentary spikes in etcd_disk_wal_fsync_duration_seconds.
• Compaction defrag. If a defrag is running on the member, the snapshot read competes with the bbolt rewrite.

gantt
    title Snapshot impact on commit latency
    dateFormat HH:mm
    axisFormat %H:%M
    section Snapshot
    snapshot save :a1, 00:05, 30s
    section commit latency
    WAL fsync p99 :crit, a2, 00:00, 60s

Plan the snapshot during a low-traffic window. The 30-60 seconds of I/O during the snapshot is the operational cost. Production clusters schedule it for the off-peak hours.

The kubeadm integration

Kubeadm does not include an out-of-the-box scheduler for snapshots; the operator adds it. A typical pattern:

1. Install a systemd timer on each control-plane node that runs the snapshot hourly.
2. Configure a sidecar process to upload snapshots to object storage.
3. Configure the off-cluster storage with retention policy (e.g. 30 days of hourly, 12 months of daily).

flowchart LR
    H1[Hourly snapshot cp-1] -->|/backup| S[local share]
    H2[Hourly snapshot cp-2] -->|/backup| S
    H3[Hourly snapshot cp-3] -->|/backup| S
    S -->|uploader cron| OBJ[object storage]

The off-cluster uploader reads the latest snapshot and copies it to object storage. It deletes the local snapshot after successful upload.

The snapshot file format

A snapshot file is a self-contained bbolt DB file:

$ file /backup/etcd-snapshot.db
/backup/etcd-snapshot.db: data

The file has the same internal layout as a live bbolt DB. Restoring creates a fresh data dir; the file is opened with bbolt.Open() and the data is loaded.

Verification after save

Always verify the snapshot after saving:

etcdutl snapshot status /backup/etcd-snapshot.db --write-out=table

+----------+----------+------------+----------------+
|   HASH   | REVISION |  TOTAL KEY |   TOTAL SIZE   |
+----------+----------+------------+----------------+
| a1b2c3d4 | 41289312 |       4123 |   82419000     |
+----------+----------+------------+----------------+

A successful snapshot has:

• A non-zero REVISION (the bbolt’s last applied revision).
• A non-zero TOTAL KEY count.
• A TOTAL SIZE consistent with the cluster’s bbolt DB size.

If HASH is empty or the size is zero, the snapshot is corrupt and must be retaken.

Quiz

Production discipline

• Snapshot to a follower when possible. Even if the follower rotates, cycle through them.
• Schedule during low traffic. Hourly at 04:00 is typically lower-traffic than 12:00.
• Always verify. etcdutl snapshot status after save.
• Off-cluster upload separately. The local snapshot is the source; the off-cluster copy is the disaster recovery. Different machines, different stores.
• Use the right TLS args. Client cert SAN must match the etcd client’s authentication policy.

Snapshot save is one command; the discipline around it is what makes it a backup primitive that survives the incident.