etcdutl snapshot restore — the per-host commands

The etcdutl snapshot restore command takes a snapshot file and writes a fresh etcd data directory, ready to be started as a member. Each member runs the command independently with member-specific flags. This lesson walks the arguments, the per-host sequence, and the timing.

The command in one line

etcdutl snapshot restore /backup/etcd-snapshot.db \
  --data-dir=/var/lib/etcd-restore \
  --name=cp-1 \
  --initial-cluster=cp-1=https://10.0.1.10:2380,cp-2=https://10.0.1.11:2380,cp-3=https://10.0.1.12:2380 \
  --initial-advertise-peer-urls=https://10.0.1.10:2380

The arguments:

The membership — name = URL = host

--initial-cluster lists every member of the cluster at bootstrap. The names and URLs must match the kubeadm cluster’s existing membership.

cp-1=https://10.0.1.10:2380,cp-2=https://10.0.1.11:2380,cp-3=https://10.0.1.12:2380

Three rules:

1. Each URL is a peer URL (port 2380), not a client URL (2379).
2. https:// — the cluster uses TLS peer communication.
3. Each host’s name appears in the cluster exactly once with the URL that resolves to that host.

flowchart LR
    cp-1 -- https://10.0.1.10:2380 --> C[Initial cluster]
    cp-2 -- https://10.0.1.11:2380 --> C
    cp-3 -- https://10.0.1.12:2380 --> C

The per-host sequence

Each host runs the same etcdutl snapshot restore command with member-specific --name and --initial-advertise-peer-urls:

Host cp-1 (10.0.1.10)

etcdutl snapshot restore /backup/etcd-snapshot.db \
  --data-dir=/var/lib/etcd-restore \
  --name=cp-1 \
  --initial-cluster=cp-1=https://10.0.1.10:2380,cp-2=https://10.0.1.11:2380,cp-3=https://10.0.1.12:2380 \
  --initial-advertise-peer-urls=https://10.0.1.10:2380

The command logs two lines, restoring snapshot and then restored snapshot, each carrying the snapshot path and the data-dir, wal-dir and snap-dir it wrote. If the second line does not appear, nothing usable was written.

Host cp-2 (10.0.1.11)

etcdutl snapshot restore /backup/etcd-snapshot.db \
  --data-dir=/var/lib/etcd-restore \
  --name=cp-2 \
  --initial-cluster=cp-1=https://10.0.1.10:2380,cp-2=https://10.0.1.11:2380,cp-3=https://10.0.1.12:2380 \
  --initial-advertise-peer-urls=https://10.0.1.11:2380

Host cp-3 (10.0.1.12)

etcdutl snapshot restore /backup/etcd-snapshot.db \
  --data-dir=/var/lib/etcd-restore \
  --name=cp-3 \
  --initial-cluster=cp-1=https://10.0.1.10:2380,cp-2=https://10.0.1.11:2380,cp-3=https://10.0.1.12:2380 \
  --initial-advertise-peer-urls=https://10.0.1.12:2380

All three commands write to a local data dir at /var/lib/etcd-restore.

The data-dir move

After the restore on each host, move the restored data into the path the kubelet expects:

# On each host:
sudo mv /var/lib/etcd-restore /var/lib/etcd

The kubelet’s static pod manifest references /var/lib/etcd (the standard kubeadm path). The restored data lands in the right place.

flowchart TB
    A["snapshot file"] --> B["/var/lib/etcd-restore"]
    B -->|mv| C["/var/lib/etcd"]
    C -->|"static pod mounts"| D["etcd reads bbolt"]

The alternative is to leave the restored data where it is and repoint the etcd-data hostPath in /etc/kubernetes/manifests/etcd.yaml at /var/lib/etcd-restore. kubeadm mounts the host data directory into the container at the same path and passes that path as --data-dir, so changing the hostPath alone is enough. The kubernetes-rb-restore-etcd runbook takes that route because it keeps the restored and pre-restore directories visibly distinct on disk. Both reach the same state; use one of them consistently on all three hosts.

The timing

The restore writes a bbolt database. The duration scales with the snapshot size:

The total time for the per-host restores is bounded by the slowest host; if the operator runs them in parallel, the total is the slowest host’s restore time.

gantt
    title Restore timing (3 hosts, parallel)
    dateFormat HH:mm
    axisFormat %H:%M
    section Per-host
    Host cp-1 :a1, 00:00, 3m
    Host cp-2 :a2, 00:00, 3m
    Host cp-3 :a3, 00:00, 3m
    section Total
    Restore complete :crit, after a3, 0m

Production restore for a 1.5 GB snapshot completes in ~5 minutes per host; ~7 minutes total for the three hosts running in parallel.

The arguments explained

--data-dir

The directory the restore writes the bbolt DB file into. On kubeadm this is /var/lib/etcd. The operator may choose to restore into a different path (/var/lib/etcd-restore) and move later for safety.

--name

This member’s name. The name must match the kubeadm configuration. Common convention is the hostname, but it does not have to be.

--initial-cluster

A comma-separated list of NAME=URL pairs for every member of the cluster at bootstrap time. The URL is the peer’s URL (port 2380, https). All three members are listed.

--initial-advertise-peer-urls

The URL this member advertises to peers as its own address. This must be reachable from the other members.

--initial-cluster-state

There is no --initial-cluster-state on the restore command, and that is the point worth understanding. etcdutl snapshot restore writes the membership into the data directory it produces. etcd reads its bootstrap flags — --initial-cluster, --initial-advertise-peer-urls, --initial-cluster-state, --initial-cluster-token — only when the data directory is empty. After a restore the directory is not empty, so whatever those flags say in /etc/kubernetes/manifests/etcd.yaml is ignored.

That is why the restore command must carry --initial-cluster and --initial-advertise-peer-urls itself. A member restored without them comes up as a single-member cluster, and no amount of editing the static pod manifest afterwards will correct it.

The two flags a Kubernetes restore should not omit

A restore normally moves the revision backwards, because the snapshot only holds the history up to the moment it was taken. The etcd documentation calls Kubernetes out by name here: controllers and operators run informers that cache state and follow watches, and they do not necessarily refresh when the revision regresses, which produces inconsistent controller behaviour against a cluster that otherwise looks healthy.

--bump-revision adds a fixed number to the snapshot’s revision so it never goes backwards, and --mark-compacted marks the resulting revision as the scheduled compaction point, which terminates every outstanding watch and invalidates the caches behind them. --mark-compacted is required whenever --bump-revision is greater than zero and rejected otherwise, so the two always travel together.

etcdutl snapshot restore /backup/etcd-snapshot.db \
  --data-dir=/var/lib/etcd-restore \
  --name=cp-1 \
  --initial-cluster=cp-1=https://10.0.1.10:2380,cp-2=https://10.0.1.11:2380,cp-3=https://10.0.1.12:2380 \
  --initial-advertise-peer-urls=https://10.0.1.10:2380 \
  --initial-cluster-token=k8s-restore-20260818 \
  --bump-revision=1000000000 \
  --mark-compacted

Size the bump above the writes the cluster could have taken since the snapshot. The etcd documentation’s own figure is 1,000,000,000, which covers a week-old snapshot on a cluster running under 1,500 writes per second.

Common mistakes

The snapshot file location

A practical question: where is the snapshot file during the restore?

• The snapshot is on a network-mounted file system (NFS, cluster-wide mount), and each host reads it.
• The snapshot is on a USB drive shared across hosts.
• The snapshot is at a different path on each host (the file is copied to each host first).

For kubeadm-managed clusters, the snapshot is usually on /backup/ of each host. The operator copies the snapshot to each host, or sets up an NFS mount that all hosts share.

# On each host:
ls -la /backup/etcd-snapshot.db
# Expected: file exists

Quiz

Production discipline

• Run the restore on every host. A partial restore leaves the cluster with fewer members than expected.
• All snapshots come from the same source. The same snapshot file; the three restored data dirs will be identical at the start.
• Member-specific flags. --name, --initial-advertise-peer-urls vary by host; the --initial-cluster is the same.
• Move the restored data, do not delete. Forensic state matters.
• Validate timing. Time the restore on each host; investigate if a host takes 10x longer than the others (likely a slow disk).

The restore is offline, independent per host, and sequentially safe. The per-host command is the same; the specifics vary by member.