etcd flags, environment, and tuning the operator controls

etcd’s runtime configuration is in two places: the command-line flags to the etcd binary and the environment variables the operator sees. In a kubeadm cluster, both live in the static pod manifest at /etc/kubernetes/manifests/etcd.yaml on each control-plane node. This lesson walks the flags a production operator must know, the place each one lives, and the discipline of changing any of them.

Where the configuration lives

# On a control-plane node (kubeadm-style)
sudo cat /etc/kubernetes/manifests/etcd.yaml

The manifest is a static-pod manifest. The kubelet watches the file and restarts the pod when it changes. A change to the manifest triggers a member restart; the rolling restart of all three members is the operational impact.

flowchart LR
    MF["/etc/kubernetes/manifests/etcd.yaml"] -->|watch| K[kubelet]
    K -->|restart pod| EP[etcd static pod]
    EP -->|flags + env| E[etcd process]
    E -->|state| D["/var/lib/etcd"]

The critical implication: editing the manifest restarts the member. Restart is not free (WAL replay, snapshot install if the WAL is large); the cadence at which flags change should be measured in months, not minutes.

The static-pod manifest

The structure of etcd.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: etcd
  namespace: kube-system
spec:
  hostNetwork: true
  priorityClassName: system-node-critical
  containers:
  - name: etcd
    image: registry.k8s.io/etcd:3.5.x-0
    command:
    - etcd
    - --advertise-client-urls=https://10.0.1.10:2379
    - --cert-file=/etc/kubernetes/pki/etcd/server.crt
    - --client-cert-auth=true
    - --data-dir=/var/lib/etcd
    - --initial-advertise-peer-urls=https://10.0.1.10:2380
    - --initial-cluster=cp-1=https://10.0.1.10:2380,...
    - --key-file=/etc/kubernetes/pki/etcd/server.key
    - --listen-client-urls=https://127.0.0.1:2379,...
    - --listen-peer-urls=https://10.0.1.10:2380
    - --name=cp-1
    - --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt
    - --peer-client-cert-auth=true
    - --peer-key-file=/etc/kubernetes/pki/etcd/peer.key
    - --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
    - --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
    - --snapshot-count=10000
    - --election-timeout=1000
    - --heartbeat-interval=100
    - --quota-backend-bytes=2147483648
    - --auto-compaction-mode=periodic
    - --auto-compaction-retention=5m
    volumeMounts:
    - mountPath: /var/lib/etcd
      name: etcd-data
    - mountPath: /etc/kubernetes/pki/etcd
      name: etcd-certs
      readOnly: true

The flags every operator must read

Election and heartbeat — what to tune

The two tunables that affect availability:

• --election-timeout: the time a follower waits without a heartbeat before becoming a candidate. Default 1000 ms; widen for WAN (typical 2000-3000 ms).
• --heartbeat-interval: how often the leader pings followers. Default 100 ms.

The ratio must be at least 1:5 (heartbeat:election). The defaults in 3.5 are 100 ms / 1000 ms. Setting --election-timeout=500ms would risk frequent spurious elections under normal network variance.

# Check the configured values on a running etcd
etcdctl --endpoints=https://127.0.0.1:2379 ... endpoint status --write-out=json | jq '.[] | {Endpoint, "raft term": .Status.raftTerm}'

Quota — the safety net

--quota-backend-bytes (in bytes) sets a hard ceiling on the bbolt file size. When the cluster exceeds the quota, writes fail with “etcdserver: request is too large” or “etcdserver: mvcc: database space exceeded”. Reads continue.

A typical production value: 8 GiB (8589934592 bytes). The quota should be set slightly above the warning bound so that the cluster fails writes before the file grows past the size where defrag becomes unbearable.

Auto-compaction

- --auto-compaction-mode=periodic
- --auto-compaction-retention=5m

These two flags together enable automatic compaction every 5 minutes of historical revisions. The mode is periodic; the alternative is revision, which requires a target revision.

A common production pattern: --auto-compaction-retention=1m or 5m. The retention should be longer than the API server’s --watch-progress-notify-interval (default 5 seconds); the watcher can always recover from a compacted revision by re-listing.

The —initial-cluster flag — bootstrap only

The --initial-cluster flag is only used at bootstrap time. After a member has joined the cluster, its configuration is replaced by the cluster’s view of itself. Editing --initial-cluster on a running member does not “re-add” it; it can break the member’s bootstrap check or trigger an inadvertent new-cluster initialisation if the data directory is empty.

The kubeadm-managed wrapper

If the cluster was bootstrapped with kubeadm init, the flag set lives in two places:

1. The static-pod manifest at /etc/kubernetes/manifests/etcd.yaml.
2. The kubeadm-config ConfigMap at kube-system/kubeadm-config (ClusterConfiguration field, etcd section).

A flag change procedure:

1. Capture the current manifest: kubectl get pod -n kube-system etcd-cp-1 -o yaml > /backup/etcd-manifest.yaml.
2. Capture the current state of the cluster: etcdctl snapshot save /backup/etcd-snapshot-*.db.
3. Edit the manifest on the target member with the new flag(s).
4. Watch the kubelet restart the pod (crictl pods or kubectl get pods -n kube-system -w).
5. Verify the flag is in effect: etcdctl endpoint status --write-out=json | jq and journalctl -u kubelet -n 100 for the new flag in the process arguments.
6. Repeat for the next member, going leader-last.

$ crictl ps -a | grep etcd | head

CONTAINER   IMAGE                              CREATED          STATE    NAME    ATTEMPT    POD ID
<id>        registry.k8s.io/etcd:3.5.x-0    10 minutes ago   Running  etcd    0          <id>

Snapshot count — the per-N-entries setting

--snapshot-count defaults to 100,000. At a steady-state write rate of 50/sec, that’s one snapshot every 33 minutes. Higher values increase restart time but reduce snapshot I/O. Lower values accelerate catch-up but increase the frequency of background snapshot work.

In a busy cluster, drop --snapshot-count to 10,000-20,000 to bound restart time. In a quiet cluster, raise to keep disk I/O down.

Tuning decisions in production

The decision trees the operator runs through:

• fsync latency p99 > 25 ms: bad disk; don’t try to “tune” etcd. Replace the disk.
• leader changes per day > 5: network jitter or election timeout too tight. Widen --election-timeout from 1000 to 2000 ms (and confirm --heartbeat-interval is 100-200 ms).
• DB size growing steadily: lower --auto-compaction-retention to 1 m, then evaluate workload.
• Catch-up of new follower slow: lower --snapshot-count to bound the WAL replay.
• WAN deployment: widen --election-timeout to 2000-5000 ms; verify --heartbeat-interval is one-third of that.

Quiz

Production discipline

• Touch the etcd manifest rarely. A flag change is an incident-class operation, not a routine optimisation.
• Snapshot before any change. Whatever flag changes follow a snapshot that has been verified (etcdutl snapshot status).
• Roll through members; leader last. One member’s restart is a maintenance event on a healthy cluster; the cost is a momentary forfeit of quorum if the restart fails. Restart followers first.
• Edit the static-pod manifest for kubeadm, the kubeadm-config ConfigMap if you want kubeadm upgrade to keep it. Two places; change both or restore both.
• Verify via running process arguments. A --flag in the YAML is not in effect until the process has restarted with it.

etcd flags are the leverage the operator has for tuning performance; they are also the source of most etcd incidents. Change one at a time, in a window, with a snapshot, on one member.