Etcd encryption — Secrets at rest

By default, Kubernetes stores Secrets in etcd in plaintext (base64-encoded, but not encrypted). An attacker with access to etcd — via a compromised control plane node, a misconfigured etcd backup, or a stolen snapshot — can read every Secret. The EncryptionConfiguration API enables encryption at rest; the right provider (aescbc, KMS, secretbox) depends on the operational context. This lesson covers the configuration, the providers, the key rotation, and the failure modes.

The threat model

etcd is the source of truth for the cluster. Every object — Pods, Services, Secrets — is stored in etcd. An attacker with etcd access has the cluster’s data:

• etcd backup. An etcd snapshot is a copy of the cluster’s state. A stolen snapshot exposes every Secret.
• etcd member compromise. A compromised control plane node has etcd access. The attacker reads Secrets directly.
• Misconfigured etcd. An etcd listener exposed to the cluster network is reachable from any Pod.

EncryptionConfiguration protects against backup theft and direct etcd access by encrypting the data before it is written to etcd.

The providers

Four providers are supported:

kms is the enterprise standard; the key is held in a KMS (AWS KMS, GCP KMS, Azure Key Vault), and etcd data is encrypted with a data encryption key (DEK) that is wrapped by the KMS key (KEK).

flowchart LR
    A[Plaintext Secret] --> B[DEK encrypts]
    B --> C[Ciphertext]
    C --> D[etcd]
    E[KEK in KMS] --> F[Wraps DEK]
    F --> G[API server]
    G --> B

The DEK is rotated automatically by the KMS plugin; the KEK is rotated manually.

The configuration

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
    - secrets
    providers:
    - kms:
        name: my-kms-plugin
        endpoint: unix:///var/run/kmsplugin.sock
        cachesize: 1000
        timeout: 3s
    - identity: {}  # fallback for reads

The API server reads the first provider for writes (the KMS) and tries all providers for reads (KMS, then identity fallback). The fallback to identity allows existing unencrypted Secrets to be read during migration.

Enabling the configuration

The configuration is mounted into the kube-apiserver Pod:

spec:
  containers:
  - name: kube-apiserver
    command:
    - kube-apiserver
    - --encryption-provider-config=/etc/encryption/config.yaml
    volumeMounts:
    - name: encryption-config
      mountPath: /etc/encryption
      readOnly: true
  volumes:
  - name: encryption-config
    hostPath:
      path: /etc/kubernetes/encryption
      type: DirectoryOrCreate

The configuration is read on startup and reloaded on SIGHUP.

Key rotation

The EncryptionConfiguration supports key rotation via the keys: list:

providers:
- aescbc:
    keys:
    - name: key2
      secret: <base64-encoded new key>
    - name: key1
      secret: <base64-encoded old key>

The first key in the list is the write key; all keys are tried for reads (the API server tries each until decryption succeeds). To rotate:

1. Generate a new key. head -c 32 /dev/urandom | base64.
2. Add the new key as the first in the list. Writes use the new key; reads try both.
3. Re-encrypt all Secrets. kubectl get secrets -A -o json | kubectl replace -f - (forces a rewrite with the new key).
4. Remove the old key from the list. All Secrets are now encrypted with the new key.

sequenceDiagram
    participant Op as Operator
    participant AS as API server
    participant E as Etcd
    Op->>AS: Add key2 as first
    AS->>AS: Writes use key2, reads try both
    Op->>AS: Re-encrypt all Secrets
    AS->>E: Writes with key2
    Op->>AS: Remove key1
    AS->>AS: Writes use key2, reads try key2

Production patterns

1. Use KMS for production. The KMS provider delegates key management to a cloud service; the key is never on the cluster nodes.
2. Rotate keys quarterly. The rotation procedure is documented; the cluster has a deadline per quarter.
3. Encrypt secrets and configmaps. Any resource that holds sensitive data should be encrypted.
4. Audit the encryption status. etcdctl get on a Secret should return ciphertext, not plaintext.

Production failure modes

1. No encryption. The cluster stores Secrets in plaintext in etcd. The fix is to enable EncryptionConfiguration.
2. KMS plugin is down. The API server cannot encrypt new Secrets; reads of old Secrets may fail. The fix is HA KMS and a kms provider with retries.
3. Lost key. All Secrets encrypted with the lost key are unreadable. The fix is multi-key configurations and key backups.
4. identity provider is not in the list. Old Secrets are unreadable during migration. The fix is to add identity as the last provider.

Cross-course references

• The Linux course covers AES-CBC, secretbox, and envelope encryption.
• The Observability course covers the audit log entries for Secret encryption events.

Quiz

Production discipline

Encryption at rest is the foundation of Secret hygiene for etcd. A defensible encryption programme uses kms for production, rotates keys quarterly, audits the encryption status with etcdctl, and includes identity as a fallback for migration. A cluster whose Secrets are encrypted at rest has a Secret programme that is auditable; a cluster whose Secrets are plaintext in etcd has a programme that is not.