Encryption at rest — EncryptionConfiguration, AES-GCM, and key rotation

Without encryption at rest, Kubernetes Secrets are stored plaintext in etcd. Anyone with file-system or network access to etcd can read them. Encryption at rest is configured on the API server with an EncryptionConfiguration file; it encrypts the Secret values before the API server writes to etcd and decrypts them on read. This lesson covers the configuration, the supported providers, and the discipline of key rotation.

What encryption at rest protects

flowchart LR
    A[Pod reads Secret] --> B[API server reads from etcd]
    B --> C{Value encrypted?}
    C -->|yes| D[Decrypt with provider]
    C -->|no| E[Plaintext]
    D --> F[Return plaintext to Pod]
    E --> F
    F --> G[Application reads]

The API server encrypts before writing; etcd stores the encrypted value; on read, the API server decrypts. Anyone with direct etcd access sees ciphertext.

What encryption at rest does not protect:

• Memory dumps. A privileged process on the API server host can read the decrypted value from memory.
• API server logs. A misconfigured log statement that prints Secret values will print the decrypted value.
• Container reads. A container with a mounted Secret sees the plaintext; the encryption does not extend to the file system.
• The API server’s view. The encryption is at the etcd-storage layer; the API server itself sees plaintext.

Encryption at rest is one layer; not all layers.

The EncryptionConfiguration file

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
  - secrets
  providers:
  - aescbc:
      keys:
      - name: key1
        secret: <base64 of 32 bytes>
  - identity: {}

The file is passed to the API server as --encryption-provider-config. The API server encrypts every Secret write with the first provider in the list; on read, it tries each provider in order.

flowchart LR
    A[Write Secret] --> B["Provider 1: aescbc"]
    B --> C["etcd<br/>stores ciphertext"]
    D[Read Secret] --> E[Try provider 1]
    E --> F{Decrypt ok?}
    F -->|yes| G[Plaintext]
    F -->|no| H[Try provider 2]
    H --> I[...etc]

Providers

identity

providers:
- identity: {}

No encryption. The default if no provider is configured. Useful as the last entry in a list (fallback for resources not in the encrypted list).

aescbc

providers:
- aescbc:
    keys:
    - name: key1
      secret: <base64 of 32 bytes>

AES-CBC with PKCS#7 padding. The key is 32 bytes (AES-256). The secret is the base64-encoded key. Generate with:

head -c 32 /dev/urandom | base64

aesgcm

providers:
- aesgcm:
    keys:
    - name: key1
      secret: <base64 of 32 bytes>

AES-GCM (authenticated encryption). Faster than AES-CBC on modern CPUs (AES-NI). The key is 32 bytes.

secretbox

providers:
- secretbox:
    keys:
    - name: key1
      secret: <base64 of 32 bytes>

XSalsa20-Poly1305. Not hardware-accelerated; slower than AES-GCM. Used when AES is unavailable.

KMS providers

providers:
- kms:
    name: myKmsPlugin
    endpoint: unix:///var/run/kms-provider.socket
    cachesize: 100
    timeout: 3s

A KMS provider calls an external key-management service (AWS KMS, GCP KMS, Azure Key Vault, HashiCorp Vault). The provider returns a data encryption key (DEK); the API server encrypts the Secret with the DEK; the DEK is encrypted with the KEK (key encryption key) from the KMS.

flowchart TB
    A[API server] -->|request DEK| B[KMS provider]
    B -->|generate DEK| C["KMS service<br/>e.g. AWS KMS"]
    C -->|DEK encrypted with KEK| B
    B -->|return encrypted DEK| A
    A -->|encrypt Secret with DEK| D[Plaintext DEK]
    D -->|wrap with KEK| E[Wrapped DEK]
    E --> F["etcd: ciphertext + wrapped DEK"]

The KMS provider is the strongest choice: the encryption key never leaves the KMS. An attacker with etcd access sees ciphertext; even with API server memory access, the KEK is in the KMS.

Configuration steps

1. Generate the key:

head -c 32 /dev/urandom | base64
# kJHJq...base64...

2. Create the EncryptionConfiguration:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
  - secrets
  providers:
  - aesgcm:
      keys:
      - name: key1
        secret: kJHJq...
  - identity: {}

3. Mount the configuration into the API server pods:

cp encryption-config.yaml /etc/kubernetes/encryption-config.yaml
chmod 600 /etc/kubernetes/encryption-config.yaml
chown root:root /etc/kubernetes/encryption-config.yaml

4. Configure the API server flag:

# kubeadm config
apiServer:
  extraArgs:
    encryption-provider-config: /etc/kubernetes/encryption-config.yaml

5. Restart the API server:

kubectl -n kube-system rollout restart deployment/kube-apiserver

6. Encrypt all existing Secrets:

kubectl get secrets -A -o json | \
  kubectl replace -f -

This re-writes every Secret; the API server encrypts each on the new write.

Key rotation

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
  - secrets
  providers:
  - aesgcm:
      keys:
      - name: key2
        secret: <new key>
      - name: key1
        secret: <old key>
  - identity: {}

Rotation steps:

flowchart TB
    A[Generate new key] --> B["Add new key as<br/>primary in config"]
    B --> C["API server reads with new key<br/>writes with new key"]
    C --> D[Re-encrypt all Secrets]
    D --> E[Old Secrets re-encrypted]
    E --> F["Remove old key<br/>after verification"]
    F --> G[Rotation complete]

1. Generate a new key (key2).
2. Add key2 as the first key; keep key1 as the second.
3. Restart the API server.
4. Re-write every Secret (kubectl get secrets -A -o json | kubectl replace -f -).
5. Verify all Secrets are encrypted with key2.
6. Remove key1 from the config.

The re-write step is critical. Without it, existing Secrets remain encrypted with key1; if key1 is removed, those Secrets become unreadable.

Verifying encryption

# Check that etcd has encrypted values
ETCDCTL_API=3 etcdctl get /registry/secrets/prod/db-credentials \
  --endpoints=https://etcd-1:2379 \
  --cacert=... --cert=... --key=... \
  | hexdump -C | head

Plaintext would be readable JSON; ciphertext is unrecognisable. Verify the field is encrypted.

# Verify the API server can still read the Secret
kubectl get secret db-credentials -n prod -o yaml
# the data is base64-encoded (always), but the API server
# has decrypted it for this response

Quiz

Production discipline

• Encryption at rest is opt-in. Default Secrets are plaintext in etcd. Configure before storing credentials.
• Use a KMS provider for production. AES-GCM is fine for development; production credentials belong behind a KMS.
• Rotate keys regularly. A 90-day rotation cadence is the standard; document the rotation runbook.
• Verify after configuration. The test is etcdctl get on a Secret’s storage path; ciphertext is unreadable.
• Back up the keys. Lost keys = unrecoverable Secrets. KMS handles this; AES keys must be backed up separately.
• Test the recovery path. A kubectl get secret that fails after a key is removed is unrecoverable. The rotation runbook must include a rollback step.

Encryption at rest is a fundamental Secret-hygiene practice. The discipline is in the configuration, the provider choice, and the rotation cadence.