Encryption at rest for snapshots — what to encrypt and how

An etcd snapshot is the cluster’s API state. Without encryption at rest, that includes Secrets in plaintext. The snapshot then sits on object storage indefinitely, visible to anyone with read access. The encryption discipline must apply at every layer: the API server, the object store, and the cross-region replication. This lesson walks each layer and the key-management discipline.

What an unencrypted snapshot contains

flowchart LR
    S["Snapshot file (no encryption)"] -->|Secrets| P1[db-credentials password]
    S -->|Tokens| P2[service account token]
    S -->|Config| P3[ConfigMap with API key]
    S -->|RBAC| P4[ServiceAccount access]
    S -->|Business| P5[Deployment env vars]

• Secrets (in plaintext — base64 is encoding, not encryption).
• Service Account tokens (the JWT stored with the SA, used by admission and API requests).
• ConfigMap data (sometimes contains API keys, passwords, or other secrets that should not be there).
• All API objects (which include metadata, status, and annotations that may be sensitive).

The snapshot is an entire API server’s view of the cluster’s state. Read access to the snapshot file is equivalent to read access to every Secret in the cluster.

The three encryption layers

flowchart TB
    Layer1[Layer 1: API server encryption at rest]
    Layer2[Layer 2: Server-side encryption in object storage]
    Layer3[Layer 3: Client-side encryption before upload]
    Layer1 -->|encrypts before| E1[etcd writes ciphertext]
    E1 -->|snapshot is ciphertext| E2[snapshot file]
    Layer2 -->|upload| OS[object store, encrypted at rest]
    Layer3 -->|encrypt the file before upload| OS

• Layer 1: API server encryption at rest. This is the most important layer. It encrypts Secret values before the API server writes to etcd; the snapshot reads from etcd and contains ciphertext. Without this layer, no downstream encryption protects a Snapshot-aware restore.
• Layer 2: Object storage SSE. S3 with SSE-KMS or SSE-S3 encrypts the snapshot file at rest in the bucket. Necessary for compliance, but it does not protect the snapshot once it is downloaded.
• Layer 3: Client-side encryption before upload. The snapshot file is encrypted with a separate key before upload. The strongest layer; it protects the snapshot even if object storage is fully compromised.

Production clusters typically run layers 1, 2, and 3. The combination is defence in depth.

Layer 1 — EncryptionConfiguration

The Kubernetes API server supports an EncryptionConfiguration file that controls encryption at rest for resources, including Secrets:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
  - secrets
  providers:
  - kms:
      name: myKmsPlugin
      endpoint: unix:///var/run/kms-provider.sock
      cachesize: 100
      timeout: 3s
  - identity: {}

The API server encrypts every Secret write through the first provider in the list. On read, it tries each provider in order. With this configuration, etcd stores ciphertext, and the snapshot reads from etcd and contains ciphertext.

# Verify the encryption is in effect
ETCDCTL_API=3 etcdctl get /registry/secrets/prod/db-credentials \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=... --cert=... --key=...
# Expected: ciphertext, not JSON with base64

The supported providers:

• identity — no encryption (default).
• aescbc / aesgcm — AES symmetric with key file.
• secretbox — XSalsa20-Poly1305.
• kms — external key management service (AWS KMS, GCP KMS, Azure Key Vault, HashiCorp Vault).

$ etcdctl snapshot save /tmp/encrypted-test.db

"saved snapshot to /tmp/encrypted-test.db"

Layer 2 — Object storage SSE

S3 SSE (server-side encryption) encrypts the snapshot file at rest in the bucket:

# AWS SSE-S3
aws s3 cp /backup/etcd-snapshot.db \
  s3://prod-etcd-backups/etcd/snapshot.db \
  --sse AES256

# AWS SSE-KMS (more control; CMK rotation)
aws s3 cp /backup/etcd-snapshot.db \
  s3://prod-etcd-backups/etcd/snapshot.db \
  --sse aws:kms \
  --sse-kms-key-id arn:aws:kms:us-east-1:111122223333:key/...

SSE-KMS provides:

• Customer-managed key (CMK) rotation.
• Audit logging of every key access.
• Role-based access (a principal without kms:Decrypt cannot read the snapshot).

The KMS key policy is the second layer of access control.

Layer 3 — Client-side encryption

The strongest layer. The snapshot is encrypted locally before upload:

# Generate a strong key
head -c 32 /dev/urandom | base64 > /etc/etcd-backup.key

# Encrypt the snapshot
gpg --batch --yes --recipient-file /etc/etcd-backup.pub \
    --output /backup/etcd-snapshot.db.gpg \
    --encrypt /backup/etcd-snapshot.db

# Upload the encrypted file
aws s3 cp /backup/etcd-snapshot.db.gpg \
  s3://prod-etcd-backups/etcd/snapshot.db.gpg

The encryption can also use age, sops, or a KMS-issued DEK:

# Using age
age -r age1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx \
    -o /backup/etcd-snapshot.db.age \
    /backup/etcd-snapshot.db

# Upload
aws s3 cp /backup/etcd-snapshot.db.age \
  s3://prod-etcd-backups/etcd/snapshot.db.age

Client-side encryption is the only protection against a fully compromised object store. Layer 2 protects only against storage server compromise; layer 3 protects against everything.

Key management

flowchart LR
    KMS[KMS key] -->|API server encryption| API[Layer 1]
    KMS -->|DEK envelope| OS[Layer 2 SSE-KMS]
    CSEK[Client-side encryption key] -->|local encryption| CSE[Layer 3]
    CSEK -->|stored in| VAULT[External secrets vault]

Three key domains:

• KMS key (layer 1). External (AWS KMS / Vault). Encrypted Secret values are only readable while the API server has access to the KMS endpoint.
• KMS bucket key (layer 2). The S3 SSE-KMS key. Decrypt requires the IAM role’s KMS permissions.
• Client-side key (layer 3). The local key for client-side encryption. Stored in a separate vault (HashiCorp Vault, AWS Secrets Manager, sealed in a sealed-secrets-style encrypted file).

The keys must be backed up separately. Loss of any one key renders the corresponding layer’s snapshots unreadable.

Rotation considerations

Layer 1 rotation typically requires the API server to re-write all Secrets — the same procedure as enabling encryption at rest in the first place (cluster-wide Secret re-write).

The “encrypt nothing” anti-pattern

A common anti-pattern in small clusters:

# EncryptionConfiguration with only identity:
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
  - secrets
  providers:
  - identity: {}

The identity provider does nothing. The cluster’s Secrets are plaintext in etcd. The snapshot contains plaintext Secrets. Uploading to S3 with SSE-KMS protects the file at rest, but anyone with KMS decrypt access (or, worse, anyone with object storage listing access before SSE was configured) sees plaintext.

The fix is to enable an actual provider (aescbc as a starting point; KMS as the production target).

The configuration chain

flowchart LR
    A["Operator edit EncryptionConfiguration"] -->|apply| B["API server"]
    B -->|restart| C["API server with new config"]
    C -->|"re-write Secrets"| D["kubectl get secrets -A -o yaml | kubectl replace -f -"]
    D -->|read by etcd| E["etcd with ciphertext"]
    E -->|snapshot| F["ciphertext snapshot"]
    F -->|upload| G["SSE-KMS encrypted at rest"]

Three steps: enable encryption in the API server, re-write all Secrets, verify that snapshots contain ciphertext.

Snapshot verification — proving encryption works

The fastest test:

# Take a snapshot, then read a known Secret's storage path
etcdctl snapshot save /tmp/test.db
# Now read the snapshot's bbolt content directly via boltviewer or strings
strings /tmp/test.db | grep -i password
# Expected: no plaintext password

With encryption at rest, the bbolt file contains opaque ciphertext. A strings grep for a known Secret name returns nothing recognisable.

Quiz

Production discipline

• Layer 1 is non-negotiable. Configure encryption at rest on the API server before storing production credentials.
• Layer 2 is required by most compliance regimes. SSE-KMS with a CMK is the standard.
• Layer 3 is defence in depth. Client-side encryption protects against every other failure.
• Key management is its own discipline. Three key domains; rotation cadences; backup of keys; restoration testing.
• Verify at every layer. A snapshot whose strings output reveals a Secret value is a snapshot that was not encrypted at the API server.

Encryption is the operational discipline that turns backups into something safe to leave in object storage indefinitely.