Secret backup — the encrypted credentials

The secret backup is the encrypted credentials. The encryption at rest, the secret management, the backup encrypted are the components. This lesson walks the secret backup, the encryption at rest, the secret management, and the production patterns.

The encryption at rest

The encryption at rest:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
  - secrets
  providers:
  - aescbc:
      keys:
      - name: key1
        secret: <base64-encoded-key>
  - secretbox:
      keys:
      - name: key2
        secret: <base64-encoded-key>
  - identity: {}

The encryption at rest is the data protection.

The key management

The key management:

# Substitute your own value before running:
KMS_KEY_ID=1234abcd-12ab-34cd-56ef-1234567890ab

# Generate the encryption key
ENC_KEY=$(head -c 32 /dev/urandom | base64)

# Store the key in KMS
aws kms encrypt --key-id "$KMS_KEY_ID" --plaintext "$ENC_KEY"

The key management is the KMS integration.

The secret management tools

The secret management tools:

flowchart LR
    A[Secrets] --> B[Vault]
    A --> C[AWS Secrets Manager]
    A --> D[External Secrets Operator]
    B --> E[Cluster]
    C --> E
    D --> E

The secret management tools are the canonical.

The External Secrets Operator

The External Secrets Operator:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-backend
  namespace: default
spec:
  provider:
    vault:
      server: https://vault.example.com
      path: secret
      version: v2
      auth:
        kubernetes:
          mountPath: kubernetes
          role: my-app

The External Secrets Operator is the integration.

The SOPS

The SOPS:

# Substitute your own value before running:
KMS_KEY_ARN=arn:aws:kms:us-east-1:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab

# Encrypt the file
sops --encrypt --kms "$KMS_KEY_ARN" secrets.yaml > secrets.encrypted.yaml

# Decrypt the file
sops --decrypt secrets.encrypted.yaml > secrets.yaml

# Edit the file
sops secrets.encrypted.yaml

The SOPS is the file-level encryption.

The secret backup

The secret backup:

# Substitute your own value before running:
KMS_KEY_ID=1234abcd-12ab-34cd-56ef-1234567890ab

# Backup the secrets encrypted
kubectl get secrets -A -o yaml > /var/backups/secrets-$(date +%Y%m%d).yaml

# Encrypt the backup
gpg --symmetric --cipher-algo AES256 /var/backups/secrets-$(date +%Y%m%d).yaml

# Upload to S3
aws s3 cp /var/backups/secrets-$(date +%Y%m%d).yaml.gpg s3://k8s-backups/secrets/ \
  --sse aws:kms --sse-kms-key-id "$KMS_KEY_ID"

The secret backup is the encrypted backup.

The secret rotation

The secret rotation:

# Add the new key at the top of the provider's key list in the
# EncryptionConfiguration that --encryption-provider-config
# points at, keeping the old key below it. The first key
# encrypts; the rest only decrypt.
sudo "$EDITOR" /etc/kubernetes/enc/encryption-config.yaml

# Make the API server read the file again. On a kubeadm
# cluster there is no kube-apiserver systemd unit, so move the
# static Pod manifest out of the kubelet's watch directory and
# back; the kubelet stops the Pod when the file goes and
# starts a fresh one when it returns.
sudo mv /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/
# wait for the kubelet's fileCheckFrequency, 20s by default
sudo mv /tmp/kube-apiserver.yaml /etc/kubernetes/manifests/

# Re-encrypt what is already stored, by rewriting every Secret
kubectl get secrets -A -o json | kubectl replace -f -

The secret rotation is the operational discipline. Existing Secrets stay on the old key until something rewrites them, which is why the re-encryption step is not optional. Only once every Secret has been rewritten can the old key be removed from the configuration.

The backup encryption

The backup encryption:

# Substitute your own values before running:
KMS_KEY_ID=1234abcd-12ab-34cd-56ef-1234567890ab
KMS_KEY_ARN=arn:aws:kms:us-east-1:111122223333:key/$KMS_KEY_ID

# Encrypt the backup with KMS
aws s3 cp /var/backups/secrets-$(date +%Y%m%d).yaml s3://k8s-backups/secrets/ \
  --sse aws:kms --sse-kms-key-id "$KMS_KEY_ID"

# Or use the SOPS
sops --encrypt --kms "$KMS_KEY_ARN" /var/backups/secrets-$(date +%Y%m%d).yaml > /var/backups/secrets-$(date +%Y%m%d).enc.yaml

The backup encryption is the data protection.

The production patterns

The production patterns:

flowchart LR
    A[Secrets] --> B[Encryption at rest]
    B --> C[KMS]
    C --> D[Backup encrypted]
    D --> E[S3]
    E --> F[External Secrets Operator]
    F --> G[Cluster]

The pattern is the production discipline.

The cross-course references

• The Security course covers the encryption.
• The Vault course covers the secret management.
• The External Secrets Operator course covers the integration.

Quiz

Production discipline

• Configure the encryption at rest. The EncryptionConfiguration.
• Use the secret management tools. Vault, External Secrets Operator.
• Use the SOPS. The file-level encryption.
• Encrypt the backup. The KMS encryption.
• Rotate the keys regularly. The security best practice.
• Document the secret backup. The configuration, the rotation.

The secret backup is the encrypted credentials. Operating it well is the encryption at rest, the secret management tools, the backup encryption, and the production patterns.