Secret best practices — the operational discipline

A defensible Kubernetes Secret programme layers controls: encryption at rest, scoped RBAC, external stores for high-value credentials, safe mounting, and rotation. This lesson walks the layers, the rotation procedures, the audit workflow, and the production failure modes.

The layered controls

Five layers, each with a different role:

flowchart LR
    L1[Layer 1: Encryption at rest] --> L2[Layer 2: Scoped RBAC]
    L2 --> L3[Layer 3: External stores]
    L3 --> L4[Layer 4: Safe mounting]
    L4 --> L5[Layer 5: Rotation]

Each layer assumes the layer below has failed.

Layer 1: encryption at rest

EncryptionConfiguration with a KMS provider is the right production default:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
  - secrets
  providers:
  - kms:
      name: my-kms-plugin
      endpoint: unix:///var/run/kmsplugin.sock

The KMS provider encrypts Secrets with a data encryption key (DEK) wrapped by the KMS KEK. The KEK is rotated quarterly.

Layer 2: scoped RBAC

Every Secret Role uses resourceNames:

rules:
- apiGroups: [""]
  resources: ["secrets"]
  resourceNames: ["db-credentials"]  # only this Secret
  verbs: ["get"]

list is denied for workloads. ClusterRoleBindings to Secrets are forbidden (namespace-scoped only).

Layer 3: external stores

High-value credentials live in Vault or a cloud KMS; ESO mirrors them into Kubernetes Secrets:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-store
  namespace: prod
spec:
  provider:
    vault:
      server: "https://vault.example.com"
      auth:
        kubernetes:
          role: "prod-api"
          serviceAccountRef:
            name: "api-sa"
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
  namespace: prod
spec:
  refreshInterval: 5m
  secretStoreRef:
    name: vault-store
    kind: SecretStore
  target:
    name: db-credentials
    creationPolicy: Owner
  data:
  - secretKey: password
    remoteRef:
      key: prod/db
      property: password

The Kubernetes Secret is a mirror; the external store is the source of truth.

Layer 4: safe mounting

spec:
  securityContext:
    runAsUser: 1000
    runAsNonRoot: true
  containers:
  - name: api
    image: myapp:v1.0
    volumeMounts:
    - name: db-credentials
      mountPath: /etc/secrets/db
      readOnly: true
  volumes:
  - name: db-credentials
    secret:
      secretName: db-credentials
      defaultMode: 0400

The Secret is a volume mount, not an env var. The file is owned by UID 1000; only UID 1000 can read it.

Layer 5: rotation

Rotation procedures vary by Secret:

The rotation is automated where possible; manual where not. The procedure is documented in the runbook.

The audit workflow

A quarterly audit walks the layers:

# 1. Encryption at rest
kubectl get pods -n kube-system -l component=kube-apiserver \
  -o jsonpath='{.items[*].spec.containers[*].args}' | \
  grep encryption-provider-config
# Should show --encryption-provider-config=<path>

# 2. Scoped RBAC
kubectl get roles,clusterroles -A -o json | \
  jq '.items[] | select(.rules[]?.resources[]? == "secrets") |
  select(.rules[]?.resourceNames == null) |
  {name: .metadata.name}'

# 3. External stores
kubectl get externalsecrets -A | head

# 4. Safe mounting
for pod in $(kubectl get pods -A -o name); do
  kubectl get "$pod" -o json | \
    jq '.spec.containers[].env[]? | select(.valueFrom.secretKeyRef) |
    "ENV VAR SECRET: " + .valueFrom.secretKeyRef.name'
done

# 5. Rotation
kubectl get externalsecrets -A -o jsonpath='{.items[*].spec.refreshInterval}'

A finding in any layer is a Critical or High remediation.

Production failure modes

1. Encryption at rest is not enabled. Secrets are plaintext in etcd. The fix is EncryptionConfiguration with KMS.
2. Broad Secrets RBAC. A ClusterRoleBinding grants secrets: get, list, watch to a SA. The fix is namespace-scoped RBAC with resourceNames.
3. High-value credentials in Git. A kubernetes.io/basic-auth Secret is committed to a public repo. The fix is external stores or sealed-secrets.
4. No rotation procedure. A credential is rotated manually; the procedure is not documented. The fix is a runbook entry.

Cross-course references

• The Linux course covers file permissions, KMS, and Vault.
• The Observability course covers the audit log entries for Secret access.

Quiz

Production discipline

A defensible Secret programme layers controls: encryption at rest with KMS, scoped RBAC with resourceNames, external stores for high-value credentials, safe mounting with 0400 and non-root, and rotation procedures documented in the runbook. The audit runs quarterly and reports findings per layer. A cluster whose Secrets pass all five layers has a Secret programme that is auditable; a cluster whose Secrets fail any layer has a programme that is not. The discipline is to ship all five layers, audit them quarterly, and remediate findings immediately.