etcd storage — where Secrets live and who can read them

A Kubernetes Secret’s lifecycle runs through three storage layers: the API server (which accepts and serves the Secret), etcd (which persists it), and the file system on the node (where mounted Secrets are projected). Each layer has its own access control; a Secret is “secure” only if every layer is locked down. This lesson walks the chain and the protections at each step.

The storage path

Secrets are stored in etcd at the key:

/registry/secrets/<namespace>/<name>

The value is the JSON-serialised Secret object, including data (base64) and metadata. With encryption at rest configured, the value is encrypted before the API server writes to etcd; the encryption is transparent to etcd.

flowchart LR
    A[Secret manifest] -->|API server| B[Validate]
    B --> C[Encode]
    C --> D{Encryption<br/>configured?}
    D -->|yes| E[Encrypt data]
    D -->|no| F[Plaintext]
    E --> G[etcd]
    F --> G

# Reading etcd directly (requires etcd access)
ETCDCTL_API=3 etcdctl get /registry/secrets/prod/db-credentials \
  --endpoints=https://etcd-1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

Without encryption at rest, the output is plaintext JSON with base64 values.

The access chain

flowchart TB
    A[Pod] -->|reads Secret| B[API server]
    B -->|validates RBAC| C[etcd]
    C -->|returns value| B
    B -->|returns to Pod| A

The chain has three access points:

1. API server — receives the Secret request, checks RBAC, reads from etcd, returns to the client.
2. etcd — stores the Secret. Anyone with direct etcd access (file system, network) can read the records.
3. File system on the node — mounted Secrets are projected into the kubelet’s data directory and bound into the container’s mount namespace. The projected files live on a tmpfs mount owned by root.

Each layer has a different protection mechanism.

Layer 1: API server protection (RBAC)

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: secret-reader
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "list"]

A Role with get on secrets grants the bound identities the ability to read Secret values. The default in most clusters: any authenticated user can get secrets in their accessible namespaces (the default RBAC grants).

A production discipline: audit every RoleBinding that grants get secrets. Every binding is a credential leak vector.

kubectl get rolebindings -A -o json | \
  jq '.items[] | select(.roleRef.name == "view") | {namespace: .metadata.namespace, name: .metadata.name}'

kubectl get clusterrolebindings -o json | \
  jq '.items[] | select(.roleRef.name == "view") | {name: .metadata.name}'

Layer 2: etcd protection

flowchart TB
    A[etcd] -->|file system| B[etcd data dir]
    B -->|disk encryption| C["Full-disk encryption<br/>LUKS / cloud KMS"]
    A -->|network| D["TLS<br/>peer and client certs"]
    D -->|auth| E["mTLS between<br/>API server and etcd"]

The default etcd setup:

• TLS — API server and etcd communicate over mTLS. Network access requires the right certs.
• File permissions — the etcd data directory is owned by the etcd user; only that user can read the files.
• Disk encryption — full-disk encryption on the etcd hosts. The OS sees plaintext; the disk is encrypted at rest.

What default etcd does not provide:

• Application-level encryption — the Secret values in etcd are plaintext. Anyone with file-system access to the etcd data directory sees the values.

# On an etcd node with root access:
strings /var/lib/etcd/member/snap/db | grep -i password

Layer 3: node-level protection

The kubelet projects mounted Secrets into the Pod’s volume directory:

/var/lib/kubelet/pods/<pod-uid>/volumes/kubernetes.io~secret-volume/<secret-name>/...

The Secret’s contents are on the node’s file system. Anyone with root on the node can read them.

# On a node with root access:
ls /var/lib/kubelet/pods/*/volumes/kubernetes.io~secret-volume/*/

# Pod UID, from kubectl get pod -o jsonpath='{.metadata.uid}':
POD_UID=3f2b7e14-9c8a-4d51-b6e0-77a1c2f8de93

cat "/var/lib/kubelet/pods/$POD_UID/volumes/kubernetes.io~secret-volume/db-credentials/password"

The mitigations:

• Node access controls — restrict root on nodes (SSH keys, audit logging).
• Container security contexts — runAsNonRoot, readOnlyRootFilesystem, allowPrivilegeEscalation: false.
• AppArmor / SELinux — confine the kubelet’s Secret projection so only the right containers can read it.

The protection summary

flowchart TB
    A[Secret value] --> B["Layer 1: API server RBAC"]
    B --> C["Layer 2: etcd access controls"]
    C --> D["Layer 3: Node file-system controls"]
    D --> E["Layer 4: Container security context"]
    E --> F[Container process reads]

The default-vs-procurement gap

Most production clusters have:

• Default RBAC (developers can get secrets).
• No encryption at rest (Secrets are plaintext in etcd).
• TLS on the etcd connection (network access requires certs, but file-system access is unprotected).
• Disk encryption on etcd hosts (protects against physical theft, not against root on the host).
• No AppArmor/SELinux profile on the kubelet (containers can read any mounted file).

A production-hardened cluster has every layer locked down: RBAC reviewed, encryption at rest configured, file-system permissions tight, container security contexts enforced. The default cluster has none of these; the operator’s responsibility is to add them.

Inspecting Secret access

# Who can read this Secret?
kubectl auth can-i get secret/db-credentials -n prod \
  --as=system:serviceaccount:prod:web

# Yes

kubectl auth can-i get secret/db-credentials -n prod \
  --as=system:serviceaccount:default:default

# No

flowchart LR
    A[kubectl auth can-i get secret] --> B[RBAC evaluation]
    B --> C{Allowed?}
    C -->|yes| D[Yes]
    C -->|no| E[No]

The kubectl auth can-i command evaluates RBAC for a given identity. Production operators use it to verify the principle of least privilege.

Quiz

Production discipline

• Audit every RoleBinding that grants get secrets. Every binding is a potential leak.
• Configure encryption at rest on the API server. Without it, etcd backups contain plaintext Secrets.
• Restrict root on etcd hosts. Disk encryption is not enough; root on the etcd host reads the data.
• Restrict root on worker nodes. The Secret mount path is readable by root; protect against node-level compromise.
• Use external secret managers for high-value credentials. Vault, AWS Secrets Manager, External Secrets Operator. The Secret in the cluster is a rendering, not the source of truth.

Secrets are stored in plaintext by default. The protections are layered; the operator’s job is to ensure every layer is correctly configured. A single missing layer is the chain that fails.