Kubernetes Secrets and the cluster PKI

A Kubernetes Secret is an API object whose data map holds base64 text. Nothing in that encoding is a protection: decoding needs no key, no password and no privilege. The object’s real defences are the API server’s authorisation layer, whatever encryption provider is configured for etcd, and the node boundary. This lesson is about where each of those three defences begins and, far more usefully in an incident, where each one stops.

Base64 is a transport encoding, not a control

The data field of a Secret carries base64 because the API is JSON and JSON has no byte-string type. Kubernetes says so plainly in its own documentation: the encoding obscures the value but provides no useful confidentiality. A stringData field exists for convenience on write and is folded into data before storage, so it never changes what is persisted.

NS=payments
kubectl -n "$NS" get secret api-credentials -o jsonpath='{.data.password}' | base64 -d

Anyone who can run that command holds the credential. That is the whole of the arithmetic. It follows that a Secret pasted into a ticket, a chat thread or a support bundle in its base64 form is a disclosed credential, and the only correct response is rotation.

Four properties of the object matter operationally. A Secret is capped at 1 MiB, deliberately, to stop very large objects exhausting API server and kubelet memory. Its type field (Opaque, kubernetes.io/tls, kubernetes.io/dockerconfigjson and the rest) drives key-name validation only and never changes how the bytes are stored. A Secret is delivered to a node only when a Pod scheduled there requires it, and the kubelet keeps the mounted copy in tmpfs so the confidential data is not written to durable storage. Finally, a mounted Secret is updated eventually after the object changes, delayed by the kubelet sync period plus cache propagation, with one important exception: a container that mounts the Secret through subPath never receives automated updates at all. Marking a Secret immutable: true removes the watch cost and is one-way; you cannot revert it.

The blast radius is the namespace, not the object

The single most misunderstood property of Kubernetes Secrets is the size of the isolation boundary. The upstream documentation states that anyone authorised to create a Pod in a namespace can use that access to read any Secret in that namespace, and it names indirect routes such as creating a Deployment. There is no per-Secret container isolation to fall back on either: a container running with privileged: true can reach every Secret in use on its node.

Two adjacent RBAC facts complete the picture. Granting list or watch on Secrets returns the data of every Secret in the namespace, not merely the ones a subject’s own workloads reference, because the response body carries the objects themselves. And anyone with direct access to etcd bypasses the API server entirely.

flowchart TD
    A["Subject with RBAC on secrets"] --> S["Secret data"]
    B["Subject who can create a Pod\nor a Deployment"] --> S
    C["Privileged container on the node"] --> S
    D["Direct etcd access"] --> S
    E["Node with a compromised kubelet"] --> S

Every arrow in that diagram terminates at the same value, and only the first one is visible in a RoleBinding audit. When you are asked who can read a credential, the honest answer enumerates all five paths. That is why namespace design is secret design: a workload that must not read a credential belongs in a different namespace, not merely in a different ServiceAccount.

The documented mitigations are, in order: enable encryption at rest, write least-privilege RBAC, restrict Secret access to the specific containers that need it, and move the material to an external secret store. Only the last of those changes the read paths above, because it removes the value from the API object entirely.

Encryption at rest, and the compromise it does not survive

By default the API server writes Secret bytes to etcd unencrypted. Encryption at rest is switched on with an EncryptionConfiguration document in the apiserver.config.k8s.io/v1 API group, wired in with the --encryption-provider-config flag.

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - kms:
          apiVersion: v2
          name: cluster-kms
          endpoint: unix:///var/run/kmsplugin/socket.sock
      - identity: {}

Provider order is the whole semantic. The first entry in the list performs every write; every entry is tried on read. Putting identity first therefore means new writes are plaintext, which is exactly how you roll the feature back. identity is also the default when no configuration is supplied.

The provider choice is not a matter of taste. Kubernetes documents aescbc as weak because CBC is vulnerable to padding-oracle attacks. Plain aesgcm must be rotated every 200,000 writes and is not recommended without an automated rotation scheme. secretbox is documented as strong but relatively new, which is a real obstacle in an environment with a review board. KMS v2 has been stable since v1.29 and is the target to aim at; KMS v1 was deprecated in v1.28 and disabled by default from v1.29, so it needs an explicit feature gate to run at all.

Now the sentence that changes how you brief an incident. Encrypting Secret data with a locally managed key protects against an etcd compromise, and it fails to protect against a host compromise, because the keys live on the control-plane host in the EncryptionConfiguration file. An attacker with a shell on that host has the ciphertext and the key in the same directory tree. KMS v2 moves the key-encryption key out of that blast radius; a local provider does not.

kubectl get secrets --all-namespaces -o json | kubectl replace -f -

Enabling a provider encrypts nothing that already exists. That rewrite is what re-persists every Secret through the new first provider, and it is safe to run more than once. Two operational constraints follow it. Every control-plane node must carry an identical configuration, or an API server may be unable to decrypt what a peer wrote. And if a key is lost and no working configuration can be restored, the documented recourse is deleting the affected entry from etcd directly, which means the Secret is gone.

Three certificate authorities and one key pair that is not one

A kubeadm control plane depends on three separate CAs, not one. Each is an independent trust domain, and confusing them produces failures that look like network problems.

/etc/kubernetes/pki/ca.crt              CN=kubernetes-ca
/etc/kubernetes/pki/etcd/ca.crt         CN=etcd-ca
/etc/kubernetes/pki/front-proxy-ca.crt  CN=kubernetes-front-proxy-ca
/etc/kubernetes/pki/sa.key              ServiceAccount signing key
/etc/kubernetes/pki/sa.pub              ServiceAccount verification key

The cluster CA signs the API server’s serving certificate and the client certificates components use to reach it. The etcd CA signs etcd server, peer and healthcheck certificates plus the API server’s etcd client certificate. The front-proxy CA signs only the aggregation-layer client certificate. A certificate issued by the right key but the wrong CA is refused, and the log line will talk about an unknown authority rather than about the mix-up.

The last two lines of that listing are not certificates at all. The ServiceAccount signing key pair is a raw asymmetric key pair: the controller manager signs tokens with the private half and the API server verifies them with the public half. It carries no validity window, so it never expires and nothing renews it. Anyone who copies sa.key can mint a token for any ServiceAccount in the cluster, and no revocation exists to stop them. Treat that file as the most valuable object on the control plane.

kubeadm issues leaf certificates with a validity of 8760h and CAs with 87600h, both settable as certificateValidityPeriod and caCertificateValidityPeriod in a kubeadm.k8s.io/v1beta4 ClusterConfiguration. Two administrator kubeconfigs exist and they are not equivalent: admin.conf carries O = kubeadm:cluster-admins, CN = kubernetes-admin, while super-admin.conf is the system:masters break-glass file. Both belong in a safe, not in a laptop’s home directory.

One more trap belongs here. The ca.crt that appears inside a projected ServiceAccount token volume is only guaranteed to verify a connection to the API server through the kubernetes.default.svc service. It is not a general-purpose organisational trust anchor, and using it as one produces verification failures that are very hard to reason about later.

Production discipline

1. Treat a base64 value as disclosed. A Secret quoted in a ticket, a screenshot or a support bundle is a rotation event, not a redaction exercise.
2. Audit namespaces, not RoleBindings. Enumerate who can create a Pod or a Deployment in the namespace before you claim to know who can read a credential in it.
3. Never ship aescbc as a final answer. If a local provider is all you can have today, record it as an interim control with a dated plan to reach KMS v2.
4. Rewrite after every provider change. Enabling, rotating or disabling a provider changes nothing already stored until every Secret is re-persisted, and the proof is read from etcd.
5. Back up sa.key like a CA key. It signs every ServiceAccount token, it never expires, and there is nothing to revoke if it leaks.

Cross-course references

• Kubernetes for Production Sysadmins - Parts LXV (Secrets Security) and LXVI (etcd) cover the cluster-side operation of these objects, the etcd datastore they live in, and the RBAC analysis this lesson assumes you can already perform.
• Linux for Production Sysadmins - Part LXXII (Secrets) covers file permissions, world-readable material and shell history, which is the layer beneath the tmpfs mount the kubelet creates.
• Observability for Production Sysadmins - Part LXXXII (Secrets and Sensitive Telemetry) covers keeping decoded values out of the logging pipeline that scrapes the workloads consuming them.

Quiz