Skip to main content
RunBook Academy

← All checklists in Secrets, PKI & Certificates

Before deploymentk8s-secret-integration

Kubernetes Secret Integration Checklist

22 items ·16 critical ·5 warn ·1 info

How to use this checklist

Run this before a workload that holds production credentials is allowed into a cluster for the first time, and again whenever the cluster crosses a minor version or a namespace gains a new tenant. It is a pre-deploy gate rather than a periodic audit, so the failing condition is simple: the workload does not go in until the critical items pass or somebody senior signs an acceptance naming what they are accepting.

The items are ordered the way an attacker would read them. Storage first, because that is what a stolen etcd snapshot gives up. Then authorisation, because in this system the namespace is the wall and the Secret is not. Then token behaviour, because most of the surprises in production are about lifetimes nobody measured. Work through them in that order and the later answers will make more sense.

Where the numbers come from

Provider configuration, flag values and static Pod manifests are read from the control-plane nodes themselves. Do not take them from the cluster documentation or from the installer variables, because the file on disk is what the process parsed and the two drift whenever somebody fixes an outage by editing a manifest.

Authorisation answers come from asking the API server directly with an impersonated identity rather than from reading role definitions. Reading roles tells you what someone wrote; impersonation tells you what the aggregate of every binding actually permits, which is the number this review needs.

Two items are attestations. Whether the incident procedure correctly states that bound tokens cannot be revoked, and whether the choice between in-cluster Secrets and an external store has been recorded with a named approver, are answered by reading a document and naming who wrote it. No command can tell you whether a team believes the right thing.

Access this needs

Read access to the namespaces under review, permission to impersonate the service accounts being checked, shell access on one control-plane node for the manifest and provider files, and a read-only etcd client for the single storage verification. One item mints a ten-minute token to prove the request-time path works; schedule it with the workload owner so nobody sees an unexplained token issued to a deployment identity.

What the review produces

A signed pre-deploy record naming the workload, the namespace, the cluster version it was reviewed against, and the disposition of every item. Attached to it: the list of identities that can create workloads in the namespace, which is the effective list of holders of every credential in it, and the accepted-risk entries with their expiry dates. The record goes to the workload owner and the platform owner. Re-run the review when the namespace tenant list changes, because that list is the answer to the most important item here.

Sign-off

  • Reviewer: ________________ Date: ___________
  • Platform owner: ___________ Date: ___________
  • Security owner: ___________ Date: ___________

Every critical item must pass. A failing critical item blocks the workload from being deployed into this namespace; it is not a note for later, because there is no later once the credential is in the cluster. Record the date, the reviewer, and the disposition of every item that did not pass.

Critical16 items

  1. kubectl get secret app-db -n payments -o jsonpath="{.data.password}" | base64 -d | wc -c
  2. sudo grep -n "encryption-provider-config" /etc/kubernetes/manifests/kube-apiserver.yaml
  3. sudo grep -nE "identity|aescbc|aesgcm|secretbox|kms" /etc/kubernetes/enc/encryption-config.yaml
  4. sudo ls -l /etc/kubernetes/enc/
  5. for h in cp-1 cp-2 cp-3; do ssh "$h" "sudo sha256sum /etc/kubernetes/enc/encryption-config.yaml"; done
  6. sudo etcdctl --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key get /registry/secrets/payments/app-db | hexdump -C | head -n 4
  7. kubectl auth can-i create pods --namespace payments --as system:serviceaccount:payments:ci-deployer
  8. kubectl auth can-i list secrets --namespace payments --as system:serviceaccount:payments:ci-deployer
  9. kubectl get pods -A -o json | jq -r '.items[] | select(any(.spec.containers[]; .securityContext.privileged == true)) | .metadata.namespace + "/" + .metadata.name'
  10. kubectl get secrets -A --field-selector type=kubernetes.io/service-account-token
  11. kubectl create token deploy-bot --namespace payments --audience vault.example.com --duration 10m
  12. kubectl get pod api-7f9c8 -n payments -o json | jq -r '.spec.volumes[] | select(.projected) | .projected.sources[] | select(.serviceAccountToken) | .serviceAccountToken'
  13. sudo grep -nE "service-account-(extend|max)-token-expiration" /etc/kubernetes/manifests/kube-apiserver.yaml
  14. sudo grep -rnE "secretRef|secretKeyRef|configMapRef" /etc/kubernetes/manifests/
  15. kubectl get --raw /.well-known/openid-configuration | jq -r ".issuer, .jwks_uri"

Warning5 items

  1. kubectl get pods -A -o json | jq -r '.items[] | select(.spec.automountServiceAccountToken != false) | .metadata.namespace + "/" + .metadata.name' | head -n 20
  2. kubectl get pods -A -o json | jq -r '.items[] | .metadata.namespace + "/" + .metadata.name + " " + ([.spec.containers[].volumeMounts[]? | select(.subPath) | .name] | tostring)'
  3. kubectl get secrets -A -o json | jq -r '.items[] | [(.data | tostring | length), .metadata.namespace, .metadata.name] | @tsv' | sort -rn | head -n 5
  4. kubectl get secrets -A -o json | jq -r '.items[] | select(.immutable == true) | .metadata.namespace + "/" + .metadata.name'

Info1 item