Bitnami Sealed Secrets — cluster-bound encryption and the controller as the trust boundary

Sealed Secrets flips the encrypted-at-rest model so the private key lives inside the cluster, never on a developer’s laptop and never in CI. The kubeseal CLI fetches the cluster’s public certificate and uses it to encrypt a Secret manifest; the resulting SealedSecret resource is committed to Git; the in-cluster controller is the only component that holds the matching private key and is the only component that can produce a Kubernetes Secret from the ciphertext.

The model

The Sealed Secrets controller runs inside the cluster. On startup it generates an RSA keypair (or the operator provides one). The public half is served over a well-known endpoint; the private half is held only in cluster memory or in a Secret resource controlled by the operator.

flowchart LR
    A["plaintext Secret.yaml"] --> B["kubeseal --cert pub-cert"]
    B --> C["SealedSecret.yaml committed"]
    C --> D["Git repository"]
    D --> E["GitOps controller applies"]
    E --> F["Sealed Secrets controller decrypts"]
    F --> G["Kubernetes Secret created"]
    H["cluster private key"] --> F

The kubeseal CLI produces a SealedSecret bound to a namespace, a name, and (optionally) a specific cluster. A SealedSecret created against cluster A’s certificate cannot be decrypted by cluster B.

The kubeseal workflow

The basic encryption flow:

kubeseal --cert "$SEALED_SECRETS_CERT" \
         < secret.yaml > sealed-secret.yaml

The variable $SEALED_SECRETS_CERT is the path to the cluster public certificate, fetched once from the controller. The input is a normal Kubernetes Secret manifest. The output is a SealedSecret manifest that the controller can decrypt.

A recover-the-plaintext flow (when the cluster is reachable):

kubeseal --recovery-unseal --recovery-private-key "$RECOVERY_KEY" \
         < sealed-secret.yaml

The recovery private key is a separate key held by the team for break-glass scenarios. It is not the cluster controller key.

Sealed Secrets versus SOPS

The two encrypted-at-rest tools solve the same problem with different trust boundaries:

• SOPS puts the key holder wherever the engineer decides: age key on engineer’s laptop, KMS in the cloud, PGP key in the team’s hardware token. The key is portable; the cluster has whatever key the engineer configures in the controller.
• Sealed Secrets puts the key holder inside the cluster. The cluster is the trust boundary. A SealedSecret committed to any Git repository readable by anyone is unreadable to anyone except the cluster that minted it.

The choice between them is a portability choice: SOPS lets the same ciphertext be decrypted by multiple clusters with the right key; Sealed Secrets binds the ciphertext to one cluster. For multi-cluster GitOps, SOPS is more flexible. For single-cluster GitOps where the cluster is the trust boundary, Sealed Secrets is simpler.

Rotation

There are two rotations, and the difference matters:

• Value rotation. The credential value changes. The engineer produces a new plaintext Secret, runs kubeseal, commits the new SealedSecret. The cluster reconciles and produces a new Kubernetes Secret. The private key is unchanged.
• Key rotation. The cluster controller’s keypair rotates. Every previously committed SealedSecret is re-encrypted with the new certificate and recommitted. If the team has not kept a copy of the previous certificate, the previous SealedSecrets are unreadable.

A controller key rotation is a cluster-wide event that touches every committed file. Most teams avoid it by controlling the certificate’s lifecycle — pinning it to a stable controller deployment — and rotating values without rotating keys.

Failure modes

Three failure modes are specific to the class:

• Private key loss. The controller’s key Secret is destroyed. Every committed SealedSecret is unreadable. Recovery requires a backup of the key Secret, which is by design not in Git.
• Cluster replacement. A new cluster has a new keypair. The team’s existing SealedSecrets do not decrypt. The team must re-seal every Secret against the new cluster’s certificate, or maintain a sealed-secret-rotation script that runs at cluster bootstrap.
• Scope confusion. A SealedSecret created for namespace prod does not decrypt in namespace staging. The scope is enforced by the controller. Re-sealing with the right scope is required.

Production discipline

1. Back up the controller key Secret. Out-of-band — S3 with restricted access, a sealed envelope in a safe, a password manager. Without the backup, a key loss is a full cluster-wide re-seal.
2. Pin the certificate. Treat the cluster certificate as a long-lived identity. Rotating it requires touching every committed file.
3. Trust the cluster, not the operator. Sealed Secrets assumes the operator who installed the controller is trustworthy. The model has no defence against an operator who installs a controller holding a backdoored keypair.

Cross-course references

• Linux for Production Sysadmins - Part XXXIV (ConfigMgmt) covers per-host key storage; the cluster analogue is per-cluster key storage.
• Ansible for Production Sysadmins - Part XXXVII (RepoArch) covers Ansible Vault’s password file; the analogy to Sealed Secrets is the per-cluster key.
• Terraform for Production Sysadmins - Part XV (SensitiveVars) covers state-side encryption keys; the pattern is conceptually similar.

Quiz