Envelope encryption and the key hierarchy: why KEK and DEK are everywhere

You want a key that lives somewhere better guarded than the host doing the work, that you can rotate on demand, whose every use is audited, and that you can destroy to make data unrecoverable. You also want to encrypt four terabytes tonight. Those two wants are in direct conflict, and envelope encryption is the standard resolution: encrypt the bulk locally, then encrypt only the local key remotely.

Two keys with two very different jobs

The vocabulary is worth getting exactly right, because every cloud service and every platform uses it and each one abbreviates differently.

• Data encryption key, the DEK. A symmetric AEAD key, generated locally from the kernel generator, used once per object, volume, table or backup set. It is short-lived in memory and never stored in the clear.
• Key encryption key, the KEK. A key held inside a key management service or hardware module. It never leaves that boundary. It encrypts and decrypts DEKs, and nothing else.
• The wrapped DEK. The DEK after the KEK has encrypted it. It is not secret in the sense that matters, so it is stored next to the ciphertext it belongs to, in the same object, the same header, the same database row.

flowchart TD
    A["Operator-held shares\nor an IAM policy"] --> B["Root key\nreconstructed, never stored"]
    B --> C["Key encryption key\ninside a KMS or HSM"]
    C --> D["Data encryption key\none per object or volume"]
    D --> E["Ciphertext plus wrapped DEK\nstored together"]

The hierarchy exists so that each level protects a much smaller amount of material than the level below it. One root key protects a handful of KEKs; one KEK protects millions of DEKs; each DEK protects one object. Compromise or loss at any level has a blast radius you can state precisely, which is the property that makes the design defensible in a review.

The two paths, and where plaintext exists

Writing an object goes like this. The application asks the key service for a new data key. The service returns two things in one response: the DEK in plaintext, and the same DEK wrapped by the KEK. The application encrypts the object with the plaintext DEK using an AEAD, writes the ciphertext together with the wrapped DEK, and then discards the plaintext DEK from memory.

Reading is the mirror image. The application reads the ciphertext and the wrapped DEK, sends only the wrapped DEK to the key service, gets back the plaintext DEK, decrypts the object, and discards the DEK again.

flowchart LR
    R["Read object"] --> W["Wrapped DEK from the header"]
    W --> K["KMS decrypt call\nauthorised by IAM policy"]
    K --> P["Plaintext DEK in memory"]
    P --> D["AEAD decrypt the object"]
    D --> X["Discard the DEK"]

Three observations decide most design arguments. The bulk data never crosses the key service, so a four-terabyte backup costs one small API call rather than four terabytes of network transfer. The plaintext DEK exists only inside the application process for the duration of the operation, which is where your process isolation and memory handling matter. And the KMS call is the audit point: every decrypt is a recorded, authorised, revocable event, which is a property a key sitting in a file on disk can never have.

Why you meet this pattern everywhere

Once you know the shape you will see it in places that do not advertise it.

• Cloud object and block storage. Server-side encryption with a customer-managed key is exactly this, with the provider holding the DEK handling and you holding the KEK policy.
• Kubernetes encryption at rest. The KMS provider wraps data keys with a remote key rather than reading key material from a file on the control-plane host. That distinction is the whole point: with locally managed keys, encryption at rest defends against a stolen etcd snapshot and not against someone who has the control-plane host, because the key is on that host.
• Disk encryption. A LUKS keyslot holds the volume key encrypted under a key derived from your passphrase. Same hierarchy, with a human at the top instead of a key service.
• Secret managers. The barrier that a secret manager writes to its storage backend is encrypted under keys that are themselves protected by a root key the running process holds only in memory.

That last case makes the hierarchy visible as an outage. When an OpenBao instance starts, the root key does not exist anywhere on disk, and until enough unseal shares have been supplied to reconstruct it, every request fails:

$ bao kv get kv/app/config
Code: 503. Errors:

* Vault is sealed

The initialisation output states the design plainly: the service does not store the generated root key, and without enough shares to reconstruct it the data stays encrypted permanently. That is not a limitation to work around. It is the guarantee, expressed as an operational obligation to keep the shares safe, separate and recoverable.

Rotation, and the thing everyone gets wrong

Rotating a KEK creates a new key version. It does not re-encrypt anything. Existing ciphertext still carries DEKs wrapped by the previous version, and those DEKs can only be unwrapped by that previous version, which therefore must remain enabled.

• To finish a rotation properly you re-wrap, not re-encrypt. Unwrap each stored DEK with the old key version and wrap it with the new one. The bulk data is untouched, so this is fast even for a very large dataset.
• Disabling an old key version breaks reads. Everything still wrapped by it becomes unreadable at that moment. Complete the re-wrap first, then disable, then verify by reading a sample from the oldest data you have.
• Destroying a KEK is a data destruction operation. Every wrapped DEK under it becomes permanently unrecoverable. Used deliberately this is crypto-shredding, which is how you honour a deletion request across backups you cannot selectively edit. Used accidentally it is the most complete data loss available to you, which is why key services impose a waiting period before destruction and why that waiting period deserves an alert rather than an email.

A DEK is rotated differently: because it protects one object, you rotate it by re-encrypting that object, which you do anyway when the object changes. This is why per-object DEKs are not a bookkeeping burden.

The dependencies you have just acquired

Envelope encryption removes a key from your host and adds a service to your critical path. Both halves are real.

Availability now couples your data plane to the key service. If it is unreachable, or the credential used to call it has expired, or its rate limit is reached during a mass restore, you cannot decrypt. The standard mitigation is caching a plaintext DEK for a bounded period, and the bound is a trade you must make consciously: a longer cache survives a longer outage and also extends the window in which a compromised process holds usable key material. Write the number down and justify it.

Authorisation becomes a distributed decision. The identity making the decrypt call is what the key policy evaluates, so workload identity, token lifetime and role assumption all become part of your encryption design rather than adjacent to it. This is exactly the ground Parts XIII and XIV cover.

Binding is the defence against substitution. Passing an encryption context, which the key service authenticates as associated data when it wraps and requires again when it unwraps, stops a wrapped DEK being lifted from one object and presented alongside another. Without it, an attacker who can write to your storage can pair any ciphertext with any wrapped key they are entitled to unwrap.

Production discipline

1. One DEK per object, one KEK per trust boundary. Sharing a DEK across objects removes the blast radius argument that justified the design.
2. Always pass an encryption context. Bind the wrap to the object identity so a wrapped key cannot be moved.
3. Treat key destruction as a destructive change. Alert on scheduled deletion, require two people, and confirm nothing still references the key before the window elapses.
4. Back up the wrapped keys with the data, and rehearse the restore. A restore test that does not call the real key service has not tested the part that fails.
5. Bound and justify the DEK cache. State the outage it is meant to survive and the exposure it accepts, and review both when either changes.

Cross-course references

• Kubernetes for Production Sysadmins - Part LXV (SecretsSec) covers the cluster encryption providers where the choice between a local key file and a remote key service is made.
• Linux for Production Sysadmins - Part XLVII (Backup) covers the archive jobs that must carry wrapped key material alongside the data for a restore to be possible.
• Git, CI/CD & GitOps for Infrastructure Engineers - Part LXXXII (GitOpsSecrets) covers the repository patterns that store wrapped secrets in version control, which is this hierarchy applied to configuration.

Quiz