Key management services: what you delegate and what it costs

A key management service is not a place to keep secrets. It is a service that carries out cryptographic operations on key material you are not permitted to read. You present an identity, a key reference and a small payload; it returns a result. The key stays inside the service’s hardware boundary for the whole of its life, and almost every operational property in this part follows from that one design choice.

The operation surface you are buying

Every key management service, whether it is a cloud API or an appliance in your own rack, exposes roughly the same catalogue. The useful way to read that catalogue is to sort it by what crosses the boundary.

Operations that never return key material
  create key            a new key object, addressed by a stable identifier
  encrypt / decrypt     small payloads only, processed inside the module
  sign / verify         the private key signs; you receive the signature
  wrap / unwrap         one key encrypted under another, never in clear
  derive                a new key computed inside the module from an
                        existing one plus public inputs
  get public key        for asymmetric keys only; the public half is
                        not a secret

Operations that do return key material
  generate data key     returns a fresh symmetric key in clear AND an
                        encrypted copy of that same key

That last line is the whole exportability boundary in one place. The key-encryption key cannot be downloaded, and no sequence of API calls will produce it. A generated data key is deliberately handed back to you, in clear, so that you can use it on your own hardware. Confusing the two is the single most common design error in this area.

Notice also what is missing. The direct encrypt and decrypt operations are sized for key material, tokens and short configuration values. They are not a bulk data path: payload ceilings, per-request latency and request quotas all rule that out, which is why the two-tier pattern in the next lesson exists at all.

This is also the boundary that separates a key management service from a secret manager. A secret manager’s entire purpose is to return the secret to an authorised caller: a database password is useless unless the application receives it. A key management service is built on the opposite promise. If your design needs the key material itself, you have chosen the wrong component.

What the delegation actually buys

The gain is not that the cryptography is stronger. AES performed by your process and AES performed inside a validated module produce identical ciphertext. The gain is that several categories of failure stop being able to reach the key at all.

• The key is not in your address space. A heap-scraping vulnerability, an unbounded core dump, a debugger attached by a support engineer or a stray /proc read cannot recover what was never loaded.
• The key is not in your backups. A restored database, a cloned volume or a copied container image carries ciphertext and a key reference, not the means to read it.
• Authorisation replaces possession. Historically the answer to “who can decrypt this?” was “everyone who has ever held a copy of the file”, which is unanswerable. With a delegated key it becomes a policy document that a reviewer can read.
• Every use is an event. The service can record the caller, the key, the operation and the outcome. That converts key usage from an invisible act into something you can alert on.

What the delegation costs

Delegation moves risk rather than deleting it, and the new risks are chiefly about availability and authority.

Failure                        Consequence                       Data lost?
-----------------------------  --------------------------------  ----------
Service unreachable            all protected data unreadable     no
Policy misconfiguration        callers denied, or over-permitted no
Key reference deleted          protected data unreadable         yes
Wrapped data key lost          that object unreadable            yes
Caller credential stolen       attacker decrypts on demand       no
Region or tenancy unavailable  ciphertext bound to that service  no

Two rows deserve particular attention. The service does not keep a record of the data keys it has generated for you, so the wrapped copy of each data key is your responsibility to store next to the ciphertext it belongs to. Lose it and the object is gone, even though the key-encryption key is perfectly healthy. And scheduling the deletion of a key reference is the one action in this whole area that destroys data rather than merely blocking access to it.

The rest are availability problems wearing a data-loss costume. The data is intact; you simply cannot reach the key that opens it. That distinction matters enormously during an incident, because it tells you that patience and a restored dependency will fix things, and that a panicked restore from an older backup will not.

Cost and throughput are real constraints too. Every operation is a network round trip that is billed and rate limited, and a design that calls the service once per object read will discover both limits under load.

The last row is the one that surfaces during a migration rather than during an incident. Ciphertext produced under a particular key is only readable through the service instance that holds it, which binds your data to an account, a tenancy and often a region. Moving to a different provider, or to a different tenancy within the same provider, is therefore not a copy operation: every object has to be read through the old service and rewritten through the new one, with both available at once. Plan that as a project with a duration, not as a step in a cutover runbook.

Where the trust boundary moves to

flowchart LR
    A["Application host\nholds ciphertext only"] --> B["Caller identity\nrole, token or certificate"]
    B --> C{"Policy evaluated\nper request"}
    C -- "denied" --> D["Refusal recorded\nno key use"]
    C -- "allowed" --> E["Module boundary\nkey material stays inside"]
    E --> F["Result returned\ndata key or signature"]

Read that diagram from the attacker’s point of view. There is no arrow that carries the key-encryption key out of the module, so stealing the key is not on the menu. What is on the menu is the arrow labelled caller identity. An adversary who obtains the role, token or client certificate that your storage tier uses does not need the key, because the service will perform the decryption on their behalf and hand back the plaintext quite happily. Delegation therefore converts a key-protection problem into an identity-protection problem, which is usually a much better trade, but only if you notice that the trade happened.

The practical consequence is that key policy and caller authentication are now first-class security controls. Scope each key to the smallest set of principals that must use it, separate the principals that may encrypt from those that may decrypt where the workflow permits it, and treat any change to key policy as a change to the data’s confidentiality.

Production discipline

1. Decide what the key protects before you create it. A key is a blast radius. One key per data class with a documented owner beats one key per account with nobody responsible for it.
2. Write down the availability contract. Name, in the service design document, what happens to each dependent system when the key service is unreachable for five minutes and for five hours. If nobody has written it down, the answer is discovered during the outage.
3. Never let the recovery path depend on the thing it recovers. A disaster-recovery runbook whose first step requires decrypting a file under the key you are trying to restore is not a runbook.
4. Alert on refusals, not only on failures. A sudden run of authorisation denials for a key is either a broken deployment or an intruder probing what a stolen credential can reach. Both merit a look.
5. Treat key deletion as a destructive change. Require the same approval, the same waiting period and the same evidence of non-use that you would demand before dropping a database.

Cross-course references

• Linux for Production Sysadmins - Part LXXII (Secrets) covers how a host holds credential material on disk and in memory, which is exactly the exposure a delegated key removes.
• Kubernetes for Production Sysadmins - Part LXV (SecretsSec) covers encryption at rest for cluster Secrets, where the key-encryption key lives outside the cluster for precisely the reasons set out here.
• Observability for Production Sysadmins - Part XVIII (AlertingRules) covers turning a stream of refusals into an alert that fires before the pattern becomes an incident.

Quiz