Entropy, random number generation and key derivation on modern Linux

Every key you have ever generated, every nonce that keeps an AEAD safe, every salt, session identifier and certificate serial began life as output from a random number generator. When that generator is healthy nobody thinks about it. When it is not, the failure does not look like a cryptography problem: it looks like two hosts with the same SSH host key, or a service that hangs for ninety seconds at boot, or a token space an attacker can enumerate.

What the kernel actually gives you now

A great deal of advice about Linux randomness was written for a kernel that no longer exists, and following it today makes things worse rather than better. The current picture is simpler than the folklore.

• There is one generator. /dev/urandom, /dev/random and the getrandom system call all draw from the same cryptographically secure generator. Once it is seeded, output from any of them is equally good and there is no quality difference to trade against.
• Blocking happens only before seeding. On a modern kernel /dev/random blocks until the generator has been initialised and never again. It does not block because an entropy counter ran low, because that model was abandoned.
• The generator does not run out. A seeded generator produces as much output as you ask for. Watching entropy_avail and panicking at a small number is a habit left over from the old model, and it drives people to install entropy daemons that add risk without adding security.
• Containers do not have their own generator. A container uses the host kernel’s, so container density is not an entropy concern. A virtual machine does have its own kernel and does need a seeding story, which is what virtio-rng provides.

The one hazard that remains is real and narrow: reading randomness before the generator has been seeded, which on physical hardware with little activity and no hardware source can take a surprisingly long time after power-on.

flowchart LR
    A["Interrupt and device timing"] --> P["Kernel CSPRNG"]
    B["CPU instruction\nRDRAND or RDSEED"] --> P
    C["virtio-rng from the hypervisor"] --> P
    D["Seed file restored at boot"] --> P
    P --> E["getrandom, /dev/urandom, /dev/random"]
    E --> F["Keys, nonces, salts,\nserials, session ids"]

Read the diagram from the right. Everything your estate treats as unguessable comes out of one generator, and that generator is only as good as the moment it was first seeded. Everything upstream of it exists to make that moment happen early and reliably.

Where randomness quietly fails

Three conditions produce predictable or duplicated key material, and all three are infrastructure problems rather than cryptographic ones.

• Early boot before seeding. A service that requests randomness during very early boot will block until the generator is ready. The symptom is a unit that hangs and then starts normally, often on physical machines and on freshly provisioned virtual machines without a hypervisor random source. The fix is to provide a real entropy source rather than to make the application read something weaker.
• A cloned image. This is the common one. If a machine template was built with SSH host keys already present, every machine cloned from it presents the same host key, and any of them can impersonate the others to every client that trusted the first. The same applies to a persisted random seed file, to pre-generated certificate keys, and to any credential baked into the image.
• A snapshot resumed twice. Restoring the same memory snapshot twice replays generator state, so both instances can produce identical output. Nonce reuse under a shared key follows directly, with the consequences from lesson 2.

The cloned-image case is worth checking on any estate you inherit, because it is silent and cheap to detect:

# Any repeated fingerprint means two hosts share the same host key.
for h in web-01 web-02 web-03; do
  ssh-keyscan -t ed25519 "$h" 2>/dev/null | ssh-keygen -lf -
done

The remedy is to remove host keys and any seed file from the template, and to regenerate them on first boot. Cloud images do this through their initialisation tooling; hand-built templates frequently do not, and the defect survives every subsequent audit because nothing in the running system looks wrong.

Two jobs called key derivation

A key derivation function turns one secret into key material. There are two entirely different situations, they have opposite performance requirements, and swapping them is a common and consequential mistake.

The first situation is a low-entropy human secret: a passphrase, a PIN, a recovery phrase. An attacker who obtains the derived value will guess candidates offline at enormous speed, so the derivation must be deliberately expensive. Argon2, scrypt, bcrypt and PBKDF2 exist for this. The cost parameters are the security control, not an implementation detail.

The second situation is an existing high-entropy secret that you need to turn into several keys: the shared secret from a key agreement, a 256-bit random master key, a token you want to diversify per purpose. Here guessing is already infeasible, so slowness buys nothing. HKDF is the standard answer, and its value is domain separation: the same input secret plus different context strings yields independent keys that cannot be substituted for one another. TLS 1.3 uses exactly this to derive handshake and application traffic keys from one agreed secret.

# Generate a high-entropy secret. There is nothing to strengthen
# here; it is already drawn from the kernel generator.
openssl rand -base64 32

# Inspect the derivation parameters an encrypted volume will use.
cryptsetup luksDump /dev/sda2

The second command matters more than it looks. A password-based derivation is calibrated at format time against the machine doing the formatting. Format a volume on a fast build host and deploy the image to a small appliance and the unlock cost is unchanged for the attacker with a fast machine while becoming painful for the appliance, or, if the parameters were tuned down to make the appliance usable, cheap for everyone. Whenever you see a passphrase-protected artefact, look at its parameters before you decide it is protected.

Salts, and the difference between secret and unique

A salt is a unique value stored alongside the derived output. It is not secret and does not need to be. Its job is to make sure two identical passphrases derive to different values, which defeats precomputed tables and stops an attacker from learning that two accounts share a password. Every password-based derivation function generates one for you, and it must come from the kernel generator rather than from a counter or a username.

A pepper is a different thing: a secret value, held outside the database, mixed into the derivation. It helps only in the specific case where an attacker steals the database and not the application’s configuration, which is a narrower scenario than it first appears. Treat it as a supplement to a properly parameterised derivation, never as a replacement, and give it the same rotation plan as any other key.

Production discipline

1. Provide a hypervisor random source to every guest. It is one line of virtual machine configuration and it removes the whole class of early-boot seeding problem.
2. Strip identity from templates. No host keys, no seed file, no service keys, no machine identifier. Generate on first boot and verify by comparing fingerprints across the fleet.
3. Use the platform generator for every secret. openssl rand or a language runtime’s secure generator. Never a shell variable, a timestamp, a process identifier or a hash of the hostname.
4. Record derivation parameters as configuration. Iterations, memory and parallelism should be reviewable values with a documented reason, revisited as hardware changes.
5. Match the function to the input. Slow derivation for human secrets, HKDF with distinct context strings for everything else. Reusing one derived key for several purposes turns any single compromise into a total one.

Cross-course references

• Linux for Production Sysadmins - Part LXXVI (VirtGuest) covers the guest configuration where a hypervisor random device is attached, which is the practical fix for guest seeding.
• Kubernetes for Production Sysadmins - Part LXXV (BuildCluster) covers node provisioning, the point at which a cloned template would propagate duplicated key material across a cluster.
• Observability for Production Sysadmins - Part LVI (LinuxObs) covers the host metrics and unit timings that make a service blocking at boot visible rather than folklore.

Quiz