Encryption in transit and at rest for backup data

Choosing a backup tool settled which repository format the estate will live in and what a client is permitted to do to it. It did not settle who can read what that client writes. The moment a backup leaves the machine that produced it, it stops being protected by that machine’s access controls and starts being protected by whatever the destination happens to enforce — and the destination is routinely a share nobody audits, a bucket created in a hurry, a disk in a rack at a colocation facility, or a cartridge in the boot of a car. A backup is simultaneously the most concentrated copy of an organisation’s information that exists anywhere: every database, every configuration file, every credential that happened to be sitting on a disk, in one object, with the change history attached. Densest data, weakest controls. That is why encryption here is not a hardening step to schedule for next quarter.

Three places encryption applies, and the adversary each one stops

“Encrypted backups” names three separate mechanisms that get collapsed into one word. They protect against different attackers, they fail independently, and an estate can have all three, exactly one, or none without any of them being aware of the others.

In transit is the encryption of the connection between the backup client and its destination: TLS to an object store or a REST server, an SSH channel to a remote repository, a tunnel carrying replication traffic between sites. It protects the bytes while they are on a wire other parties can touch — a network observer on a shared segment, a compromised switch or proxy, an intermediary that terminates one connection and originates another. Its defining limitation is that it stops precisely where the connection stops. At the far end the transport hands over whatever it was carrying, and if what it was carrying was plaintext, the destination now holds plaintext. Transport encryption is a statement about the path, never about the resting place.

At rest, by the backup tool is encryption the client performs before the data leaves the host. restic and Borg both work this way, and it is the model the rest of this part is built on. What travels is already ciphertext, so what the destination writes is ciphertext, and what a destination administrator can read is ciphertext. This layer is what protects against the storage operator, against a disk that leaves the building in a skip or a courier bag, against a bucket that someone made publicly readable while debugging, and against a support engineer at a provider you have no particular reason to distrust and no technical means to trust.

At rest, by the storage platform is the encryption the destination applies after it receives the object — the checkbox usually labelled “server-side encryption” or “encryption at rest”. The platform holds the keys, applies them on write, and applies them again on read. Nothing about reading changes: an authorised request receives plaintext, exactly as it would have without the feature. What this layer removes is the case where the physical medium leaves the platform’s control while still holding recoverable data.

These are three layers rather than three options, and a sound arrangement uses all of them, because they are cheap and they compose. The failure is not choosing wrongly among them. The failure is crediting the third with the properties of the second.

Server-side encryption stops the adversary you were not worried about

The threat that actually removes organisations’ backups is not somebody walking out of a datacentre with a drive under a coat. It is a credential: an access key that was on a compromised application server, a CI runner token, a long-lived key in a configuration file, an administrator’s session. That is the adversary the backup design has to survive, and server-side encryption contributes nothing at all to surviving it.

The reason is structural rather than a weakness in any particular implementation. Server-side encryption is designed to be transparent. The platform decrypts on the read path for every request it authorises, because if it did not, every consumer of the data would need key material and the feature would not be transparent. An attacker holding a valid credential is, by definition, a request the platform authorises. They issue an ordinary read and receive ordinary plaintext, and nothing in the audit trail distinguishes their read from the backup verification job’s read.

The AWS documentation for SSE-KMS describes the one variant where this gets more interesting: when an object is encrypted under a KMS key, decrypting it requires permission on that KMS key in addition to permission on the object, so the read path passes through a second authorisation decision with its own policy and its own logging. That is a genuine additional control, and it is worth having. It is also worth being honest about what it buys in a backup context, where the principal that writes the backups usually needs to read them back and therefore usually holds both permissions. When one identity holds both halves, a second authorisation check on the same identity stops nothing.

What the destination actually holds under client-side encryption

The model that survives an untrusted destination is the one restic and Borg implement: the client encrypts, and the destination is handed bytes it cannot interpret. That statement is easy to make and worth examining concretely, because the repository does contain a key, and people reasonably ask what protection a key inside the repository can possibly offer.

A key-loss capture answers it directly. A restic repository was created with its passphrase in a file, backups were taken, and then the production host was destroyed. The repository survived untouched — 11M of data across six files. Its keys/ directory was intact:

$ ls /work/repo/keys/

d5f39ef5517fab2e4d1e6dc1d9d5b4b1f1ca4eeacecde523af8b5a0c3c3d120e

That file is the master key, and it is stored encrypted under a key derived from the passphrase. Holding the repository is therefore not the same as holding the ability to open it, and copying a repository offsite copies the lock along with the door. Asked for its contents with no passphrase available, that same intact repository answered Fatal: wrong password or no key found and exited 12.

That single result carries the lesson’s central point in both directions, which is why the next lesson is devoted to it. Facing outward it is the security property: a thief who takes the repository — the whole repository, key file included — gets exactly that answer. There is no setting on the destination that changes it, no support ticket that recovers from it, and no difference between a hostile storage administrator and an idle one, which is what makes an untrusted destination usable at all. Facing inward it is the operational hazard, because the same wall stands in front of the recovery team, and the transcript is a recovery team’s view of a backup that survived a disaster perfectly and is nonetheless unreadable for good.

Borg states the arrangement at initialisation rather than leaving it to be discovered. Creating a repository under repokey-blake2 prints IMPORTANT: you will need both KEY AND PASSPHRASE to access this repo! and then explains that under repokey modes the key is stored in the repository directory, while under keyfile modes it is stored in the home directory of the user running the backup. Those two sentences are an architecture decision in disguise. Under repokey, every copy of the repository carries the encrypted key with it, so a copy plus a passphrase is sufficient to recover. Under keyfile, the key lives on the machine being protected, so a copy of the repository plus a passphrase is not sufficient — you also need a file that the disaster is likely to have taken.

Encryption is orthogonal to authorisation, retention and custody

Everything above concerns reading. A backup system can be destroyed without anybody reading anything, and encryption has no opinion whatsoever about that class of failure. Three cases are worth stating plainly, because each one gets quietly assumed away once the word “encrypted” is in the design document.

The first is deletion. The credentials that let a client write backups normally let it remove them, and the repository being encrypted changes nothing about that, because deletion never requires understanding the contents. A Borg capture ran the case with ordinary repository access — the access an ordinary backup client holds every night:

$ borg delete /work/repo::day1

>>> exit code: 0

day2                                 Fri, 2026-08-28 13:58:07 [85d3e533e094eb96663fd26a148b14737667bfc4251eea6b7144827ea2d279b8]
>>> exit code: 0

The repository was encrypted throughout. An attacker holding the backup credentials removed a recovery point and the repository cooperated, because ordinary access includes delete. Protection against this is immutability and access architecture, which Parts X and XI take up; it is not something a cipher can offer.

The second is retention. Pruning is authorised destruction of recovery points — the one operation in a backup system whose entire purpose is to remove them. In the same capture, borg prune --list --dry-run --keep-daily=1 kept a single archive and named it, which is a rehearsal of exactly how a retention change removes history. A policy edited from ninety days to seven destroys eighty-three days of recovery points on the next scheduled run, and every one of them was encrypted right up to the moment it stopped existing.

The third is an attacker who holds the key. Client-side encryption protects the data against the destination. It does not protect the data against the source, and an intruder who has been resident in the environment for weeks holds whatever the backup process holds — the passphrase file, the environment variable, the systemd credential, the agent’s session. This is why the placement of the key matters as much as its strength, and why a passphrase stored on the machine being backed up fails in both directions at once: an attacker who takes the machine gets it, and a disaster that takes the machine destroys it.

The question the rest of this part answers

The capture that opened this lesson is the shape of the problem the remaining lessons address. The data survived. The medium survived. The integrity was never in question — six files, 11M, the key file present and readable. What did not survive was the ability to read any of it, and no integrity check reports that condition, no retention policy prevents it, and no monitoring system has an alert for it, because from every angle except one the backup is perfectly healthy.

So the question that governs Part IX is not whether the backups are encrypted. It is this: if the environment is destroyed, who can still decrypt? Name the people, name where their key material lives, name the second custodian, and name the last date on which somebody proved the answer by actually decrypting something. The lessons that follow work through key custody and escrow, split knowledge, the dependency a KMS or HSM introduces into a recovery path, how to rotate without orphaning the history you still need, and how to test decryption so the answer is evidence rather than an assumption.

Production discipline

1. Encrypt on the client and treat every destination as untrusted. Make the bytes that leave the host already unreadable, so that a misconfigured bucket, a stolen medium and a curious operator are all the same non-event. Server-side encryption is then a supplementary control rather than the control.
2. Name the adversary each layer stops, in writing. Transport encryption stops an observer on the path and ends at the far endpoint; client-side encryption stops the destination; platform encryption stops physical media loss. A control nobody can attach an attacker to is a control nobody is entitled to count.
3. Read “encryption at rest: enabled” as a statement about media, not access. The platform decrypts for every request it authorises, so a stolen credential yields plaintext through the ordinary API. If SSE-KMS is in use, check whether the reading and writing permissions are actually held by different identities, because a second authorisation check on one identity changes nothing.
4. Keep the key out of the blast radius of the thing it protects. The measured failure was a passphrase file inside the directory being backed up: the repository survived at 11M across six files and restic snapshots answered Fatal: wrong password or no key found with exit code 12. Custody belongs somewhere the disaster cannot reach.
5. Prove decryption on a schedule, from outside the environment. A restore test driven by the production host proves the production host can decrypt. The claim that matters is that someone else can, after the production host is gone, and that claim decays until it is re-measured.

Cross-course references

• Secrets, PKI & Certificate Management for Infrastructure Engineers — Part XV (KMS, HSM and Key Protection) examines what a managed key service actually guarantees and where its boundaries are, which is the material needed to judge the third layer described here: a platform that decrypts for any caller it authorises is a different security object from one that hands out key material, and that distinction decides whether putting a KMS in a recovery path adds protection or adds a dependency.
• Linux for Production Sysadmins — Part LXXI (TLS and PKI) supplies the transport layer treated here as the first of the three places encryption applies, including the certificate and trust decisions behind a backup client’s TLS or SSH channel; it is also where the reason transport encryption ends at the far endpoint is developed properly, which is the premise this lesson builds the client-side argument on.
• Kubernetes for Production Sysadmins — Part LXV (Secrets Security) is the concentrated-copy problem in its sharpest form, because a cluster’s backing store holds whatever the cluster was told to keep; the client-side model argued for here is what decides whether a snapshot of that store can safely be written to a destination the platform team does not operate.

Quiz