Offsite copies and the 3-2-1 rule · Docker & Containers

The rule is old enough to be furniture:

3 copies of the data.
2 different media or storage types.
1 copy offsite.

It is good advice and it is routinely satisfied on paper by an arrangement that one command can destroy. This lesson is about the test the rule does not contain.

The blast-radius test

For each copy of your data, ask: what single event destroys it? Then look for events that appear in more than one row.

Here is the arrangement most single-host Docker deployments actually have.

Copy	Where	Destroyed by
Live data	`app_pgdata` volume on host-01	Host loss, `docker volume rm`, ransomware on host-01
Nightly tar	`/backup` on host-01	Host loss, disk failure, ransomware on host-01
restic repository	S3, credentials in `/etc/restic` on host-01	Ransomware on host-01 — the credentials are right there

Three copies. Two media. One offsite. It satisfies 3-2-1 completely, and a single compromise of host-01 destroys all three, because the third copy is deletable by a credential stored on the first.

That is the failure mode 3-2-1 was written before, and it is now the common one. A ransomware operator’s first action after gaining a foothold is to enumerate and delete backups — the encryption is worthless to them if you can restore. They look in /etc/restic, ~/.aws/credentials, /root/.borg-passphrase and the crontab, and they find them, because that is where a working backup job keeps them.

Making the third copy independent

Three mechanisms, in increasing order of strength. Pick based on what your backend supports.

Append-only credentials

The backup client can write new data and read existing data, but not delete. restic supports this directly via a REST server, and Borg via borg serve --append-only over SSH.

Configuration changeappend-only restic server— Runs on the backup host, not the production host. --append-only is what makes it work.

# On the BACKUP host. Clients get write+read, never delete.
docker run -d --name restic-server --restart unless-stopped -p 8000:8000 -v /srv/restic-repos:/data -v /srv/restic/.htpasswd:/data/.htpasswd:ro restic/rest-server:latest --append-only --private-repos --path /data

Read-only / Safeprove it is append-only— Run from the PRODUCTION host. This command must FAIL.

export RESTIC_REPOSITORY='rest:http://backup-01.example.com:8000/app-01'
export RESTIC_PASSWORD_FILE=/etc/restic/password

# Pick any snapshot and attempt to remove it
SNAP=$(restic snapshots --json | head -c 200)
echo "$SNAP"

# This is the verification. A zero exit status here means the
# append-only property is NOT in effect and the design has failed.
if restic forget --keep-last 1 --prune; then
echo 'FAIL: production host can delete backup history' >&2
exit 1
else
echo 'PASS: deletion refused by the server'
fi

Read-only / Safethe refusal you want— Run from production. The non-zero exit is the evidence that the copy is independent.

$ restic forget --keep-last 1 --prune

Applying Policy: keep 1 latest snapshots
remove 41 snapshots:
ID        Time                 Host        Tags
--------------------------------------------------
7c1a9e04  2026-08-11 02:00:14  app-01      app_uploads
...
--------------------------------------------------

Save(<lock/9d3f2a71bc>) returned error, retrying after 552.330144ms:
server response unexpected: 403 Forbidden (403)
Fatal: unable to create lock in backend: repository is in append-only mode

PASS: deletion refused by the server

Illustrative output

Note that restic gets as far as listing what it would remove before the server refuses. Reading only the first half of that output — 41 snapshots queued for deletion — and concluding the protection failed is an easy mistake. The 403 Forbidden and the non-zero exit are the answer.

This is verification that can fail, which is the point. Most “we have offsite backups” claims have never been tested this way, and a surprising number of append-only configurations turn out to have been applied to the wrong repository path or lost across a container recreation.

Append-only and `restic forget` do not get along by default

Somebody still has to expire old snapshots or the repository grows without bound. On an append-only repository the production host cannot, which is correct — so the pruning has to run from the backup host, or from a separate privileged credential used by a scheduled job that the production host cannot reach.

Miss this and one of two things happens. Either the repository fills the backup volume in eight months and nightly backups start failing — loudly, which is the good outcome — or somebody “fixes” the append-only restriction to make forget work, and the protection is gone with no record of why it was there.

There is a subtler trap the restic documentation names explicitly: on an append-only repository, an attacker who can add snapshots can add enough garbage ones that a --keep-last-style policy expires all the legitimate ones. Use --keep-within on append-only repositories, which expresses retention as a time window rather than a count and cannot be gamed by volume.

Object Lock (WORM)

Stronger, because it is enforced by the storage service rather than by an application on a host you also operate.

Configuration changeS3 Object Lock— Object Lock works only on versioning-enabled buckets. Enable at bucket creation.

BUCKET=example-app-backups

# Object Lock requires versioning; the flag enables both at creation time.
aws s3api create-bucket --bucket "$BUCKET" --region eu-west-1 --create-bucket-configuration LocationConstraint=eu-west-1 --object-lock-enabled-for-bucket

# Default retention: every object placed in the bucket is locked for 35 days.
aws s3api put-object-lock-configuration --bucket "$BUCKET" --object-lock-configuration '{
  "ObjectLockEnabled": "Enabled",
  "Rule": { "DefaultRetention": { "Mode": "GOVERNANCE", "Days": 35 } }
}'

# Confirm it applied
aws s3api get-object-lock-configuration --bucket "$BUCKET"

GOVERNANCE, COMPLIANCE, and what a delete actually does

The two modes are not a strength dial; they are different threat models.

GOVERNANCE blocks deletion for anyone without the s3:BypassGovernanceRetention permission. A holder of that permission can delete a locked object by sending x-amz-bypass-governance-retention:true. So GOVERNANCE protects against your production host, a leaked application credential, and ordinary mistakes — which is the realistic ransomware and fat-finger threat model — while leaving you a way out if you locked something for thirty years by accident.

COMPLIANCE blocks deletion for everyone, including the account root user, until the retention expires. AWS is explicit that the only way to delete a COMPLIANCE-locked object before expiry is to delete the AWS account. That is the right choice for a regulatory requirement and a serious hazard otherwise: a misconfigured retention period becomes a bill you cannot stop paying.

Also understand what a delete request does to a locked object, because it determines what you see:

A versioned delete (specifying the version ID) returns 403 Forbidden. This is the visible refusal.
A simple delete returns 200 OK and inserts a delete marker. The object is still there, protected, under its version ID — but a naive aws s3 ls shows it gone.

That second case is why a “can I delete it?” test must be a versioned delete. A simple-delete test appears to succeed and teaches you the wrong thing.

Note also that retention and legal holds protect specific object versions. They do not prevent new versions being written on top. That is fine for backup archives, which are immutable objects with unique names, and matters for anything you overwrite in place.

A pull-based backup

The strongest arrangement and the least used: the backup server initiates the connection and pulls, so the production host holds no backup credentials at all and does not know where the backups live.

The production host writes its dumps to a local staging directory. The backup host connects over SSH with a restricted key, reads the staging directory, and stores the result in a repository the production host cannot address. A compromise of production yields the staging directory — one night’s data — and nothing else.

The cost is a backup host that has read access to production, which is its own concentration of risk and needs the same care. It is the right shape when the data justifies it.

Two media, honestly

The “2” in 3-2-1 predates cloud storage and gets treated as a formality. Its actual purpose is to avoid a correlated failure across copies, and correlation today comes from software far more than from hardware.

Failures that hit every copy in the same technology at once:

A bug in your backup tool that writes archives it cannot read. Same tool everywhere, same corruption everywhere.
A cloud provider region or account issue. Same provider for the “offsite” copy and the primary means one billing dispute takes both.
A filesystem or storage-driver bug. Two ZFS pools on two hosts running the same kernel are one bug apart.

Which gives a practical reading of “2 media”: make the second copy differ in the dimension most likely to fail. For most Docker stacks, the highest-value diversity is format, not hardware — a restic repository and a plain pg_dump file, because a plain SQL dump is readable by anything and depends on no tool being intact.

Retention that survives slow corruption

Deletion is the loud failure. The quiet one is corruption that started weeks ago and has been faithfully backed up ever since.

A retention policy of “keep 7 daily” gives you a one-week window to notice. A table dropped by a bad migration five weeks ago, or a slow encryption run that began before anyone noticed, is beyond recovery — every retained snapshot contains the damage.

Data-loss riskrestic retention— forget removes snapshots and --prune deletes their data irreversibly. Dry-run first.

export RESTIC_REPOSITORY='rest:http://backup-01.example.com:8000/app-01'
export RESTIC_PASSWORD_FILE=/etc/restic/password

# ALWAYS run without --prune first and read the list of what would go
restic forget --keep-daily 14 --keep-weekly 8 --keep-monthly 12 --keep-yearly 3 --host app-01 --dry-run

# Only then, with --prune to reclaim the space
restic forget --keep-daily 14 --keep-weekly 8 --keep-monthly 12 --keep-yearly 3 --host app-01 --prune

forget removes snapshots from the index; prune removes the data that no remaining snapshot references. Running forget alone is reversible in practice until a prune happens, which is why --dry-run before --prune is the habit worth having. Note also --host app-01 — without it, a policy expressed as “keep 14 daily” is applied across every host’s snapshots collectively, which silently thins the machines that back up less often.

The shape above — a fortnight of dailies, two months of weeklies, a year of monthlies — costs very little with deduplication and moves your detection window from one week to one year. For most stacks the incremental storage is a few percent, because the monthly snapshots share almost all their blocks.

What if the offsite copy is encrypted and the person with the passphrase has left?

This is a real and unglamorous way to lose everything, and it happens more often than a compromised bucket.

restic and Borg encrypt client-side. Without the passphrase the repository is mathematically inaccessible: there is no support ticket, no vendor recovery, no brute force. Same for a borg init --encryption=repokey repository whose passphrase lived in one engineer’s password manager.

Treat the passphrase as a distinct asset with its own recovery plan:

Store it where the backup system is not. A passphrase in the same secret manager that the backup job reads at runtime is not escrowed; it is co-located.
At least two people, or a documented break-glass. A sealed envelope in a safe is unfashionable and works.
Test the escrow, not just the backup. Once a year, have somebody who does not normally run backups retrieve the passphrase by the documented route and perform a restore. That drill tests the human path, which is the part that decays.
Borg specifically: borg key export writes the repository key to a file you can escrow. In repokey mode the key is inside the repository, so the passphrase is the whole secret; in keyfile mode the key lives on the client and losing the client loses the archives regardless of the passphrase. Know which mode you are in before you need to.

The test that catches this is not a backup test. It is asking a colleague who has never run the restore to run it, using only the documentation, while you say nothing.