Immutable and offline copies - the last line of defence · Linux

Immutable backups cannot be modified or deleted for a retention period. Air-gapped backups are physically unreachable from the network. These are two different properties, and this lesson keeps them apart, because conflating them is how organisations end up believing they have a last-resort copy that they do not have.

Why immutability matters

In 2021, a ransomware attack on a backup vendor (approximately) demonstrated the risk: the attacker encrypted production AND backups because both were reachable from the network.

An immutable backup cannot be encrypted by an attacker who has admin on the production network. Even if the attacker gains root, they cannot delete the immutable copies.

S3 Object Lock

S3 Object Lock makes objects immutable for a retention period:

# Create a bucket with Object Lock
aws s3api create-bucket --bucket my-backup --object-lock-enabled-for-bucket

# Upload with retention
aws s3api put-object --bucket my-backup \
    --key backup-2026-08-09.tar \
    --body backup.tar \
    --object-lock-mode COMPLIANCE \
    --object-lock-retain-until-date 2026-09-09T00:00:00Z

Two modes:

GOVERNANCE: can be overridden by any principal holding s3:BypassGovernanceRetention. Against an attacker who has taken over an administrative IAM role, this is not a control at all.
COMPLIANCE: cannot be overridden by anyone, including the account root, until the retention date passes.

For ransomware protection, use COMPLIANCE mode with long retention.

What Object Lock does not give you:

It is not air-gapped. The bucket is online and reachable with credentials.
It protects the locked object versions only. Unlocked objects in the same bucket can still be deleted, and an attacker can stop future backups from being locked by changing the default retention or the upload path.
It does not protect the bill. An attacker who cannot delete can still upload petabytes.

So it is the right immutable copy, and it is not the air-gapped copy.

Tape rotation

For air-gapped tape backups:

# Backup to tape
tar -cf /dev/nst0 /var/data

# Eject the tape
mt -f /dev/nst0 eject

# Store offsite
# (Physical rotation: take the tape offsite)

Tape rotation schedule:

Daily tape: 7 tapes (one per day, rotate weekly).
Weekly tape: 4 tapes (one per week, rotate monthly).
Monthly tape: 12 tapes (one per month, rotate yearly).
Yearly tape: indefinite (compliance).

Each tape is offsite for some period. An attacker on the network cannot reach the offline tape.

Disk rotation

For SMB-scale operations:

# Mount external disk
mount /dev/sdb1 /mnt/backup

# Backup
borg create /mnt/backup/repo::backup-2026-08-09 /var/data

# Unmount
umount /mnt/backup

# Store offsite

Rotation: 5-7 external disks, rotate. One is always offsite. Less secure than tape (recoverable from network if the disk is plugged in) but more practical for small operations.

BorgBackup with append-only

BorgBackup has an append-only mode, and it is very easy to configure the version of it that protects nothing.

`borg init --append-only` on its own is not a control

# This is the form that does NOT stop an attacker
borg init --append-only /backup/repo

All this does is write append_only = 1 into the repository’s own config file, inside the repository directory. Anyone with write access to that directory can open the file in an editor, set it to 0, and prune freely - or skip the flag entirely and rm -rf the segment files.

“Write access to the repository directory” is exactly what “an attacker with admin on the backup host” means. The flag is a guard rail against your own mistakes, not against an adversary.

Append-only is a property of how the repository is served, not of the repository itself. The enforcement point is the backup server’s SSH configuration, where a forced command pins every connection from that key to borg serve --append-only. The client then physically cannot issue a destructive operation, no matter what it has been told to run.

# On the BACKUP SERVER, in ~backup/.ssh/authorized_keys.
# One line - wrapped here for readability.
command="borg serve --append-only --restrict-to-path /backup/repo",\
restrict ssh-ed25519 AAAA...client-key... backup-client

# On the CLIENT, backups work normally
borg create ssh://backup@backup-host/backup/repo::{now} /var/data

# ...and destructive operations are refused by the server
borg prune --keep-daily 7 ssh://backup@backup-host/backup/repo
# Remote: Refusing to delete/prune, running in append-only mode

Two consequences to plan for:

Pruning must happen on the backup server, out of reach of the client. Schedule it there, with its own credentials. If you never prune, the repository grows without bound.
Append-only is not undo-proof by itself. A compromised client can still append garbage. Combine it with retention on the server and with a periodic borg check run server-side.

Cost vs security

Different options have different trade-offs:

Immutable and air-gapped are separate columns, because they are separate properties:

Option	Cost	Immutable	Air-gapped	Notes
S3 Object Lock COMPLIANCE	$$	Yes	No	Online. Immutable, but reachable with cloud credentials.
S3 Object Lock GOVERNANCE	$$	Only against unprivileged callers	No	Bypassable with `s3:BypassGovernanceRetention`.
Tape, rotated offsite	$	While offline (always, if WORM media)	Yes	Physically unreachable once ejected. Restore is slow.
Disk rotation	$$	No	Yes, while unplugged	Fully mutable whenever it is mounted.
Borg served with `--append-only`	$	Yes, against the client	No	Enforced by the server’s forced command, not by the repo flag.
Local disk	$	No	No	Not a ransomware copy at all.

For ransomware-critical data the standard answer is both: S3 Object Lock in COMPLIANCE mode for a fast restore, plus an offline copy for the day the cloud account is the thing that was compromised. Picking one and calling it 3-2-1-1-0 is the mistake this lesson exists to prevent.