Borg append-only: what it stops, and recovering when it does not

Objective

Append-only is the control teams reach for when the threat is a compromised backup client, and the sentence usually attached to it is wrong in the way that matters at 03:00. This lab reproduces the misconception on a real repository: append_only 1 set with borg config, three archives written under it, and then an attacker deleting every one of them with exit code 0.

The second half is what makes the control worth having anyway. You will follow the upstream rollback procedure, put all three archives back, extract one, prove the recovered order ledger against a manifest taken before the repository existed, and record the single condition under which none of that works.

Architecture

The transaction numbers are the diagram. Everything the rollback does is decided by which of them was the last good one.

flowchart TD
    A["borg init repokey-blake2\nborg config append_only 1"] --> B["day1 day2 day3 written\ntransactions 5, 9, 13"]
    B --> C["attacker deletes all three\nevery exit code 0\nborg list returns empty"]
    C --> D["transactions 17, 21, 25\ndata still 41M on disk"]
    D --> E["cp -al forensic copy\nremove hints index integrity\nremove segments above 13"]
    E --> F["borg list refused\ncache newer than repository"]
    F --> G["clear cache and security directory"]
    G --> H["day1 day2 day3 return\nextract exit 0, ledger intact"]

Read the edge from C to D. The archives are gone from the listing while the data is still on disk, because append-only forbids the compaction that would have reclaimed it. That gap is the entire recovery window.

Read the edge from A to B too. The flag goes on before the first archive, not after it, and Task 3 explains why that ordering is load-bearing rather than tidy.

Requirements

• A disposable host and a normal user account. Everything created is prefixed rbdr-, so Cleanup can be scoped and asserted. No root is required; the repository lives under $HOME.
• borg on PATH. The transcripts quoted below were captured on:

$ borg --version

borg 1.4.0

• That capture recorded 1.4.x as the release line under test because Borg 2.0 was still in beta when it ran, so 1.4 is the production line here.
• Roughly 200 MB of free disk. The captured figures come from one run on one machine; your fingerprints, timestamps and segment numbers will differ, and the exit codes will not.

Scenario

A backup account’s credentials have leaked. The attacker reaches the repository exactly the way the nightly job does, and that is enough to run borg delete. What the attacker does not have is filesystem access to the repository server, which is the asymmetry append-only actually buys: the delete happens, and someone standing on the repository host can undo it. You are that someone, and the clock is whether a maintenance job reclaims space before you get there.

Tasks

Task 1 — Record the pre-lab state, then build the source tree and its manifest

Cleanup compares against this file. Record it before anything exists.

LAB="$HOME/rbdr-lab-13"
SRC="$LAB/rbdr-src"
REST="$LAB/rbdr-restore"
REPO="$LAB/rbdr-aorepo"
export BORG_PASSPHRASE='rbdr-lab-13-throwaway'
mkdir -p "$LAB"

{
  date -Is
  ls -d "$LAB"/rbdr-* 2>&1
} | tee "$LAB/pre-state.txt"

Now build the tree and record its checksums.

mkdir -p "$SRC/app" "$SRC/db"
printf '%s\n' 'ORDER-1001,4500.00' 'ORDER-1002,1250.00' 'ORDER-1003,880.00' > "$SRC/app/orders.csv"
dd if=/dev/urandom of="$SRC/db/data.bin" bs=1M count=40 status=none
( cd "$SRC" && md5sum ./app/orders.csv ./db/data.bin ) > "$LAB/src.md5"
cat "$LAB/src.md5"

$ md5sum the two files from inside the source tree

--- source tree ---
9eb4e2ad8e08e1dcaaf87ababab964b0  ./app/orders.csv
707ce8e1148e10fd35bc49c97e54543f  ./db/data.bin

The manifest uses ./-relative paths so it stays runnable from inside a restored copy at a different path in Task 7. Your digests are the property you will prove in Task 7; they only have to be stable, not equal to anyone else’s.

Task 2 — Initialise the repository

borg init --encryption=repokey-blake2 "$REPO"
echo "init exit code: $?"

$ borg init --encryption=repokey-blake2

  
IMPORTANT: you will need both KEY AND PASSPHRASE to access this repo!

Key storage location depends on the mode:
- repokey modes: key is stored in the repository directory.
- keyfile modes: key is stored in the home directory of this user.

For any mode, you should:
1. Export the borg key and store the result at a safe place:
   borg key export           REPOSITORY encrypted-key-backup
   borg key export --paper   REPOSITORY encrypted-key-backup.txt
   borg key export --qr-html REPOSITORY encrypted-key-backup.html
2. Write down the borg key passphrase and store it at safe place.
>>> exit code: 0

With repokey, the key material sits in the repository directory and the passphrase is what protects it. BORG_PASSPHRASE in Task 1 is acceptable only because this repository is a throwaway fixture; a production passphrase must be escrowed away from the host it protects.

Task 3 — Append-only first, then three archives

The order of these two operations decides whether the rest of the lab is possible, so do not swap them.

borg config "$REPO" append_only 1
borg config "$REPO" append_only

for DAY in day1 day2 day3; do
  borg create "$REPO::$DAY" "$SRC"
  printf 'created %s: exit code %s\n' "$DAY" "$?"
done

borg list "$REPO"
tee "$LAB/transactions-before.txt" < "$REPO/transactions"

$ borg config append_only 1, then read the value back

$ borg config /work/aorepo append_only 1     -> append_only = 1

$ borg list, then read the repository transactions file

--- archives before the attack ---
day1                                 Fri, 2026-08-28 13:58:09 [edf6f20cb15b5a2ed56ed0a8f4abe6e02e4d6a1e305729d1febb01608482195f]
day2                                 Fri, 2026-08-28 13:58:09 [dc55669922a0cbf88e0c74390976f69dd69c7ae81f90d3ea22b977c4b24c35e7]
day3                                 Fri, 2026-08-28 13:58:09 [06305e70a6a1ea7dc8c56e68eae5290ad7c52a4f22b559e8c46ac03b6398ad6a]

--- the transaction log append-only maintains ---
transaction 5, UTC time 2026-08-28T13:58:09.445859
transaction 9, UTC time 2026-08-28T13:58:09.662365
transaction 13, UTC time 2026-08-28T13:58:09.863420

Three archives, three ledger entries. The capture names that file exactly as it behaves — “the transaction log append-only maintains” — and that is why the flag goes on first. A repository switched to append-only after its archives were written starts its ledger at the switch, and the commits you would want to roll back to were never recorded. There is then nothing to aim at, and the rest of this lab has no target.

The last entry — 13 in the capture — is the last good transaction, and it is the only value the rollback in Task 5 depends on. Read it from your own file, not from this page.

Task 4 — The attacker deletes everything, and every delete succeeds

for DAY in day1 day2 day3; do
  borg delete "$REPO::$DAY"
  printf 'delete %s: exit code %s\n' "$DAY" "$?"
done | tee "$LAB/attack-log.txt"
borg list "$REPO" | tee -a "$LAB/attack-log.txt"
du -sh "$REPO/data"
tee "$LAB/transactions-after.txt" < "$REPO/transactions"

$ three borg delete runs, then borg list, du on the data directory, and the transactions file

THE COMMON BELIEF: append-only makes delete fail. Test it.
$ borg delete /work/aorepo::day1

>>> exit code: 0
$ borg delete /work/aorepo::day2
>>> exit code: 0
$ borg delete /work/aorepo::day3
>>> exit code: 0

--- what the attacker now sees ---
(empty)

The deletes SUCCEEDED. The upstream documentation is explicit about this:
"Please note that this only affects the low level structure of the
 repository, and running borg delete or borg prune or reading from
 the repository will still be allowed."

Append-only forbids COMPACTION. The segments are still on disk:
repository data still occupying: 41M	/work/aorepo/data

--- the transaction log after the attack ---
transaction 5, UTC time 2026-08-28T13:58:09.445859
transaction 9, UTC time 2026-08-28T13:58:09.662365
transaction 13, UTC time 2026-08-28T13:58:09.863420
transaction 17, UTC time 2026-08-28T13:58:10.262693
transaction 21, UTC time 2026-08-28T13:58:10.480675
transaction 25, UTC time 2026-08-28T13:58:10.686367

Three facts to hold together. The listing is empty. The data directory still holds 41M. The ledger grew from three entries to six, and every deletion is recorded in it rather than erasing what came before. Append-only does not refuse the delete; it forbids compaction, which is what leaves the bytes addressable and the history replayable.

Task 5 — Roll the repository back to the last good transaction

Take the forensic copy first. cp -al hard-links rather than duplicating, so the compromised state is preserved for roughly no extra disk.

GOOD=$(awk 'END {print $2}' "$LAB/transactions-before.txt" | tr -d ',')
CUR=$(awk 'END {print $2}' "$LAB/transactions-after.txt" | tr -d ',')
printf 'last good transaction: %s\ncurrent transaction: %s\n' "$GOOD" "$CUR" | tee "$LAB/rollback-log.txt"

cp -al "$REPO" "$LAB/rbdr-aorepo-evidence"
rm -f "$REPO"/hints.* "$REPO"/index.* "$REPO"/integrity.*
find "$REPO/data" -type f | while read -r SEG; do
  NUM=$(basename "$SEG")
  if [ "$NUM" -gt "$GOOD" ]; then
    rm -f "$SEG"
    echo "removed segment $NUM"
  fi
done | tee -a "$LAB/rollback-log.txt"

$ cp -al the repository, remove hints/index/integrity, remove segments above the last good transaction

  last good transaction (3rd line, after day1..day3 were written): 13
current transaction                                            : 25

$ cp -al /work/aorepo /work/aorepo-evidence     (preserve the compromised state)
hard-link copy taken for forensics

$ rm -f /work/aorepo/hints.* /work/aorepo/index.* /work/aorepo/integrity.*
$ rm segment files numbered above $GOOD
removed segment 14
removed segment 15
removed segment 16
removed segment 17
removed segment 18
removed segment 19
removed segment 20
removed segment 21
removed segment 22
removed segment 23
removed segment 24
removed segment 25

The capture reads the last good transaction off the third line, because three archives had been written; the awk 'END' above reads the last line of the pre-attack ledger, which is the same value and stays correct if you write a different number of archives.

The hints, index and integrity files describe the repository as the attacker left it. The upstream rollback procedure removes them rather than trying to repair them. Everything numbered above the last good transaction is the attacker’s work.

Task 6 — The failing case: the cache refuses the rolled-back repository

Read the repository now, before touching anything else. This step is supposed to fail.

borg list "$REPO"
echo "borg list exit code: $?"

$ borg list against the rolled-back repository

--- first attempt to read the rolled-back repository ---
Cache, or information obtained from the security directory is newer than repository - this is either an attack or unsafe (multiple repos with same ID)

The client still remembers the post-attack repository, which is newer than what is now on disk. From the client’s point of view that is indistinguishable from a rollback attack, so it refuses. Clearing the two state directories is the documented follow-up.

rm -rf "$HOME/.cache/borg" "$HOME/.config/borg/security"
borg list "$REPO" | tee -a "$LAB/rollback-log.txt"
echo "borg list exit code after clearing state: ${PIPESTATUS[0]}"

$ borg list after clearing the cache and the security directory

--- archives after the rollback ---
day1                                 Fri, 2026-08-28 13:58:09 [edf6f20cb15b5a2ed56ed0a8f4abe6e02e4d6a1e305729d1febb01608482195f]
day2                                 Fri, 2026-08-28 13:58:09 [dc55669922a0cbf88e0c74390976f69dd69c7ae81f90d3ea22b977c4b24c35e7]
day3                                 Fri, 2026-08-28 13:58:09 [06305e70a6a1ea7dc8c56e68eae5290ad7c52a4f22b559e8c46ac03b6398ad6a]

Compare the fingerprints against Task 3. They are the same archives, not recreated ones.

Task 7 — Extract the recovered archive and prove the ledger

A listing is not a restore. Extract and check the bytes.

T0=$(date +%s)
rm -rf "$REST"
mkdir -p "$REST"
( cd "$REST" && borg extract "$REPO::day3" )
echo "extract exit code: $?"
RESTORED="$REST$SRC"
( cd "$RESTORED" && md5sum -c "$LAB/src.md5" ) | tee "$LAB/restore-verify.txt"
echo "md5sum -c exit code: ${PIPESTATUS[0]}" | tee -a "$LAB/restore-verify.txt"
T1=$(date +%s)
printf 'Actual restore time: %s seconds\n' "$((T1 - T0))" | tee -a "$LAB/restore-verify.txt"
printf 'Actual RPO observed: 0 recovery points lost - day1, day2 and day3 all returned\n' \
  | tee -a "$LAB/restore-verify.txt"
tee -a "$LAB/restore-verify.txt" < "$RESTORED/app/orders.csv"

$ borg extract from the rolled-back repository, then read the recovered ledger

--- and can the recovered archive still be extracted? ---
>>> extract exit code: 0
recovered orders.csv:
  ORDER-1001,4500.00
  ORDER-1002,1250.00
  ORDER-1003,880.00

$ md5sum -c from inside the extracted tree

--- verifying the restored tree against the source checksums ---
./app/orders.csv: OK
./db/data.bin: OK
>>> verification exit code: 0

Three order lines and two OK lines is the claim this lab exists to produce. The archives an attacker deleted with exit code 0 came back with their content matching a property recorded before the repository was created.

Validation

Each row names the command, the string to expect and the exit code.

test -s "$LAB/transactions-before.txt"; echo "ledger recorded: $?"
grep -c 'exit code 0' "$LAB/attack-log.txt"
wc -l < "$LAB/transactions-before.txt"
wc -l < "$LAB/transactions-after.txt"
borg config "$REPO" append_only
borg list "$REPO" | wc -l
grep -c ': OK$' "$LAB/restore-verify.txt"
grep -c '^Actual restore time:' "$LAB/restore-verify.txt"
grep -c '^Actual RPO observed:' "$LAB/restore-verify.txt"

The fifth row is the failing case and it is supposed to fail. A borg list that succeeds immediately after Task 5 means the segment removal did not happen, so nothing was rolled back and there is nothing for the cache to disagree with.

The first row is the other check with teeth. An empty or missing transactions-before.txt means append-only was not on while the archives were written, and everything from Task 5 onwards would be aimed at a number that was never recorded.

Expected Outcome

Three archives existed, an attacker deleted all three with exit code 0, the repository was rolled back to the last good transaction, and all three archives returned and extracted with their content matching the Task 1 manifest.

Both numbers below are written into restore-verify.txt by Task 7:

• Actual restore time: T1 - T0 from Task 7, covering the extract and the checksum verification together, since verification is part of the recovery. Time the manual rollback in Tasks 5 and 6 separately in your own notes; it dominates the total and it is the part that does not scale with data size.
• Actual RPO observed: zero for this incident. Every archive that existed before the attack came back, so no recovery point was lost. That holds only because compaction had not run; had it run, the observed RPO would be the entire retained history.

Troubleshooting

Cleanup

rm -rf "$REST" "$REPO" "$LAB/rbdr-aorepo-evidence" "$SRC"
unset BORG_PASSPHRASE

{
  date -Is
  ls -d "$LAB"/rbdr-* 2>&1
} | tee "$LAB/post-state.txt"

diff <(tail -n +2 "$LAB/pre-state.txt") <(tail -n +2 "$LAB/post-state.txt") \
  && echo "CLEAN: post-state matches the pre-state recorded in Task 1"

src.md5, transactions-before.txt, transactions-after.txt, attack-log.txt, rollback-log.txt and restore-verify.txt are deliverables and are deliberately kept. Note that Task 6 removed ~/.cache/borg and ~/.config/borg/security; if you use borg for real work on this host, run the whole lab somewhere else.

Production notes

• The rollback is only possible while borg compact has not run. Compaction is the operation append-only forbids, and it is what actually reclaims the segments; once it has run, the segments the rollback needs are gone and the deletion is final. Any scheduled maintenance that compacts the repository is, in this scenario, the attacker’s cleanup crew.
• Turn append-only on when the repository is created, not when someone gets nervous. The ledger only covers transactions committed while the flag was set, so a repository switched over later can only be rolled back to the switch.
• Append-only is worth deploying for one reason: it converts a delete into something a person with filesystem access to the repository server can undo. So the repository host must be administered by an identity the backup client does not hold. If the same credentials reach both, the control buys nothing.
• Alert on archive count and ledger growth, not on delete failures. A delete that fails is not the signal this control produces.
• Take the cp -al copy before the first removal, every time. It costs almost no space, and it is all that stands between a mistyped segment number and a repository that is genuinely unrecoverable.
• Rehearse the rollback before you need it. It is manual and multi-step, with an intermediate state that looks exactly like a failure, and nobody should meet the cache refusal for the first time during an incident.

What You Learned

• Append-only does not refuse the delete. Three borg delete runs returned exit code 0 and borg list came back empty. The upstream documentation says so plainly: delete, prune and reads are still allowed.
• What it forbids is compaction, and that is the protection. The data directory still held 41M after every archive had disappeared from the listing, which is what made the recovery possible at all.
• The transaction ledger is the recovery plan, and it starts when the flag does. Three archives written under append-only produced three entries; the attack added three more. The last pre-attack entry is the number the entire rollback is aimed at.
• The cache refusal is a correct result, not a fault. A client whose state is newer than the repository cannot tell a rollback from an attack, and saying so is the safe behaviour.
• A returned listing is not a restore. The lab is finished when an archive extracts and its content matches a manifest recorded before the repository existed.