Archive fidelity: what default flags silently discard

Objective

A restore that returns every path, every size and every byte can still produce a system that does not work. What goes missing is the metadata that decides who may read a file, what an unprivileged binary is permitted to do, how much disk a file consumes, and whether two paths are the same file.

You build one source tree carrying six such attributes, archive and restore it four ways, and run the same report after each restore. Two of the four restores lose three attributes each, one of those two also breaks a hard link, and all four pass a content checksum comparison. You finish with a report script you can point at any restored tree, and with the flag sets that carried everything.

Architecture

One source, four paths through an archiver, one report run five times. The diagram is the result table: the two left-hand paths are the defaults most people type.

flowchart TD
    S["src\nACL, user xattr, file capability,\nsetuid, 200M sparse with 0 allocated,\npayload.a and payload.b hard linked"]
    S --> A["tar -cf"]
    S --> B["tar --acls --xattrs --sparse"]
    S --> C["rsync -a"]
    S --> D["rsync -aAXH --sparse"]
    A --> A1["201M archive\nACL, xattr, capability gone\nsparse allocates 200M\nhard link kept"]
    B --> B1["100K archive\nall six attributes kept"]
    C --> C1["ACL, xattr, capability gone\nsparse allocates 200M\nhard link broken"]
    D --> D1["all six attributes kept"]
    A1 --> Z["file CONTENT compares equal\nin all four restores"]
    B1 --> Z
    C1 --> Z
    D1 --> Z

Requirements

• Root on a disposable Debian 13 container or VM. setcap, setfacl and useradd all need it, and so does extracting a setuid file with its ownership intact.
• A block-backed filesystem. ext4, xfs or btrfs. An overlayfs upper layer or a tmpfs will refuse the security.* namespace that holds the file capability, and the lab then fails at the build step.
• Packages: tar, rsync, acl, attr, libcap2-bin, coreutils.
• About 500 MiB free. The default tar archive alone is 201 MiB.

Everything created lives under $HOME/rbdr-lab-05, plus one prefixed state file and one prefixed user account, so cleanup is a scoped removal.

Scenario

A service was rebuilt onto a clean host from an archive taken with tar -cf. Every file is present. A colleague checksummed the restored tree against a surviving copy and reported that every file matched.

The service refuses to start. Its packet-capture helper, which ran unprivileged before, now needs root. The monitoring account can no longer read the config file it has always read. The filesystem that held 40 GiB now reports 240 GiB used.

Nobody has lost a byte. Reproduce all four symptoms, and name the flag that would have prevented each.

Tasks

Task 1 - Record the pre-lab state

Cleanup is checked against this file at the end, so write it before you create anything.

RBDR_LAB="$HOME/rbdr-lab-05"
RBDR_PRE="$HOME/rbdr-lab-05-prestate.txt"

{
  echo "date: $(date -Is)"
  echo "tar: $(tar --version | head -1)"
  echo "rsync: $(rsync --version | head -1)"
  echo "lab root present: $(test -e "$RBDR_LAB" && echo yes || echo no)"
  echo "rbdr objects: $(find "$HOME" -maxdepth 1 -name 'rbdr-*' \
      ! -name 'rbdr-lab-05-prestate.txt' | wc -l)"
  echo "user rbdr-reader: $(id -u rbdr-reader >/dev/null 2>&1 && echo present || echo absent)"
  echo "free on home: $(df -Pm "$HOME" | awk 'NR==2 {print $4}') MiB"
} > "$RBDR_PRE"
cat "$RBDR_PRE"

Expect lab root present: no, rbdr objects: 0 and user rbdr-reader: absent. If any of those disagree, an earlier run was not cleaned up; finish that first.

Task 2 - Build the source tree

Six attributes, each attached to its own file so the report can name them individually.

mkdir -p "$RBDR_LAB"/{src,r1,r2,r3,r4,reports}
cd "$RBDR_LAB" || exit 1

useradd --no-create-home --shell /usr/sbin/nologin rbdr-reader

printf 'listen = 127.0.0.1:8080\n' > src/app.conf
chmod 0640 src/app.conf
setfacl -m u:rbdr-reader:r-- src/app.conf

printf 'index payload\n' > src/index.dat
setfattr -n user.checksum -v 'sha256:deadbeef' src/index.dat

cp /bin/true src/netcheck
setcap cap_net_raw=ep src/netcheck

cp /bin/true src/admin-tool
chmod 4755 src/admin-tool

truncate -s 200M src/sparse.img

printf 'shared payload\n' > src/payload.a
ln src/payload.a src/payload.b

src/app.conf is mode 0640 owned by root, so rbdr-reader can read it only through the ACL. src/netcheck holds cap_net_raw=ep in the security.capability extended attribute. src/admin-tool carries a setuid bit in its ordinary mode bits - that one is the control, because it travels a different mechanism from the other five.

Task 3 - Write the report you will run five times

The whole method is one report, run identically against the source and against each restore. A per-restore ad-hoc check is how differences get missed.

cat > "$RBDR_LAB/rbdr-report.sh" <<'REPORT'
#!/bin/bash
# Usage: rbdr-report.sh TREE LABEL
root=$1
label=$2
echo "  [$label]"
printf '    ACL on app.conf        : %s entr(y|ies)\n' \
  "$(getfacl -cp "$root/app.conf" 2>/dev/null | grep -c '^user:[^:]')"
if v=$(getfattr --only-values -n user.checksum "$root/index.dat" 2>/dev/null); then
  printf '    xattr on index.dat     : %s\n' "$v"
else
  # The blank line marks an absence, so the five reports stay aligned in a diff.
  printf '    xattr on index.dat     : ABSENT\n\n'
fi
cap=$(getcap "$root/netcheck" 2>/dev/null | awk '{print $2}')
printf '    capability on netcheck : %s\n' "${cap:-ABSENT}"
if [ -u "$root/admin-tool" ]; then s=present; else s=absent; fi
printf '    setuid bit on admin-tool: %s\n' "$s"
printf '    sparse.img apparent    : %s\n' "$(du -h --apparent-size "$root/sparse.img" | cut -f1)"
printf '    sparse.img allocated   : %s\n' "$(du -h "$root/sparse.img" | cut -f1)"
printf '    payload.a link count   : %s\n' "$(stat -c %h "$root/payload.a")"
if [ "$(stat -c %i "$root/payload.a")" = "$(stat -c %i "$root/payload.b")" ]; then
  printf '    payload a/b same inode : yes\n'
else
  printf '    payload a/b same inode : NO - now two separate files\n'
fi
REPORT
chmod +x "$RBDR_LAB/rbdr-report.sh"
./rbdr-report.sh src source | tee reports/source.txt

$ ./rbdr-report.sh src source

  [source]
  ACL on app.conf        : 1 entr(y|ies)
  xattr on index.dat     : sha256:deadbeef
  capability on netcheck : cap_net_raw=ep
  setuid bit on admin-tool: present
  sparse.img apparent    : 200M
  sparse.img allocated   : 0
  payload.a link count   : 2
  payload a/b same inode : yes

apparent 200M with allocated 0 is the sparse file: 200 MiB of addressable zeroes that occupy no blocks at all. Keep that pair in view - it is the line that turns a 40 GiB filesystem into a 240 GiB one.

Task 4 - Restore 1: the flags everyone types first

cd "$RBDR_LAB" || exit 1
tar -cf naive.tar src
tar -xf naive.tar -C r1
du -h naive.tar
./rbdr-report.sh r1/src 'restored from default tar' | tee reports/r1.txt

$ tar -cf naive.tar src
tar -xf naive.tar -C r1
./rbdr-report.sh r1/src 'restored from default tar'

  archive size: 201M
[restored from default tar]
  ACL on app.conf        : 0 entr(y|ies)
  xattr on index.dat     : ABSENT

  capability on netcheck : ABSENT
  setuid bit on admin-tool: present
  sparse.img apparent    : 200M
  sparse.img allocated   : 200M
  payload.a link count   : 2
  payload a/b same inode : yes

Three losses and one surprise. The ACL, the extended attribute and the capability are gone. The setuid bit survived, because mode bits ride in the tar header itself, and so did the hard link - tar records those without being asked. The sparse file was read as 200 MiB of zeroes and restored as 200 MiB of allocated blocks.

Task 5 - Restore 2: tar told to carry the metadata

cd "$RBDR_LAB" || exit 1
tar --acls --xattrs --xattrs-include='*' --sparse -cf full.tar src
tar --acls --xattrs --xattrs-include='*' -xf full.tar -C r2
du -h full.tar
./rbdr-report.sh r2/src 'restored from tar --acls --xattrs --sparse' | tee reports/r2.txt

$ tar --acls --xattrs --xattrs-include='*' --sparse -cf full.tar src
tar --acls --xattrs --xattrs-include='*' -xf full.tar -C r2
./rbdr-report.sh r2/src 'restored from tar --acls --xattrs --sparse'

  archive size: 100K
[restored from tar --acls --xattrs --sparse]
  ACL on app.conf        : 1 entr(y|ies)
  xattr on index.dat     : sha256:deadbeef
  capability on netcheck : cap_net_raw=ep
  setuid bit on admin-tool: present
  sparse.img apparent    : 200M
  sparse.img allocated   : 0
  payload.a link count   : 2
  payload a/b same inode : yes

Identical to the source on every line, from an archive two thousand times smaller. The flags cost nothing and were simply not typed.

--xattrs-include='*' is the part that carried the capability, which lives in security.capability, outside the user.* namespace. Consult the GNU tar manual before narrowing that glob.

Task 6 - Restore 3: rsync -a, and the attribute tar kept

cd "$RBDR_LAB" || exit 1
rsync -a src/ r3/
./rbdr-report.sh r3 'restored from rsync -a' | tee reports/r3.txt

$ rsync -a src/ r3/
./rbdr-report.sh r3 'restored from rsync -a'

  [restored from rsync -a]
  ACL on app.conf        : 0 entr(y|ies)
  xattr on index.dat     : ABSENT

  capability on netcheck : ABSENT
  setuid bit on admin-tool: present
  sparse.img apparent    : 200M
  sparse.img allocated   : 200M
  payload.a link count   : 1
  payload a/b same inode : NO - now two separate files

The last two lines are the unexpected ones. -a is “archive mode”, a name easy to read as “everything”. It does not include -H, so the two paths that shared an inode arrived as two independent files. On a tree with tens of thousands of hard links that restore is arbitrarily larger than its source, and no longer behaves the same way when one path is written to.

Task 7 - Restore 4: rsync told to carry the metadata

cd "$RBDR_LAB" || exit 1
rsync -aAXH --sparse src/ r4/
./rbdr-report.sh r4 'restored from rsync -aAXH --sparse' | tee reports/r4.txt

$ rsync -aAXH --sparse src/ r4/
./rbdr-report.sh r4 'restored from rsync -aAXH --sparse'

  [restored from rsync -aAXH --sparse]
  ACL on app.conf        : 1 entr(y|ies)
  xattr on index.dat     : sha256:deadbeef
  capability on netcheck : cap_net_raw=ep
  setuid bit on admin-tool: present
  sparse.img apparent    : 200M
  sparse.img allocated   : 0
  payload.a link count   : 2
  payload a/b same inode : yes

-A carries the ACL, -X the extended attributes including the capability, -H the hard link, --sparse the holes. Four additions, and the restore matches the source.

Task 8 - The failing case: prove the checksum check has no teeth

Hash every regular file in all five trees and compare.

cd "$RBDR_LAB" || exit 1
hash_tree() { ( cd "$1" && find . -type f -printf '%P\n' | sort | xargs -d '\n' sha256sum ); }

hash_tree src    > reports/content-source.sha256
hash_tree r1/src > reports/content-r1.sha256
hash_tree r2/src > reports/content-r2.sha256
hash_tree r3     > reports/content-r3.sha256
hash_tree r4     > reports/content-r4.sha256

for n in r1 r2 r3 r4; do
  if diff -q reports/content-source.sha256 "reports/content-$n.sha256" >/dev/null; then
    echo "$n: file content identical to source"
  else
    echo "$n: file content DIFFERS from source"
  fi
done

Four identical lines. The capture records the same conclusion:

Every restore above produced files with the right names, the right sizes and the right contents. A checksum comparison of file CONTENT would have passed for all four.

Now make one of the losses bite, using the account the ACL was granted to:

cd "$RBDR_LAB" || exit 1
sudo -u rbdr-reader cat src/app.conf  >/dev/null; echo "source  exit $?"
sudo -u rbdr-reader cat r3/app.conf   >/dev/null; echo "rsync -a exit $?"
getcap r3/netcheck; echo "getcap on r3 exit $?"

src/app.conf reads back with exit code 0, because the ACL grants it. r3/app.conf is the same bytes at mode 0640 owned by root with no ACL, so the read is refused with exit code 1. getcap r3/netcheck prints nothing: the helper that ran unprivileged before now needs root.

Validation

Run from $RBDR_LAB. Each check is a literal string match against a saved report and prints OK or FAIL; the underlying grep -Fq exits 0 on a pass and 1 on a miss. The run passes only when the last line reads validation failures: 0.

cd "$RBDR_LAB" || exit 1
fails=0
check() {
  if grep -Fq "$2" "$1"; then printf 'OK   %-18s %s\n' "$1" "$2"
  else printf 'FAIL %-18s %s\n' "$1" "$2"; fails=$((fails + 1)); fi
}

check reports/source.txt "ACL on app.conf        : 1 entr(y|ies)"
check reports/source.txt "capability on netcheck : cap_net_raw=ep"
check reports/source.txt "sparse.img allocated   : 0"
check reports/source.txt "payload.a link count   : 2"

check reports/r1.txt "ACL on app.conf        : 0 entr(y|ies)"
check reports/r1.txt "xattr on index.dat     : ABSENT"
check reports/r1.txt "capability on netcheck : ABSENT"
check reports/r1.txt "sparse.img allocated   : 200M"
check reports/r1.txt "payload.a link count   : 2"

check reports/r2.txt "ACL on app.conf        : 1 entr(y|ies)"
check reports/r2.txt "xattr on index.dat     : sha256:deadbeef"
check reports/r2.txt "capability on netcheck : cap_net_raw=ep"
check reports/r2.txt "sparse.img allocated   : 0"
check reports/r2.txt "payload a/b same inode : yes"

check reports/r3.txt "capability on netcheck : ABSENT"
check reports/r3.txt "sparse.img allocated   : 200M"
check reports/r3.txt "payload.a link count   : 1"
check reports/r3.txt "payload a/b same inode : NO - now two separate files"

check reports/r4.txt "capability on netcheck : cap_net_raw=ep"
check reports/r4.txt "sparse.img allocated   : 0"
check reports/r4.txt "payload.a link count   : 2"

echo "validation failures: $fails"

Two size checks and one setuid check, all of which must hold:

cd "$RBDR_LAB" || exit 1
du -h naive.tar full.tar
grep -c 'setuid bit on admin-tool: present' reports/source.txt reports/r1.txt \
  reports/r2.txt reports/r3.txt reports/r4.txt

du must print 201M beside naive.tar and 100K beside full.tar, and each of the five report files must contain exactly one setuid bit on admin-tool: present line - the control proving mode bits travelled everywhere while the capability did not.

Expected Outcome

All 21 metadata checks print OK and the tally reads validation failures: 0. naive.tar measures 201M against full.tar at 100K for the same tree. The four content comparisons in Task 8 all report identical, and its two reads return exit code 0 against src and exit code 1 against r3.

• Actual restore time: ______ . Time each of the four restores. The capture backing this lab recorded archive sizes, not durations, so there is no published reference figure; keep your own numbers with the reports.
• Actual RPO observed: not applicable. This lab archives a static tree once. With no schedule there is no window of unprotected writes to observe, and a figure invented here would describe the tool rather than an architecture.

The durable result: tar -cf and rsync -a both discarded the ACL, the extended attribute and the file capability; rsync -a also broke the hard link; and no comparison of file content could see any of it.

Troubleshooting

setfattr: src/index.dat: Operation not supported - the lab directory is on a filesystem that does not carry extended attributes. Move $RBDR_LAB to an ext4, xfs or btrfs path and rebuild from Task 2.

setcap: Failed to set capabilities on file 'src/netcheck' - the same class of problem narrowed to the security.* namespace, which an overlayfs upper layer and a tmpfs commonly refuse even when user.* works. Check with setfattr -n security.test -v 1 src/netcheck before blaming setcap.

setfacl: Option -m: Invalid argument near character 3 - the user named in the ACL does not exist yet. useradd rbdr-reader comes before setfacl.

The source report itself shows 0 entr(y|ies) or ABSENT - the build step failed, not the archiver. Stop and fix Task 2; every later comparison is meaningless while the source is missing the attribute under test.

r2 loses the capability although --acls --xattrs were passed - the include pattern. The capture passed --xattrs-include='*' on both the create and the extract, and it is the extract side that is most often forgotten.

r4 breaks the hard link with -H present - the two paths were transferred by two separate rsync invocations. Link detection works within one run, so both paths must be inside the same transfer.

sparse.img allocated reads 200M in the source report - truncate landed on a filesystem that does not implement holes, or something read and rewrote the file. Recreate it and confirm with du -h src/sparse.img before archiving.

Cleanup

rm -rf "$RBDR_LAB"
userdel rbdr-reader 2>/dev/null || echo "rbdr-reader already absent"

echo "rbdr objects: $(find "$HOME" -maxdepth 1 -name 'rbdr-*' \
    ! -name 'rbdr-lab-05-prestate.txt' | wc -l)"
echo "user rbdr-reader: $(id -u rbdr-reader >/dev/null 2>&1 && echo present || echo absent)"
echo "free on home: $(df -Pm "$HOME" | awk 'NR==2 {print $4}') MiB"

echo "--- recorded in Task 1 ---"
grep -E '^(rbdr objects|user rbdr-reader|free on home)' "$RBDR_PRE"
rm -f "$RBDR_PRE"

The three lines above the separator must match the three below it: zero rbdr- objects, rbdr-reader absent, and free space back within a few MiB of the Task 1 figure. A shortfall of roughly 400 MiB means naive.tar and one of the expanded sparse files are still on disk somewhere.

Production notes

• The flag sets that carried everything are tar --acls --xattrs --xattrs-include='*' --sparse and rsync -aAXH --sparse. Put them in the job, not in a wiki page, and pass the tar flags on the extract as well as the create.
• Test the restore’s metadata, not only its contents. A check built on sha256sum saw none of the four failures here; one built on getfacl, getfattr, getcap and stat -c %h sees all of them.
• Sparse files decide capacity planning. Without --sparse a host restores into several times its original footprint, and the restore that runs out of free space does so hours in, after the transfer.
• File capabilities are a security control. A helper that lost cap_net_raw=ep either stops working or gets run as root by whoever is on the bridge at 03:00, and the second outcome outlives the incident.
• Repository tools such as restic and Borg have their own metadata behaviour. Verify it the same way: restore into a scratch tree, run one report.

What You Learned

• Content equality is not restore equality. All four restores passed a checksum comparison of every file; two of them lost three attributes each.
• Defaults are the failure. tar -cf and rsync -a both discard ACLs, extended attributes and file capabilities with no warning and a zero exit.
• -a does not mean everything. It excludes -A, -X, -H and --sparse, and the missing -H turned one file with two names into two files.
• Mode bits travel free; extended attributes do not. The setuid bit survived everywhere because it lives in the inode header. The capability, an extended attribute, survived only where it was asked for.
• Sparseness is a capacity fact. 200 MiB apparent with 0 allocated became 200 MiB allocated in two restores, and produced a 201 MiB archive of a tree whose real content is a few kilobytes.
• One report, run identically after every restore, made the differences visible. Ad-hoc per-restore checks would have found none of it.