Archive semantics: permissions, ACLs, xattrs, sparse files and hard links

A mirror that propagates a deletion, the failure the previous lesson measured, at least leaves visible evidence behind: the file is gone, and its absence can be counted. This lesson is about the opposite kind of failure — a restore that leaves nothing visibly wrong. Every path is present, every size matches, and a checksum comparison of file content passes on every file in the tree, while the restored system does not work. What went missing was not bytes. It was the metadata that decides who may read a file, what a binary is permitted to do, how much disk a file consumes, and whether two paths are the same file.

Six pieces of metadata, and what each one decides

The capture behind this lesson used a single small source tree, built so that every class of metadata a production tree actually carries was present on some file in it, and then restored that tree four different ways. Nothing in the tree was large; the point was never volume.

app.conf carried a POSIX ACL with one entry. An ACL is how a file grants access to a principal that the owner/group/other triple has no room for — a monitoring account permitted to read a configuration file, a deployment account permitted to write one — and it is the mechanism that makes the true answer to “who can read this” different from what ls -l displays.

index.dat carried an extended attribute in the user namespace, user.checksum, holding the string sha256:deadbeef. Extended attributes are name/value pairs attached to an inode rather than to its contents, and applications use them for precisely this sort of thing: a content hash, an origin marker, an indexing state, a Samba DOS attribute, a security label.

netcheck carried the file capability cap_net_raw=ep. A file capability is the modern, narrow alternative to making a binary setuid root: it grants one specific privilege to one executable — here the ability to open raw sockets — without granting the rest of what root can do. The capability is stored on the file and consulted at exec, so a copy of that binary without it is a binary that can no longer do its job, no matter how identical its bytes are.

admin-tool carried a setuid bit: the classic mode-bit privilege that every reviewer already knows to look for, included here specifically so that the capture would show what a naive copy does preserve.

sparse.img had an apparent size of 200M and 0 blocks allocated. A sparse file is one the kernel reports as long while backing almost none of it with real blocks, because the unwritten regions are holes rather than stored zeros. Disk images, database preallocations, VM volumes and reserved log files are routinely sparse, and nothing bounds how far the two sizes can diverge: in this tree the file is two hundred mebibytes long and occupies no blocks at all.

Finally, payload.a and payload.b were two directory entries pointing at one inode: link count 2, same inode number. A hard link is how a filesystem expresses that two names are one file, so that a write through either name is visible through the other and the storage is charged once.

Not one of those six is visible to a comparison of file contents. The capture makes that point explicitly in its own summary: every restore below produced files with the right names, the right sizes and the right contents, so a checksum comparison of file content would have passed for all four.

Measured: what the default flags dropped

The two commands most likely to be found in a hand-written backup script are tar -cf and rsync -a. Both were run against this tree, with the flags people usually type, and the restored trees were then inspected attribute by attribute.

$ tar -cf naive.tar src && tar -xf naive.tar -C r1

  archive size: 201M
[restored from default tar]
  ACL on app.conf        : 0 entr(y|ies)
  xattr on index.dat     : ABSENT

  capability on netcheck : ABSENT
  setuid bit on admin-tool: present
  sparse.img apparent    : 200M
  sparse.img allocated   : 200M
  payload.a link count   : 2
  payload a/b same inode : yes

Three attributes went missing without a warning, an error, or a non-zero exit: the ACL that was one entry became zero entries, the extended attribute became ABSENT, and the capability became ABSENT. The setuid bit survived, because mode bits are part of the inode that tar already reads for every member it records. The hard link survived too — tar tracks device and inode numbers as it walks and records the second occurrence as a link rather than as a second copy.

The sparse file did not survive as a sparse file. It came back with an apparent size of 200M and 200M allocated, which is to say the holes were read as zeros, written into the archive as zeros, and restored as two hundred mebibytes of real, allocated, zero-filled blocks. That is also why the archive itself is 201M for a source tree whose actual content is a few kilobytes.

$ rsync -a src/ r3/

  [restored from rsync -a]
  ACL on app.conf        : 0 entr(y|ies)
  xattr on index.dat     : ABSENT

  capability on netcheck : ABSENT
  setuid bit on admin-tool: present
  sparse.img apparent    : 200M
  sparse.img allocated   : 200M
  payload.a link count   : 1
  payload a/b same inode : NO - now two separate files

rsync -a lost everything the default tar lost — ACL, extended attribute, capability, sparseness — and then lost one more thing that tar had kept. The payload a/b same inode line reads NO - now two separate files. Two names that were one file are now two files with identical contents and independent inodes. A write through one of them will no longer be visible through the other, and the destination is charged twice for the storage.

That last difference is worth dwelling on, because it is a case where the tool with the friendlier reputation did the more surprising thing. The letter a in rsync -a stands for archive, and it is documented in rsync(1) as a shorthand for a specific list of options — recursion, symlinks, permissions, times, group, owner and device/special files. It has never stood for “everything about a file”, and the four classes of metadata measured above are each requested by a separate flag.

The flag sets that carried everything, and a 100 KiB archive

The same source tree, archived by the same version of tar, with the metadata flags supplied:

$ tar --acls --xattrs --xattrs-include='*' --sparse -cf full.tar src && tar --acls --xattrs --xattrs-include='*' -xf full.tar -C r2

  archive size: 100K
[restored from tar --acls --xattrs --sparse]
  ACL on app.conf        : 1 entr(y|ies)
  xattr on index.dat     : sha256:deadbeef
  capability on netcheck : cap_net_raw=ep
  setuid bit on admin-tool: present
  sparse.img apparent    : 200M
  sparse.img allocated   : 0
  payload.a link count   : 2
  payload a/b same inode : yes

Every attribute came back. The ACL is one entry again, user.checksum reads sha256:deadbeef, netcheck carries cap_net_raw=ep, and sparse.img is once more 200 MiB long with 0 blocks allocated. The archive is 100K instead of 201M — a difference of more than three orders of magnitude, from the same input, produced by a flag rather than by a compressor.

Two details in that command repay attention. The include mask was written explicitly as --xattrs-include='*' rather than left to a default, and the two attributes that had to survive lived in different namespaces: the capability in security. and the application’s hash in user.. An include mask narrower than the set you actually need carries some of your attributes and not others, which is a worse outcome than carrying none, because it is harder to notice. The second detail is that --acls --xattrs --xattrs-include='*' appear on the extraction command too. An archive that contains the metadata still restores without it if the reader is not told to apply it.

$ rsync -aAXH --sparse src/ r4/

  [restored from rsync -aAXH --sparse]
  ACL on app.conf        : 1 entr(y|ies)
  xattr on index.dat     : sha256:deadbeef
  capability on netcheck : cap_net_raw=ep
  setuid bit on admin-tool: present
  sparse.img apparent    : 200M
  sparse.img allocated   : 0
  payload.a link count   : 2
  payload a/b same inode : yes

-A requests ACLs, -X requests extended attributes, -H requests hard-link preservation and --sparse requests that holes be reproduced as holes. With those four added to -a, the restored tree matches the source on all six attributes, including the link count of 2 and the shared inode that plain -a had broken.

The operational reading is not that these are the correct flags to memorise. It is that fidelity is a property of the command, not of the tool, and that the difference between a faithful copy and a lossy one is invisible in the output of both. Neither lossy run printed a warning. Both exited normally.

Two failures that do not present as restore failures

The reason this material belongs in a recovery course rather than a filesystem course is that the resulting incidents are misdiagnosed, sometimes for days.

The first is the restored binary. netcheck came back from two of the four restores byte-identical and without cap_net_raw=ep. Its checksum matches, it is present, it is executable, and its mode bits are right. When the service runs it as an unprivileged user it fails on the operation that needed the privilege — opening a raw socket — and what the operator sees is an application-level error from the application’s own code path. The investigation goes to the application team, then to the network, then to the kernel, and the restore is never a suspect, because the restore reported success and the file compares equal. The getcap output that would settle it in one second is not part of anyone’s checklist.

The second is the archive that is a thousand times too big. 201M from a tree whose real content is kilobytes is not a subtle signal; it is a screaming one, if anybody looks. Sparse mishandling is the one failure in this lesson that announces itself for free, in the size of the artefact, before any restore is attempted. A backup of a host with 4 TiB of sparse VM images that produces 4 TiB of archive, night after night, is telling you exactly what the restore will do to the target filesystem — and the target filesystem is usually smaller than the source’s apparent size, so the restore fails on space in the middle, long after the point where anything can be done about it quickly.

Verifying fidelity instead of assuming it

The check that matters is not “did the files arrive” but “did the attributes arrive”. It is cheap, it is scriptable, and it belongs in the same job that performs a restore test.

Start from the copy itself, with the flags stated rather than implied:

SRC=/srv/app
ARCHIVE=/backup/app-$(date +%F).tar

tar --acls --xattrs --xattrs-include='*' --sparse -cf "$ARCHIVE" "$SRC"
tar --acls --xattrs --xattrs-include='*' -xf "$ARCHIVE" -C /restore

Then compare the metadata, not the contents. find can print mode, link count, apparent size and allocated blocks in one pass, which covers sparseness and hard links; the attribute tools cover the rest:

SRC=/srv/app
DEST=/restore/srv/app

fingerprint() {
  find "$1" -printf '%P\t%m\t%n\t%s\t%b\n' | sort
}
acls() { ( cd "$1" && getfacl -R -n . ) | sort; }
caps() { ( cd "$1" && getcap -r . ) | sort; }

diff <(fingerprint "$SRC") <(fingerprint "$DEST")
diff <(acls "$SRC") <(acls "$DEST")
diff <(caps "$SRC") <(caps "$DEST")

The %n field is the link count and %b is the number of allocated blocks, so a broken hard link and an expanded sparse file both appear as ordinary diff lines. The two attribute comparisons run from inside each tree on purpose: getfacl and getcap echo the path they were handed, so comparing /srv/app against /restore/srv/app directly reports every single line as changed and the one line that matters is lost in it. Run the whole thing against the restore-test target rather than against production, and run it on the tree the service will actually use, because the attributes that matter are the ones on the paths the service opens.

What to take from this

• Default tar -cf produced a 201M archive from a tree whose real content was kilobytes, and restored sparse.img with 200M allocated where the source had 0. Archive size is the cheapest sparse-mishandling detector available, and it works before any restore is attempted.
• Default tar and rsync -a both restored app.conf with 0 ACL entries against a source of 1, index.dat with its user.checksum attribute ABSENT against sha256:deadbeef, and netcheck with its capability ABSENT against cap_net_raw=ep.
• rsync -a additionally reported payload a/b same inode : NO - now two separate files and a link count of 1, where default tar kept the link count at 2. The friendlier tool lost more, not less.
• The setuid bit on admin-tool was present in all four restores. The metadata everyone inspects survived every command; the metadata nobody inspects survived only two of them.
• tar --acls --xattrs --xattrs-include='*' --sparse produced a 100K archive and returned all six attributes, and rsync -aAXH --sparse returned all six as well. Both flag sets have to be asked for, and the tar extraction needs --acls --xattrs too.
• All four restored trees came back with the right names, the right sizes and the right contents, so a content comparison would have passed on every one. “The files are identical” is a true statement about two of those trees that would not run the service they came from.

Cross-course references

• Linux for Production Sysadmins — Part XXX (Linux Capabilities and Privilege) explains what cap_net_raw=ep actually grants and how security.capability is consulted at exec, which is the material that turns the capability loss measured here from an unexplained runtime error into a one-command diagnosis; Part III (Filesystems and Files) covers the inode, link-count and allocated-block fields the fingerprint comparison above reads.
• Docker & Containers for Production Sysadmins — Part XXI (Backup) covers volume backups, which are conventionally taken by running tar inside a helper container against a mounted volume. That is exactly the command measured in this lesson, so the flag set chosen there decides whether a restored volume still carries the ACLs, attributes, capabilities and sparse layout the workload depends on.
• Secrets, PKI & Certificate Management for Infrastructure Engineers — Part XI (SSH Keys, Host Trust and SSH CAs) establishes that sshd refuses to read an authorized_keys file whose ownership or directory permissions are wrong, no matter what the file contains. Restoring SSH state is therefore a metadata-fidelity problem before it is a cryptographic one, and it fails in exactly the silent way measured here: the bytes compare equal and the login is rejected anyway.

Quiz