Secrets at rest - ownership, modes and the copies you forgot

The previous lesson named the leak paths. This one is about the file itself: a credential sitting on a disk, and the short list of things that decide who can read it.

The list is shorter than people expect. It is the mode, the owner, the group, every directory above it, and every copy that was made without anyone deciding to make one. Four of those five are checkable in a single command. The fifth is where the incidents come from.

The target state

A credential file has exactly one reader. Write that down first, because it drives everything else:

sudo install -o myapp -g myapp -m 0700 -d /run/secrets/myapp
sudo stat -c '%a %U %G %n' /run/secrets/myapp

install -d creates the directory with the mode you asked for, in one operation. mkdir then chmod is two, and between them the directory is whatever your umask produced.

The directory matters as much as the file

A 0400 file is unreadable to everyone but its owner - and that is true regardless of the directory. What the directory controls is different and easier to get wrong: the execute bit on a directory grants traversal, and the read bit grants listing.

So a world-listable directory does not leak the contents of the file. It leaks the fact that /etc/myapp/prod-db.key exists, which is most of what an attacker needs to know where to spend effort. And if any directory in the path is group-writable by a broad group, someone in that group can replace the directory entry entirely.

namei -l walks the whole path and prints every component:

$ namei -l /etc/myapp/secrets/db.conf

f: /etc/myapp/secrets/db.conf
drwxr-xr-x root  root  /
drwxr-xr-x root  root  etc
drwxr-xr-x root  root  myapp
drwxrwxr-x root  devs  secrets
-rw-------  myapp myapp db.conf

db.conf is mode 0600 and looks correct in any audit that checks files. The directory above it is group-writable by devs, so any member of devs can rename db.conf out of the way and put their own file there - which the service will then read at its next restart. Auditing files without auditing their path is auditing half the problem.

The window that redirection opens

This is the trap that survives code review, because the finished state is correct:

# Wrong: there is a window where the file is world-readable
umask 022
printf '%s\n' "$PW" > /etc/myapp/db.pw
chmod 600 /etc/myapp/db.pw

The shell creates the file before the command that writes to it runs, and it creates it with mode 0666 masked by the umask - 0644 under a default umask. The secret is written into a world-readable file, and only afterwards is the mode corrected. On a busy multi-user host or a CI runner the window is milliseconds and it is not zero. Worse, if the chmod line fails or the script exits between the two, the file stays world-readable forever and nothing reports it.

Two correct forms:

# Create the file empty at the right mode, then write into it
sudo install -o myapp -g myapp -m 0400 /dev/null /etc/myapp/db.pw
printf '%s\n' "$PW" | sudo tee /etc/myapp/db.pw >/dev/null

# Or set the umask for the whole scope, in a subshell so it does not leak
( umask 077; printf '%s\n' "$PW" > /etc/myapp/db.pw )

The second form gives 0600; use the first when you want 0400 or a different owner. Note that umask can only remove bits, so umask 077 never produces a file more open than 0600 - it cannot give you 0400 on its own.

printf is a shell builtin, so printf '%s\n' "$PW" never forks a process and the value never appears in ps or in a /proc/PID/cmdline. echo is a builtin too. That is a real difference from /bin/echo, /usr/bin/printf, or any external command you hand the value to as an argument.

Auditing a host

Two questions, two commands. First, is any credential material readable by group or world:

sudo find /etc /srv /opt -xdev -type f \
     \( -name '*.key' -o -name '*.pem' -o -name '*.pw' -o -name '*credential*' \) \
     -perm /g=r,o=r -printf '%M %u:%g %p\n'

-perm /g=r,o=r matches a file with any of those bits set (the leading slash means “any of these”, as opposed to - which means “all of these”). -xdev keeps the search on one filesystem so it does not wander into a network mount or /proc.

Second, do the well-known credential files have the modes their own tools demand:

stat -c '%a %U %G %n' ~/.ssh/id_ed25519 ~/.pgpass ~/.my.cnf ~/.netrc 2>/dev/null
sudo stat -c '%a %U %G %n' /etc/shadow /etc/ssl/private

Some tools enforce this and some do not, and the difference is worth knowing:

• OpenSSH refuses a private key that is group- or world-readable: Permissions 0644 for '/home/u/.ssh/id_ed25519' are too open.
• PostgreSQL refuses a ~/.pgpass that is not 0600 or stricter, with WARNING: password file "/home/u/.pgpass" has group or world access.
• curl does not check ~/.netrc at all.
• MySQL warns about a world-readable ~/.my.cnf and keeps going.

So “the tool would have complained” is true for two of these four and false for the rest. Check the mode yourself.

The copies nobody made on purpose

The original file is the easy part. These are the copies that outlive it:

Backups. A backup of /etc contains /etc/ssl/private. If the backup is not encrypted, every private key on the host is now in the backup store, at the backup job’s permissions, retained for the backup retention period - which is usually far longer than the credential rotation period. This is the single most common way a rotated secret remains live.

Core dumps. A process holding a secret in memory that segfaults writes that memory to a core file. With systemd-coredump that lands under /var/lib/systemd/coredump, compressed but not encrypted.

coredumpctl list --no-pager | tail -5

For a service that holds credentials, disable it in the unit:

[Service]
LimitCORE=0

Swap. Memory holding a secret can be paged out to swap, which is a plain block device unless you encrypted it. Swap survives nothing but a reboot, but “nothing but a reboot” is weeks on a server. Encrypted swap, or no swap on hosts that hold key material.

Editor artefacts. vim writes .db.conf.swp while you edit and can leave it behind after a crash; a configured backup produces db.conf~. Both are created with your umask, not the mode of the file you were editing:

sudo find /etc -name '.*.sw[a-p]' -o -name '*~' | head

Old versions in the same directory. db.conf.bak, db.conf.20260401, db.conf.orig. They hold superseded credentials, which are still live if nobody rotated them.

/run is the right place

/run is a tmpfs. Nothing written there reaches a block device, it is cleared at boot, and it cannot appear in a filesystem backup or a snapshot:

findmnt -no FSTYPE,OPTIONS /run

For a systemd service, RuntimeDirectory= creates and owns one for you, with the mode you specify, and removes it when the service stops:

[Service]
User=myapp
RuntimeDirectory=myapp
RuntimeDirectoryMode=0700

That gives /run/myapp, mode 0700, owned by myapp, cleaned up on stop. It is the least effort of any option in this lesson and it removes three of the four secondary-copy problems above.

Knowledge check