Secrets in scripts, logs and shell history

A script that never puts a password in argv, never writes it to a world-readable file, and reads it from a proper store can still hand it to every engineer in the organisation. The mechanism is not the script. It is the debugging machinery wrapped around the script, and it is switched on by someone trying to fix an unrelated problem.

Where the value comes into the script

Start with the boring part, because getting it right removes several of the later problems.

# From a file, so nothing is in argv and nothing is in history
PW=$(< /run/secrets/db.pw)

# Interactively, without echoing to the terminal
IFS= read -rs -p 'Token: ' TOKEN
echo                       # read -s swallows the newline

# From a systemd credential (the unit sets LoadCredentialEncrypted=)
PW=$(< "$CREDENTIALS_DIRECTORY/dbpw")

read -s suppresses the terminal echo, -r stops backslash processing mangling the value, and IFS= stops leading and trailing whitespace being stripped - which matters, because a token with a trailing space fails authentication in a way that looks like a wrong token.

$CREDENTIALS_DIRECTORY is set by systemd for any unit using LoadCredential= or LoadCredentialEncrypted=; it is the same directory %d expands to in the unit file.

xtrace: the leak that debugging turns on

set -x makes bash print every command after expansion. That is what makes it useful, and that is exactly the problem:

$ bash -x ./deploy.sh

+ PW=REDACTED_VALUE_APPEARS_HERE
+ curl -H 'Authorization: Bearer REDACTED_VALUE_APPEARS_HERE' https://api.example.com/deploy
+ echo 'deploy started'
deploy started

The trace goes to stderr. Where stderr goes decides how bad this is:

• Run from a systemd unit: stderr is captured by the journal. Every value is now in journalctl -u deploy, readable by every member of the systemd-journal and adm groups, and persisted for the journal retention period.
• Run from cron: stderr is mailed, or captured by the cron log.
• Run in CI: stderr is the build log, which is retained, searchable, and usually readable by everyone with access to the project.
• Run by hand with bash -x: it is on your terminal, and in your terminal scrollback, and in the ticket you pasted it into.

The last one is how most of these are discovered.

Turning it off without tracing the switch-off

Wrapping the sensitive section is the fix, but the naive version traces its own first line:

set -x
# ... normal traced work ...

{ set +x; } 2>/dev/null      # the disable is not itself traced
PW=$(< /run/secrets/db.pw)
curl -H "Authorization: Bearer $PW" https://api.example.com/deploy
set -x                       # back to tracing

The braces group the command and the redirection sends that group’s trace output to /dev/null, so the set +x line does not appear. Without the braces, set +x is traced before it takes effect - which does not print the secret, but does make people believe the section is protected when the very next construct they add is not.

The stronger form sends all trace output somewhere private for the life of the script:

install -m 0600 /dev/null /var/log/deploy-trace.log
exec {trace_fd}>/var/log/deploy-trace.log
BASH_XTRACEFD=$trace_fd
set -x

Note the install before the exec: opening the file with a redirection creates it at your umask, which is the same window the previous lesson described. Create it at 0600 first, then open it.

BASH_XTRACEFD names the file descriptor bash writes trace output to. It is still a file containing your secrets - the point is that it is a file whose permissions you chose, rather than a journal whose audience you did not.

Verbose clients

set -x is not the only debugging switch that prints credentials. curl -v prints the request headers it sends, including Authorization:

curl -v -H "Authorization: Bearer $TOKEN" https://api.example.com/v1/status

Use curl -v freely while developing against a test credential, and never in a script that runs unattended with a production one. When you need the timing and status without the headers, --write-out gives you the useful part with none of the exposure:

curl -sS -o /dev/null \
     --write-out 'http=%{http_code} time=%{time_total}\n' \
     -H "Authorization: Bearer $TOKEN" https://api.example.com/v1/status

The same pattern applies elsewhere: ssh -vvv prints key paths and agent negotiation but not key material; ansible -vvv prints module arguments, which do include passwords unless the task sets no_log: true; openssl s_client prints certificates, which are public, and never the private key.

- name: Set the application database password
  ansible.builtin.command: /usr/local/bin/set-db-password
  args:
    stdin: "{{ db_password }}"
  no_log: true

History files are not just the shell

Everyone knows about ~/.bash_history. The others are routinely missed:

ls -la ~/.*history ~/.*_history 2>/dev/null

psql has a specific answer worth knowing: \password builds the ALTER ROLE statement client-side, sends the hashed value, and does not put the plaintext in the history file. Typing ALTER ROLE app PASSWORD 'REPLACE_ME'; by hand does put it there, and also puts it in the PostgreSQL server log if statement logging is on. Two copies, one command.

The shell settings, honestly

echo "HISTFILE=${HISTFILE:-unset} HISTCONTROL=${HISTCONTROL:-unset}"
set +o history          # bash: stop recording for this shell

These are hygiene, not security. They keep clutter out of a file; they do nothing about /proc/PID/cmdline, ps, the auditd execve record, or a terminal-recording tool. The previous lesson made this point about HISTCONTROL and it is worth repeating in the context of scripts: if a secret was ever an argument to an external command, it was exposed, and the remedy is rotation rather than a history setting.

The journal, and what a service prints at startup

Applications log their configuration at startup more often than their authors admit. The database URL is the usual offender, because the password is inside it:

sudo journalctl -u myapp --since '-7d' --no-pager \
  | grep -iE 'password=|token=|secret=|://[^ ]*:[^ @]*@'

The last pattern catches credentials embedded in a URL, which is the form that slips past a search for the word “password”.

If that grep finds something, the finding is not “fix the log line”. It is:

1. Rotate the credential. It has been in a readable log for the retention period.
2. Then fix the log line, so the rotation is not undone at the next restart.
3. Then check whether the journal is forwarded anywhere - a central log store, an index, a SIEM - because each of those is another copy with its own retention.

CI masking only masks what it recognises

Every CI system offers to mask secret values in build output. The masking is a literal string replacement on the exact value. It therefore fails on any transformation:

# Masked: the exact value appears in the output
echo "$API_TOKEN"

# Not masked: the value has been transformed
echo "$API_TOKEN" | base64
echo "${API_TOKEN:0:8}"
jq -n --arg t "$API_TOKEN" '{token: $t}'

Base64 is the common one, because encoding a value to put it in a header or a Kubernetes manifest is routine and the encoded form is trivially reversible. Truncation is next - “just log the first eight characters so we can tell which token it is” defeats the masker and hands over a useful fingerprint.

Knowledge check