Storage anti-patterns II - logs, the environment, the process table and people

The exposures in this lesson are all transient in appearance and durable in reality. A log line scrolls past, an environment block is set up and torn down, a command runs for two hundred milliseconds, a message is deleted from a channel. Every one of those events is written to something with retention, and the retention is usually longer than the credential lifetime.

Log masking is documented as best effort

The platform promise is real and narrower than teams read it as. GitHub states that Actions automatically redacts the contents of all secrets printed to workflow logs, and then states in the same family of documents that this redaction is not guaranteed and that automatic redaction is not guaranteed. Four documented conditions break it.

• Structured data. Redaction largely relies on finding an exact match for the specific secret value, and structured output such as JSON or YAML can cause that match to fail because quoting, escaping and whitespace alter the byte sequence.
• Transformations. Registration applies to a specific value, so a base64 or URL-encoded derivative is not redacted unless it is separately registered as a secret in its own right.
• Job scope. The runner can only redact secrets used within the current job. A value that reaches a log through a different job has never been registered on that runner.
• Ordering. The mask directive is not retroactive. The documentation is explicit that the value must be registered before it is output, and it notes a consequence people trip over: after masking a value you cannot set that value as a step output.

# Register a value the platform never issued, before it is used.
- name: Exchange for a short-lived token
  run: |
    TOKEN="$(./scripts/exchange-assertion.sh)"
    echo "::add-mask::$TOKEN"
    ./scripts/deploy.sh --token-stdin <<<"$TOKEN"

Registration is per whitespace-separated word, so a value containing spaces is masked word by word rather than as one unit. Two limits are worth remembering because they shape design: a secret is limited to 48 KB, and secrets are not passed to the runner for workflows triggered from a forked repository, with the sole exception of the platform token. That last protection has a documented hole. A workflow triggered by the pull request target event runs with a read and write platform token even from a public fork, and has secret access, which is why the guidance forbids checking out untrusted code under that trigger.

Environment variables deserve an assessment, not a slogan

“Never use environment variables for secrets” is repeated widely and is not what the operating system says. On Linux the environment of a running process is exposed through the process filesystem with mode 0400 owned by the process owner, while the full argument vector of the same process is exposed with mode 0444. The environment is readable by that user and by root. The command line is readable by every local account on the machine.

# The permission difference is visible directly.
ls -l /proc/1/cmdline /proc/1/environ

# Every local user can read every argument vector by design.
ps -eo pid,user,args

That inverts the usual ranking. Between the two carriers people argue about, the environment is the stronger one. The genuine exposures for an environment variable are specific and worth naming rather than gesturing at.

• Inheritance. Every child process receives a copy. The size of that risk is the size of the process tree, which is small in a single-process container and large on a shared host with a shell.
• Serialisation into reports. Crash handlers, error tracking agents, application performance monitoring integrations and framework debug pages routinely include the environment in a report. This is the most common real leak, and it exports the value to a third party.
• Platform introspection. A container runtime displays a container configured environment on inspection, and an orchestrator object that carries literal environment values exposes them to anyone who can read the object.
• Unit configuration. Environment directives written into a service unit become part of the unit properties the service manager publishes, so the value is readable without reading the unit file. An environment file referenced by path, with restrictive ownership and mode, keeps the value out of those properties.

[Service]
# Better: the value lives in a mode 0600 file, not in the unit.
EnvironmentFile=/etc/payments/db.env

# Best: the service manager passes a credential the unit never holds.
LoadCredential=db-password:/run/secrets/payments-db

An environment variable is a defensible carrier when the value is injected at start by the orchestrator from a real store rather than read from a checked-in file, when the process tree is effectively one process, when nothing in the stack serialises the environment into a report, and when the value is short lived enough that a copy in a crash report has a bounded window. It is the wrong carrier on a shared multi-user host, in any application with an error tracker attached, and for anything long lived.

The process table is the exposure with no permissions to set

The argument vector is public metadata by design, because process listings are an operating system feature that every account expects to work. This makes a credential on a command line the single worst placement available on a host, and it is the one that keeps appearing because command line flags are how tools are documented.

# ANTI-PATTERN: any local account can read this argument vector.
mysql -h db-03.example.com -u app -p"$DB_PASSWORD" -e "SELECT 1"

# The value never enters the argument vector.
mysql --defaults-extra-file=/run/payments/my.cnf -e "SELECT 1"

The window is short and does not need to be long. A loop that samples the process table every second on a shared host will catch a credential that appears for two hundred milliseconds often enough, and monitoring agents already sample the process table continuously and ship what they find to a metrics or logs backend with months of retention. Mounting the process filesystem with the hidepid option restricts which processes an unprivileged account can see, which reduces the exposure without removing it, since the account running the command still sees it.

flowchart TD
    C["Command line argument"] --> R1["Any local account\nplus every agent\nsampling the process table"]
    E["Environment variable"] --> R2["Process owner and root\nplus every child process"]
    F["File, mode 0400,\nmemory-backed mount"] --> R3["Service account and root"]
    L["Service manager credential"] --> R4["The service, for its lifetime"]

The diagram is the ranking made concrete: each carrier on the left has a reader set on the right, and the design question is always which reader set you are willing to defend. Note that root appears in three of the four, which is a reminder that host compromise defeats all of them and that the compensating control is credential lifetime rather than placement.

The human layer is a store with retention and no owner

Spreadsheets, tickets and chat are storage systems. They replicate, they index, they export, they back up, and they are administered by teams who were never told they are custodians of credentials.

A shared spreadsheet has no attribution, so nobody can say who read a value or when, and it has no rotation trigger, so an offboarding does not invalidate anything. Its collaboration platform keeps a version history, which means deleting the cell leaves the value in a prior revision that the deletion did not touch.

A ticket is worse in one specific way: ticket systems are built for search and for retention measured in years, so a credential pasted into a comment is indexed, is included in exports and analytics extracts, and frequently survives the deletion of the ticket itself as an orphaned attachment.

Chat is worst of all, because it combines the retention of a ticket with the volume of a conversation. The search index over a workspace is, in effect, a full text index of every credential ever pasted into it. Deleting a message removes it from the current view rather than from exports, from compliance archives or from a recipient local cache. And a credential shared as a screenshot defeats text-based scanning entirely, which is why scanner coverage numbers on collaboration platforms should be read sceptically.

Production discipline

1. Register derived values with the masker at the moment they are produced. A token minted mid-job was never registered, and registration after the fact does nothing for lines already written.
2. Forbid the pull request target trigger from checking out untrusted code. It carries a read and write token and secret access even from a public fork, which is the documented hazard.
3. Never place a credential in an argument vector. Use a defaults file with restrictive mode, a file descriptor, or standard input, and treat any tool that offers only a flag as a wrapper to be written.
4. Audit what serialises the environment. Error trackers, crash handlers and performance agents are the common path by which an environment variable leaves the estate entirely.
5. Instrument the sanctioned path and publish its latency. A slow correct path is the cause of the human layer, and it is the only part of the human layer an engineering team can actually fix.

Cross-course references

• Linux for Production Sysadmins - Part LXXII (Secrets) covers secrets in scripts, logs and shell history, which is the same exposure class one layer below the pipeline.
• Observability for Production Sysadmins - Part LXXXII (SensitiveTelemetry) covers redaction in telemetry pipelines and the audit that finds credentials already shipped into a metrics or logs backend.
• Git, CI/CD & GitOps for Infrastructure Engineers - Part XLII (CISecrets) covers masking limits and log leakage from the pipeline side, including the environment scoping that bounds which jobs can see a value at all.

Quiz