Instance identity and cloud IAM from the Linux side

A backup job on an instance needs to write to object storage. A log shipper needs to call a monitoring API. Something has to answer the question “who is this process, and may it do that?”

There are two families of answer. One puts a long-lived key on the filesystem. The other gives the instance an identity and lets the platform mint a short-lived credential on demand. The second is better in every respect, and it still fails regularly

• not because of the cloud, but because of Linux details the platform documentation does not cover.

Instance identity

Every major cloud can attach an identity to an instance: an instance profile carrying an IAM role on AWS, a service account on GCP, a managed identity on Azure. The mechanism is the same in all three:

1. The instance is launched with an identity attached.
2. A process asks the instance metadata service for credentials.
3. The platform returns a token that expires in minutes to hours.
4. The SDK refreshes it before expiry, without being asked.

Nothing is written to disk. Nothing needs rotating. If the instance is terminated, the credential dies with it, and the audit trail names the role and the instance.

The retrieval is the same endpoint the previous lessons warned you about, because it is the same security boundary:

TOKEN=$(curl -sX PUT "http://169.254.169.254/latest/api/token" \
  -H "X-aws-ec2-metadata-token-ttl-seconds: 60")

ROLE=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/meta-data/iam/security-credentials/)

echo "instance role: $ROLE"

Anything that can reach that endpoint holds the role. That is the whole security model, and it is why the metadata hardening in linux-cloud-init-and-metadata is not optional once a role is attached.

The resolution chain, and how it betrays you

Cloud SDKs and CLIs look for credentials in a fixed order and take the first source that yields anything. Roughly:

The instance role is last. That ordering produces the single most confusing cloud failure a Linux administrator meets:

The role has the permission. The policy simulator says allow. The command says access denied.

Because a file in a home directory, or an environment variable in a unit, is winning the race - and it holds a key that expired, belongs to a different account, or was scoped for something else entirely. The role is never consulted.

The triage is entirely local:

# 1. What identity is actually in effect? Ask the API, not the config.
aws sts get-caller-identity

# 2. Is something in the environment shadowing the role?
env | grep -iE 'aws_|google_|azure_' || echo "no credential vars set"

# 3. Is there a credentials file for this user?
ls -l ~/.aws/credentials /etc/aws 2>/dev/null

# 4. For a service, look at the unit rather than at your own shell
systemctl show backup.service -p Environment -p EnvironmentFile
sudo tr '\0' '\n' < /proc/"$(systemctl show -p MainPID --value backup.service)"/environ \
  | grep -iE 'aws_|google_|azure_'

Step 4 matters because your interactive shell and the service are different environments. A service inheriting a stale AWS_ACCESS_KEY_ID from /etc/environment or from a EnvironmentFile= will fail while the same command works perfectly when you run it by hand, and the reverse happens just as often.

Where a secret leaks on a Linux box

If you cannot use an instance role - a hybrid host, a third-party API, a database password - the secret has to live somewhere, and Linux offers several places that look private and are not.

The process environment. /proc/<pid>/environ is readable by the process owner and by root. On a multi-user host, a service running as the same UID as an operator account exposes its whole environment to that operator. It also survives into core dumps and into anything that logs the environment on crash.

tr '\0' '\n' < /proc/self/environ | head -5

The command line. /proc/<pid>/cmdline is world-readable. A password passed as an argument is visible to every user on the system for as long as the process runs:

ps -eo pid,user,args | grep -i password

There is no fix for this other than not doing it. Tools that accept --password-file or read from stdin exist for exactly this reason.

A unit file. Environment=DB_PASSWORD=... in a unit is readable by anyone who can run systemctl show, and unit files are usually mode 0644:

$ systemctl show ssh.service -p Environment

Environment=

EnvironmentFile=/etc/myapp/secrets with mode 0600 and owner root is better - the value is no longer in systemctl show - but it is still a plaintext file, and it is still in the process environment once loaded.

The best available answer on modern systemd is LoadCredential=, which places the secret in a per-service directory that only that service can read, and does not put it in the environment at all:

# /etc/systemd/system/myapp.service
[Service]
LoadCredential=db-password:/etc/myapp/db-password
ExecStart=/usr/local/bin/myapp --password-file ${CREDENTIALS_DIRECTORY}/db-password

systemd mounts the credential into a tmpfs private to the service, unmounts it when the service stops, and keeps it out of /proc/<pid>/environ. LoadCredentialEncrypted= goes further and stores the file encrypted at rest, decrypted with a key held by the TPM or the host:

# Encrypt a secret so the file on disk is not the secret
echo -n 'REPLACE_ME' | sudo systemd-creds encrypt --name=db-password - \
  /etc/myapp/db-password.cred

Whether the TPM is available on a cloud instance depends on the platform and the instance type; systemd-creds falls back to a host key under /var/lib/systemd/credential.secret when it is not, which protects against a stolen file but not against root on the running host.

Scoping the identity

An instance role applies to the whole instance. Every process on it - every container, every cron job, every compromised dependency - gets the same permissions.

• One role per workload, not one role per instance type. A node that runs three services with an instance role holding the union of their permissions has given each of them the other two sets. If the platform offers a per-workload identity, use it.
• Containers inherit the node role by default. A container that can route to 169.254.169.254 reads the node’s credentials. Deny the address from container networks, or use the platform’s scoped identity mechanism - this is the same point made in linux-cloud-init-and-metadata, and it is the most commonly skipped control in a container deployment.
• Read the audit log, not the policy. Cloud audit logs record which identity made each call. That is how you discover that a role you believed was used by one service is being used by four, and it is the only reliable input to narrowing a policy without breaking something.

Knowledge check