Prometheus and Secrets

A team scrapes a SaaS partner’s /metrics endpoint with a bearer token. They paste the token into prometheus.yml as bearer_token: tk_live_8f3a2c... because that is the shortest path. A backup agent snapshots the home directory every night, including the Git repository that holds the config. A new joiner needs to read the config; the README points them at the Git repo. The token is now visible to anyone with read access to the repository. Six weeks later the partner rotates the token and asks “who leaked this?”. The audit trail points at the Git commit. The fix is to use bearer_token_file: /etc/prometheus/secrets/partner-token and render the file from a secrets manager.

Secrets management in Prometheus 2.55.x is not a built-in feature. It is a deliberate choice between three shapes — file-based, environment-variable, and Vault-backed — each of which trades operational complexity against the size of the blast radius when something goes wrong. This lesson is about the shapes, the leak paths, and the right posture per deployment.

What it is

Prometheus 2.55.x consumes secret material at four points:

1. Scrape credentials. basic_auth.password, basic_auth.password_file, bearer_token, bearer_token_file, authorization.credentials, authorization.credentials_file, oauth2.*.
2. TLS material. tls_config.ca_file, tls_config.cert_file, tls_config.key_file, plus the tls_server_config fields in web.yml.
3. Remote-write / federation credentials. The same basic_auth and bearer_token blocks under the remote_write and federation scrape jobs.
4. Alertmanager and Alertmanager webhook receivers. Basic auth on the Alertmanager endpoint (configured in Prometheus’s alerting block).

Three approaches to render the secret material to Prometheus:

• File-based. password_file, bearer_token_file, tls_config.ca_file, etc. Prometheus reads the file on every scrape (for credentials) or once at start (for TLS material). The file is rendered by a secrets manager (HashiCorp Vault, AWS Secrets Manager, Kubernetes Secrets) at boot or via a sidecar.
• Environment-variable substitution. Prometheus 2.55.x does not perform environment-variable substitution inside scrape config credential fields. The substitution works only at the command-line flag level (--config.file, --storage.tsdb.path, etc.) and only when the env var is referenced as ${VAR}.
• Vault-backed. A sidecar (Vault Agent, the external- secrets operator, a custom init container) renders the secret files at boot and refreshes them on a schedule. Prometheus reads the files; the secrets manager handles the source of truth.

        Secrets manager                  Prometheus
   +----------------------+          +-----------------+
   | HashiCorp Vault      |          | --config.file   |
   | AWS Secrets Manager  |---render |   /etc/prometheus/
   | Kubernetes Secret    |   file   |   prometheus.yml |
   | Doppler              |          |                 |
   +----------------------+          | password_file:  |
                                     |   /etc/prometheus/
                                     |   secrets/app.pass
                                     +-----------------+
                                            |
                                            v
                                     +-----------------+
                                     |  /api/v1/...    |
                                     +-----------------+

The mental model: the secrets manager is the source of truth; the rendered file is the cache; Prometheus reads the file. The Git-tracked config references the file path, never the contents.

Why a sysadmin cares

The blast radius of a leaked Prometheus secret is the entire fleet of scrape targets and federation endpoints. The most common leak paths in real incidents:

• Git repository. The literal password: or bearer_token: in a committed config file. The blast radius is “every developer with read access to the repo”. The fix is password_file / bearer_token_file referencing a file rendered by a secrets manager.
• /api/v1/status/config. The endpoint returns the resolved configuration, including the contents of any password_file and bearer_token_file. The blast radius is “every caller that can reach the Prometheus API”. The fix is server-side authentication (lesson 02) and a restrictive bind address (lesson 01).
• /proc/<pid>/cmdline. A --bearer-token=tk_live_... command-line argument puts the token in /proc/<pid>/cmdline, which is readable by every user on the host. The fix is to keep the token out of the command line entirely; use bearer_token_file.
• Log files. A scrape error that includes the credential in the message. The fix is to keep the credential in a file and let Prometheus log “credentials missing” rather than the literal token.
• Backup snapshots. A nightly backup of /etc/prometheus that includes the secret files. The fix is to render the secret files at boot and exclude /etc/prometheus/secrets from backups.

The right answer is to use the file-based shape for every credential, render the files from a secrets manager, restrict the file mode to 0640 owned by the Prometheus user, and treat the directory as if it were a credential map.

How it works

The file-based shape

# /etc/prometheus/prometheus.yml
scrape_configs:
  - job_name: 'application_exporter'
    static_configs:
      - targets: ['app-svc.internal:9100']
    basic_auth:
      username: monitoring
      password_file: /etc/prometheus/secrets/app-exporter.pass

# /etc/prometheus/secrets/app-exporter.pass (mode 0640, owned by prometheus)
s3cr3t-passw0rd

The password_file directive tells Prometheus to read the file on every scrape. The file’s contents are the password; the trailing newline is optional. The file’s mode must allow the Prometheus process user to read it; nothing else should.

The same shape applies to:

bearer_token_file: /etc/prometheus/secrets/partner-token
authorization:
  credentials_file: /etc/prometheus/secrets/custom-auth
tls_config:
  ca_file: /etc/prometheus/ca/internal-ca.crt
  cert_file: /etc/prometheus/client.crt
  key_file: /etc/prometheus/client.key

For TLS material, Prometheus reads the files once at start (for tls_config under scrape jobs) or on SIGHUP. The file mode for key_file should be 0600 owned by the Prometheus user; the key is a credential in its own right.

Environment-variable substitution

Prometheus 2.55.x does not perform environment-variable substitution inside scrape config credential fields. A config like:

scrape_configs:
  - job_name: 'application_exporter'
    basic_auth:
      username: monitoring
      password: ${APP_EXPORTER_PASSWORD}

fails to load with the literal string ${APP_EXPORTER_PASSWORD} sent as the password. The substitution works only at the command-line flag level:

# This works: env var substitution at the flag level.
PROM_CONFIG=/etc/prometheus/prometheus.yml
prometheus --config.file=${PROM_CONFIG}

# This does NOT work: env var substitution inside the YAML.
# Prometheus treats ${APP_EXPORTER_PASSWORD} as a literal string.

The right shape for environment-variable-driven configuration is to render the file from the environment at boot (with a templating tool, an init container, or a sidecar) and let Prometheus read the rendered file.

The Vault-backed shape

HashiCorp Vault is the canonical secrets manager for a VM or hybrid estate. The integration shape:

   HashiCorp Vault          Vault Agent            Prometheus
   +-----------------+      +----------+           +---------+
   | secret/data/    |      | template |---render--> config  |
   |  prometheus/    |----->|          |           +---------+
   |  app-exporter   |      | renew    |
   +-----------------+      +----------+
                                  |
                                  v
                          /etc/prometheus/secrets/app-exporter.pass

The Vault Agent runs as a sidecar (or as a systemd unit on the Prometheus host), reads the secret from Vault, renders the file under /etc/prometheus/secrets/, and renews the secret on the configured rotation schedule. Prometheus reads the file on every scrape and picks up the new contents automatically.

For Kubernetes deployments, the external-secrets operator (or the upstream secrets-store-csi-driver) plays the same role: a SecretStore points at Vault (or AWS Secrets Manager or GCP Secret Manager); an ExternalSecret renders a Kubernetes Secret; the Secret is mounted as a file at the password_file path. Prometheus sees a file; the secrets manager handles the rotation.

How to configure it

File-based secrets with restrictive permissions

# READ-ONLY: create the directory with restrictive mode.
sudo install -d -m 0750 -o prometheus -g prometheus \
  /etc/prometheus/secrets

# CONFIGURATION: render the credential at boot from a secrets manager.
# (Example: Vault Agent template block)
# template {
#   destination = "/etc/prometheus/secrets/app-exporter.pass"
#   perms       = "0640"
#   contents    = "{{ with secret \"secret/data/prometheus/app-exporter\" }}{{ .Data.data.password }}{{ end }}"
# }

# CONFIGURATION: confirm the file mode after rendering.
ls -l /etc/prometheus/secrets/
# -rw-r----- 1 prometheus prometheus 17 Aug 14 10:00 app-exporter.pass

# CONFIGURATION: confirm Prometheus can read the file.
sudo -u prometheus cat /etc/prometheus/secrets/app-exporter.pass
# s3cr3t-passw0rd

Scrape config referencing the file

# /etc/prometheus/prometheus.yml
scrape_configs:
  - job_name: 'application_exporter'
    scheme: https
    static_configs:
      - targets: ['app-svc.internal:9100']
    basic_auth:
      username: monitoring
      password_file: /etc/prometheus/secrets/app-exporter.pass
    tls_config:
      ca_file: /etc/prometheus/ca/internal-ca.crt
      server_name: app-svc.internal
      min_version: TLS12

Excluding the secrets directory from backups

# /etc/backup/excludes.d/prometheus.conf
/etc/prometheus/secrets/
/etc/prometheus/web.yml

The web configuration file (when it contains bcrypt hashes) is not a credential in itself, but it is part of the authentication posture and is usually excluded from backups that are stored off-host.

Kubernetes with external-secrets

# SecretStore pointing at Vault.
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-backend
  namespace: monitoring
spec:
  provider:
    vault:
      server: "https://vault.internal:8200"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "prometheus"
---
# ExternalSecret that renders the credential as a Kubernetes Secret.
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: app-exporter-credential
  namespace: monitoring
spec:
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: app-exporter-credential
  data:
    - secretKey: password
      remoteRef:
        key: prometheus/app-exporter
        property: password
---
# Prometheus pod spec mounts the Secret as a file.
spec:
  containers:
    - name: prometheus
      image: prom/prometheus:v2.55.1
      args:
        - --config.file=/etc/prometheus/prometheus.yml
        - --storage.tsdb.path=/prometheus
        - --web.listen-address=127.0.0.1:9090
      volumeMounts:
        - name: secrets
          mountPath: /etc/prometheus/secrets
          readOnly: true
  volumes:
    - name: secrets
      secret:
        secretName: app-exporter-credential

# /etc/prometheus/prometheus.yml (in the same pod)
scrape_configs:
  - job_name: 'application_exporter'
    basic_auth:
      username: monitoring
      password_file: /etc/prometheus/secrets/password

The credential rotates in Vault; the external-secrets operator refreshes the Kubernetes Secret on the configured schedule; the mounted file updates; Prometheus reads the new contents on the next scrape.

How to validate it

# READ-ONLY: confirm the secret file exists and is readable.
sudo -u prometheus test -r /etc/prometheus/secrets/app-exporter.pass \
  && echo "OK: prometheus user can read"

# READ-ONLY: confirm the file mode is restrictive.
ls -l /etc/prometheus/secrets/app-exporter.pass
# -rw-r----- 1 prometheus prometheus 17 Aug 14 10:00 app-exporter.pass

# READ-ONLY: confirm Prometheus can read it (manual scrape).
curl -fsS -u monitoring:$(sudo -u prometheus cat /etc/prometheus/secrets/app-exporter.pass) \
  http://app-svc.internal:9100/metrics | head -3
# (200 OK)

# READ-ONLY: confirm the file does not appear in /proc/<pid>/cmdline.
cat /proc/$(pgrep -f 'prometheus --config.file')/cmdline | tr '\0' ' '
# (no credential strings; only flags and paths)

# READ-ONLY: confirm the file is not in the config endpoint response
# when authentication is required.
curl -s -u grafana:$PASS http://127.0.0.1:9090/api/v1/status/config \
  | jq '.data.yaml | test("password:")'
# false

# CONFIGURATION: rotate the credential.
echo 'new-passw0rd' | sudo tee /etc/prometheus/secrets/app-exporter.pass
sudo systemctl reload prometheus

# READ-ONLY: confirm the new credential is in effect.
sleep 15
curl -fsS http://127.0.0.1:9090/api/v1/targets \
  | jq '.data.activeTargets[] | select(.labels.job == "application_exporter") | .health'
# "up"

A clean validation: the file exists with 0640 permissions owned by the Prometheus user; the credential is not in /proc/<pid>/cmdline; the scrape succeeds with the file’s contents; the rotation takes effect on the next scrape without a Prometheus restart.

How it can fail

The six most expensive secrets-management failures in real Prometheus installs.

1. Literal password: in prometheus.yml. The config is in Git. The credential is in the diff. The symptom is a git grep finding the password in a historical commit. The fix is to rotate the password and rewrite the config to use password_file.
2. password_file world-readable. The credential leaks to every user on the host. The symptom is that the password appears in /etc dumps taken by a backup agent or a system inventory tool. The fix is chmod 0640 and chown prometheus:prometheus.
3. Secret file not excluded from backups. The nightly backup of /etc/prometheus includes the secrets directory. The blast radius is “every host that receives the backup”. The fix is to exclude the directory from the backup configuration.
4. Vault Agent template renders with world-readable permissions. The template block in Vault Agent sets perms = "0644" by default in some configurations. The fix is perms = "0640" and a user / group block pointing at the Prometheus user.
5. ExternalSecret refresh interval too long. The credential is rotated in Vault; the Kubernetes Secret is not refreshed for an hour. The blast radius is “Prometheus scrapes with a stale credential for the duration of the refresh window”. The fix is a refreshInterval of 1m or less for high- rotation secrets.
6. Secret file rendered before the directory exists. A misconfigured init container runs the Vault Agent template before /etc/prometheus/secrets/ is created. The render fails; Prometheus starts with no credential file; every scrape fails with credentials missing. The fix is to create the directory in an init container with the right ownership before the Vault Agent template runs.

How to troubleshoot it

Diagnostic order: does the file exist, can Prometheus read it, what does the scrape error actually say.

1. Check the file.

   ls -l /etc/prometheus/secrets/
   sudo -u prometheus cat /etc/prometheus/secrets/app-exporter.pass

   The file must exist, be readable by the Prometheus user, and contain the credential the target expects.
2. Check /proc/<pid>/cmdline. A literal --password=... flag is visible. The fix is to remove the flag and use password_file.
3. Check the targets page. last error: basic auth credentials missing or empty means the file path is wrong or the file is unreadable. last error: 401 Unauthorized means the file is readable but the contents are wrong.
4. Check the rotation chain. If Vault is the source of truth, validate that the Vault Agent template rendered the file, that the Kubernetes Secret refreshed (in K8s), and that the Prometheus pod sees the new mount.
5. Reload carefully. A SIGHUP reloads the scrape config; the password_file directive is read again. The file’s contents are re-read on the next scrape; no reload is needed for a contents change. (CONFIGURATION.)

Security implications

The secrets-management surface is where the threat model collapses or holds. The right posture combines:

• Source of truth in a secrets manager. Vault, AWS Secrets Manager, Kubernetes Secrets with external-secrets, Doppler. The manager handles rotation, audit, and access control.
• File-based rendering. Prometheus reads files; the secrets manager renders them. The contents never appear in Git, in /proc/<pid>/cmdline, or in prometheus.yml.
• Restrictive file permissions. 0640 for credentials; 0600 for TLS keys. Owned by the Prometheus user. No world-readable or group-readable secrets.
• Backup exclusion. The secrets directory is excluded from off-host backups. The audit log of the secrets manager is the source of truth for “who read this secret”.
• Authentication and bind address. Server-side authentication (lesson 02) and a restrictive bind address (lesson 01) keep /api/v1/status/config from leaking the resolved configuration.

A secret rotation pipeline that takes a credential from “in Vault” to “in /etc/prometheus/secrets/” to “in the Authorization header on the next scrape” without any human touching the contents is the production target.

Verification

You should now be able to answer:

• What is the difference between password_file and a literal password: in prometheus.yml?
• Does Prometheus 2.55.x perform environment-variable substitution inside scrape config credential fields?
• What renders the secret file when the secrets manager is HashiCorp Vault? When it is Kubernetes Secrets with external-secrets?
• Which leak path does the file-based shape close that the literal password: shape does not?

Quiz