Prometheus Authentication

A team scrapes a SaaS partner’s API for latency metrics. The partner gave them a bearer token, tk_live_8f3a2c.... The team pastes the token directly into prometheus.yml under bearer_token: and commits the config to Git. A former contractor still has read access to the repository. Two months later the partner rotates the token and asks “who leaked this?” The audit trail points at the commit hash. The token in the config file was the leak.

Prometheus has two different authentication concepts that look similar and behave differently: client-side authentication, which is what Prometheus uses to authenticate itself to a scrape target, and server-side authentication, which is what Prometheus uses to authenticate the callers of its own API and UI. This lesson covers both, the right approach per scrape, and the most common ways the credentials leak.

What it is

Prometheus 2.55.x supports three client-side credential shapes on every scrape target and federation endpoint:

1. HTTP Basic Auth — username and password (or password_file) sent as a Authorization: Basic <b64> header on every scrape. Use when the target expects a username and password (most common with exporters that support --web.config.file and traditional services that speak HTTP basic).
2. Bearer Token — bearer_token (literal string) or bearer_token_file (path to a file containing the token) sent as Authorization: Bearer <token>. Use when the target expects a token (most modern SaaS metrics endpoints, the federation /federate endpoint, and many managed observability services).
3. mTLS — client certificate authentication via the tls_config block. Covered in lesson 03 (TLS).

The server side has its own authentication:

• Basic auth users for the Prometheus HTTP API, configured in the web configuration file (--web.config.file) under basic_auth_users. The passwords are bcrypt hashes.

   scrape target             Prometheus                    callers of /api
+------------+   GET /metrics  +------------+    GET /api/v1/...
|   user:    |<--Basic---------|            |<--Basic----------<operator>
|   pass:    |                 |            |
+------------+                 |            |    GET /federate
                              |            |<--Bearer----------<peer Prometheus>
+------------+   GET /metrics  +------------+
| bearer:    |<--Bearer--------|            |
| tk_live... |                 |            |
+------------+                 +------------+

The mental model: scrape credentials live with the scrape job; API credentials live in the web configuration file; and neither should appear in a Git-tracked file.

Why a sysadmin cares

Credentials in Prometheus configs leak in three well-trodden ways:

• The config file is in a Git repository. Anyone with read access to the repo sees every scrape credential. A former contractor or a compromised CI token is enough.
• The config is exposed via /api/v1/status/config. An authenticated or unauthenticated caller who can reach the Prometheus API gets the entire scrape config including any literal password: or bearer_token:. Lesson 04 covers the admin API surface; the takeaway is that every credential reachable from the Prometheus HTTP port is one access away from the config endpoint.
• The credential is in process arguments. A prometheus --bearer-token=tk_live_... invocation puts the token in /proc/<pid>/cmdline and visible to any user with read access to /proc. The right answer is password_file / bearer_token_file.

The most expensive credential failures in real installs are not “Prometheus could not authenticate”; they are “an attacker authenticated because the credential was visible”.

How it works

Client-side: the scrape request

For every scrape target, Prometheus builds the HTTP request from the scrape job config. If the job has basic_auth set, Prometheus adds Authorization: Basic base64(user:password) to the Accept-Encoding / User-Agent headers it always sends. If the job has bearer_token or bearer_token_file, Prometheus adds Authorization: Bearer <token> instead. The two blocks are mutually exclusive at the scrape job level; setting both produces a parse error at config reload.

The credentials are evaluated in this order:

1. The basic_auth block (if set).
2. The authorization block (if set, for custom credentials).
3. The bearer_token / bearer_token_file (if set).
4. The oauth2 block (if set, for token exchange).

For TLS-encrypted scrapes, the tls_config block controls client certificate authentication and CA validation. Lesson 03 covers that path.

Server-side: the web configuration file

Prometheus’s own HTTP listener can require authentication when the operator passes --web.config.file=/etc/prometheus/web.yml. The web config file has its own schema:

# /etc/prometheus/web.yml
basic_auth_users:
  alice: $2y$10$bcrypt-hash-of-password-here
tls_server_config:
  cert_file: /etc/prometheus/server.crt
  key_file: /etc/prometheus/server.key

The basic_auth_users map accepts usernames mapped to bcrypt password hashes. The format is the same one htpasswd -B emits. Plaintext passwords are not accepted at config-load time; promtool check web-config validates the hash format.

When basic_auth_users is set, every request to the Prometheus HTTP listener (UI, API, /-/reload, /metrics) requires an Authorization: Basic header from a user whose bcrypt hash is in the map. Without the header, Prometheus answers 401 Unauthorized. The /-/quit and /-/reload endpoints behave the same way when authentication is required.

Federation and remote write

Federation (/federate) and remote write use the same bearer_token / basic_auth blocks under their respective configuration stanzas. A typical federation job:

# /etc/prometheus/prometheus.yml
scrape_configs:
  - job_name: 'federate'
    honor_labels: true
    metrics_path: '/federate'
    params:
      'match[]':
        - '{job="prometheus"}'
        - '{__name__=~"job:.*"}'
    static_configs:
      - targets:
        - 'prometheus-b.internal:9090'
    bearer_token_file: /etc/prometheus/federation-token

The bearer_token_file points at a file Prometheus reads on every scrape; the file must be readable by the Prometheus process user only. A common mistake is to give the file world- readable permissions and defeat the purpose.

How to configure it

Scrape job with basic auth

# /etc/prometheus/prometheus.yml
scrape_configs:
  - job_name: 'application_exporter'
    metrics_path: /metrics
    static_configs:
      - targets:
        - 'app-svc-1.internal:9100'
        - 'app-svc-2.internal:9100'
    basic_auth:
      username: monitoring
      password_file: /etc/prometheus/secrets/app-exporter.pass
    # tls_config covered in lesson 03
    # tls_config:
    #   ca_file: /etc/prometheus/ca/internal-ca.crt
    #   server_name: metrics.internal

The password_file directive tells Prometheus to read the file on every scrape. The file should be 0640, owned by the Prometheus user, and stored on the same volume as the rest of the secrets. Production secret managers (Vault, AWS Secrets Manager, Kubernetes secrets) typically render the file at boot via an init container or a sidecar.

Scrape job with bearer token

# /etc/prometheus/prometheus.yml
scrape_configs:
  - job_name: 'saas_metrics'
    metrics_path: /api/v1/metrics
    scheme: https
    static_configs:
      - targets:
        - 'metrics.partner.example.com'
    bearer_token_file: /etc/prometheus/secrets/partner-token

The token file should contain only the token, with a trailing newline optional. A multi-line file or a JSON object produces an HTTP 401 with no useful diagnostics.

Server-side authentication (the Prometheus API)

# /etc/default/prometheus
ARGS="--config.file=/etc/prometheus/prometheus.yml \
      --storage.tsdb.path=/var/lib/prometheus \
      --web.listen-address=127.0.0.1:9090 \
      --web.config.file=/etc/prometheus/web.yml \
      --web.external-url=https://prometheus.example.com"

# /etc/prometheus/web.yml
basic_auth_users:
  grafana: $2y$10$N9qo8uLOickgx2ZMRZoMyeIjZAgcfl7p92ldGxad68LJZdL17lhWy
  oncall: $2y$10$kE3gZ3F0m3n0Yt8aP0QkC.4pRqW6bV5sG2m8aN0vT0n1kS5bU7wCe

Generate the bcrypt hash for a new user:

# READ-ONLY: bcrypt-hash a password for inclusion in web.yml.
htpasswd -nB oncall
# New password: ********
# Re-type new password: ********
# oncall:$2y$10$kE3gZ3F0m3n0Yt8aP0QkC.4pRqW6bV5sG2m8aN0vT0n1kS5bU7wCe

Copy only the hash (the part after the colon) into web.yml. Then validate the web config before restarting:

# READ-ONLY: validate the web config schema.
promtool check web-config /etc/prometheus/web.yml
# web.yml: VALID

Remote write with basic auth

# /etc/prometheus/prometheus.yml
remote_write:
  - url: 'https://prometheus-remote.internal/api/v1/write'
    basic_auth:
      username: remote-write
      password_file: /etc/prometheus/secrets/remote-write.pass
    write_relabel_configs:
      - source_labels: [__name__]
        regex: 'go_.*'
        action: drop

Remote write with AWS SigV4 (the SigV4 alternative)

For managed Prometheus receivers that use AWS signature version 4 (Amazon Managed Prometheus, Mimir on AWS with IAM auth), Prometheus 2.55.x supports a separate sigv4 block that does not require a bearer token:

remote_write:
  - url: 'https://aps-workspaces.us-east-1.amazonaws.com/workspaces/ws-.../api/v1/remote_write'
    sigv4:
      region: us-east-1
      access_key: ${AWS_ACCESS_KEY_ID}      # not the right shape; use secret_file
      secret_key: ${AWS_SECRET_ACCESS_KEY}  # see note

The access_key / secret_key fields accept literal strings only. For production, prefer AWS IAM roles for service accounts or the file-based credentials_file if the receiver supports it. The credential shape is fundamentally different from basic_auth and bearer_token; it does not derive from HTTPS client auth.

How to validate it

# READ-ONLY: confirm the scrape credentials are working.
curl -fsS -u monitoring:$PASS \
  http://app-svc-1.internal:9100/metrics | head -5
# # HELP go_gc_duration_seconds A summary of the GC invocation durations.
# # TYPE go_gc_duration_seconds summary
# go_gc_duration_seconds{quantile="0"} 1.23e-05

# READ-ONLY: confirm Prometheus's own API requires authentication.
curl -sI http://127.0.0.1:9090/api/v1/targets
# HTTP/1.1 401 Unauthorized
# www-authenticate: Basic realm="Prometheus"

# READ-ONLY: confirm the same call works with credentials.
curl -fsS -u grafana:$PASS \
  http://127.0.0.1:9090/api/v1/status/config | jq '.data | keys'
# [
#   "yaml"
# ]

Three things should be true after this validation: the scrape target responds to the credentials Prometheus sends, the Prometheus API rejects unauthenticated calls, and the config endpoint is reachable only with credentials.

How it can fail

The five most expensive authentication failures in real Prometheus installs.

1. Literal password: in prometheus.yml. The config is in Git. The credential is in the diff. The symptom is a git grep finding the password in a historical commit. The fix is to rotate the password and rewrite the config to use password_file.
2. password_file points at a wrong path. Prometheus answers the scrape with a 401 and the UI shows context deadline exceeded for the affected targets. The symptom in the targets page is last error: ... basic auth credentials missing or empty. The fix is to render the file at boot and validate with cat /etc/prometheus/secrets/... | wc -c.
3. Bearer token rotated by the partner, Prometheus still using the old one. Every scrape returns 401. The targets page shows all targets up = 0. The fix is to update the secret file and curl -X POST http://127.0.0.1:9090/-/reload (lesson 04 covers the lifecycle controls).
4. Server-side basic_auth_users misconfigured. A user pasted the literal password instead of the bcrypt hash. The promtool check web-config catches this. The symptom on a live install is 401 Unauthorized for every credential the user thought was correct.
5. bearer_token_file world-readable. The token leaks to every user on the host. The symptom is that the token appears in /etc dumps taken by a backup agent. The fix is chmod 0640 /etc/prometheus/secrets/* and chown prometheus:prometheus.

How to troubleshoot it

Diagnostic order: which credential shape, which direction (client or server), what does the error say.

1. Inspect the targets page. GET /api/v1/targets shows lastError per target. Common strings: server returned HTTP status 401 Unauthorized, context deadline exceeded, basic auth credentials missing or empty.
2. Reproduce the scrape manually.

   curl -v -u monitoring:$PASS http://app-svc:9100/metrics

   The -v output shows the TLS handshake and the Authorization header. A successful response means Prometheus should also succeed; a 401 means the credentials or the target’s user database is wrong.
3. Inspect the file permissions.

   ls -l /etc/prometheus/secrets/
   # -rw-r----- 1 prometheus prometheus 42 Aug 14 10:00 app-exporter.pass

   Prometheus needs read access; everyone else should not.
4. Validate the server-side config.

   promtool check web-config /etc/prometheus/web.yml
   # web.yml: VALID

   A failed check points at the line and column of the problem.
5. Reload carefully. A SIGHUP reloads the scrape config but not the --web.config.file. Changes to web.yml require a Prometheus restart. (CONFIGURATION.)

Security implications

The most important security property of authentication in Prometheus 2.55.x is that file-based credentials are read on every scrape. This means rotating a credential is a matter of replacing the file’s contents and reloading Prometheus; the scrape will pick up the new value on the next interval. The downside is that the file is read continuously and must be secured continuously: a permissive file mode is a permanent leak.

Two specific attack classes:

• Config endpoint disclosure. A Prometheus with no server-side auth and a permissive bind address lets any caller read every scrape credential via /api/v1/status/config. Lesson 01 covers the bind; this lesson covers the auth requirement. Both must hold.
• Credential reuse. A token used for federation that has read access to /federate on a higher-privilege Prometheus should be treated as a high-privilege credential. Rotate it on the same schedule as any other high-privilege credential.

Verification

You should now be able to answer:

• What is the difference between basic_auth and bearer_token in a Prometheus scrape config, and which one does the federation endpoint use?
• Why is password_file preferable to a literal password: in prometheus.yml?
• What does the --web.config.file flag do, and where does Prometheus look for it?
• Which endpoint exposes the resolved scrape configuration (including the contents of password_file) to any authenticated caller?

Quiz