Prometheus TLS

A SaaS partner exposes /metrics over HTTPS. The team adds the target to prometheus.yml without tls_config and the scrape succeeds. Six months later the partner’s certificate authority rotates; the partner’s metrics endpoint presents a new chain. The Prometheus scrape continues to succeed because the partner’s metrics endpoint is signed by a public CA already in the system trust store. So far so good. Then the team’s reverse-proxy in front of a managed Prometheus receives an internal CA-signed certificate, the scrape fails with x509: certificate signed by unknown authority, and the on-call adds insecure_skip_verify: true to the scrape config “to unblock the alert”. The alert unblocks. The team is now scraping metrics over a connection that an attacker on the same network can intercept and rewrite.

TLS in Prometheus is two different things that the same config file holds: the client-side tls_config that governs Prometheus’s connection to a scrape target, and the server-side tls_server_config that governs the TLS listener Prometheus exposes on its own port. This lesson is about both, and about the certificate-validation failures that are the most common cause of “Prometheus will not scrape” incidents.

What it is

Prometheus 2.55.x uses Go’s crypto/tls package for every TLS connection. Two configuration surfaces matter:

   scrape target             Prometheus                    callers of /api
+------------+   HTTPS over  +------------+   HTTPS over   +-----------+
|  server    |   TLS 1.3      |            |   TLS 1.3       |  operator |
|  cert      |<--validation---| tls_config |<-server cert-->|  browser  |
|  (target)  |   mTLS opt.    |            |   mTLS opt.     |           |
+------------+                +------------+                +-----------+

• tls_config in scrape jobs, remote_write, and remote_read. Controls the outbound TLS handshake Prometheus performs when connecting to a target. Server-cert validation (always), CA pinning (optional), client certificate (for mTLS), and the minimum TLS version.
• tls_server_config in the web configuration file (--web.config.file). Controls the inbound TLS listener Prometheus exposes. Server certificate and private key, optional client CA for mTLS, minimum TLS version.

Both surfaces default to no TLS. A scrape with no tls_config still works because the https:// URL scheme triggers a default TLS handshake with system trust store validation. A Prometheus with no tls_server_config exposes plaintext HTTP on its listener. Both defaults are wrong for production.

Why a sysadmin cares

TLS in Prometheus has three failure shapes that recur in real incidents:

• Server-cert validation disabled to “fix” an expired certificate. A insecure_skip_verify: true is added to a scrape job to clear a x509: certificate has expired or is not yet valid alert. The scrape resumes. The connection is now unauthenticated and unencrypted from the operator’s perspective. An attacker on the same network can rewrite metrics at will.
• mTLS configured on the server side, no CA rotation discipline. The client_ca_file references a CA that has not been renewed. Three years later, the CA expires and every client with a valid certificate is rejected. The symptom is a clean tls: failed to verify client certificate error in Prometheus’s log.
• Minimum TLS version not set. Prometheus 2.55.x defaults to TLS 1.2 minimum, which is still acceptable for most compliance frameworks. A misconfigured min_version: "TLS10" allows TLS 1.0 connections and fails modern compliance scans.

The cost of getting TLS wrong is not “Prometheus does not scrape”. The cost is “Prometheus scrapes a target whose metrics have been rewritten by an attacker”, or “Prometheus answers TLS 1.0 connections to a browser that has long deprecated them”.

How it works

Client-side: the tls_config block

For every scrape job, the tls_config block (if present) controls the outbound TLS handshake. The relevant fields:

tls_config:
  # Path to a CA bundle in PEM format. Defaults to the system
  # trust store when omitted.
  ca_file: /etc/prometheus/ca/internal-ca.crt

  # Inline PEM data for the CA bundle. Useful in container
  # deployments where the CA is shipped as a Secret.
  ca: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----

  # Skip server-cert verification entirely. NEVER set true in
  # production. The flag exists for development against self-
  # signed certificates and is a misconfiguration elsewhere.
  insecure_skip_verify: false

  # Expected server name for SNI and certificate validation.
  # Defaults to the host portion of the target URL.
  server_name: metrics.internal

  # Client certificate for mTLS. Pairs with key_file.
  cert_file: /etc/prometheus/client.crt
  key_file: /etc/prometheus/client.key

  # Minimum TLS version. Defaults to TLS 1.2.
  min_version: TLS12
  max_version: TLS13

The handshake proceeds in this order:

1. Prometheus opens a TCP connection to the target.
2. The target presents its server certificate.
3. Prometheus validates the certificate against the configured ca_file (or the system trust store if ca_file is unset).
4. Prometheus checks the certificate’s Subject Alternative Name (SAN) against the server_name (or the URL host if unset).
5. If cert_file and key_file are set, Prometheus presents the client certificate; the target validates it against its configured CA.
6. The minimum TLS version is negotiated; if neither side can speak the configured minimum, the handshake fails.

A failure at any step closes the connection and the target shows up = 0 with the TLS error in lastError.

Server-side: tls_server_config

In the web configuration file (passed via --web.config.file), the tls_server_config block controls the TLS listener Prometheus exposes:

# /etc/prometheus/web.yml
tls_server_config:
  cert_file: /etc/prometheus/server.crt
  key_file: /etc/prometheus/server.key

  # For mTLS: clients must present a certificate signed by
  # this CA. The default is no client cert (TLS only).
  client_ca_file: /etc/prometheus/client-ca.crt
  client_auth_type: RequireAndVerifyClientCert

  # Minimum TLS version. Defaults to TLS 1.2.
  min_version: TLS12

client_auth_type accepts three values:

• NoClientCert (default) — server TLS only; clients do not present certificates.
• RequestClientCert — server requests a client cert but does not fail the handshake if absent.
• RequireClientCert — server requires a client cert; the handshake fails without one. The cert is not validated against a CA.
• RequireAndVerifyClientCert — server requires a client cert and validates it against client_ca_file. The mTLS shape.

For most Prometheus deployments the NoClientCert shape is correct: TLS terminates at Prometheus (or at the reverse proxy in front of it), and the upstream caller is authenticated through other means (basic_auth, network ACLs).

How to configure it

Scrape job with server-cert validation only

The most common production shape. Prometheus validates the target’s certificate but does not present a client certificate.

# /etc/prometheus/prometheus.yml
scrape_configs:
  - job_name: 'application_exporter'
    scheme: https
    metrics_path: /metrics
    static_configs:
      - targets:
        - 'app-svc-1.internal:9100'
    tls_config:
      ca_file: /etc/prometheus/ca/internal-ca.crt
      server_name: app-svc.internal
      min_version: TLS12

The ca_file references the CA bundle that signed the target’s server certificate. For a self-signed target, the file contains the target’s own certificate (or the CA that signed it). For a publicly-signed target, ca_file can be omitted; Prometheus uses the system trust store.

Scrape job with mTLS

For high-trust environments where the scrape target requires Prometheus to authenticate itself.

# /etc/prometheus/prometheus.yml
scrape_configs:
  - job_name: 'database_exporter'
    scheme: https
    metrics_path: /metrics
    static_configs:
      - targets:
        - 'db-svc.internal:9104'
    tls_config:
      ca_file: /etc/prometheus/ca/internal-ca.crt
      cert_file: /etc/prometheus/client.crt
      key_file: /etc/prometheus/client.key
      server_name: db-svc.internal
      min_version: TLS12

The cert_file and key_file are PEM-encoded. The key file must be readable only by the Prometheus process user.

Server-side TLS (the Prometheus HTTP listener)

# /etc/default/prometheus
ARGS="--config.file=/etc/prometheus/prometheus.yml \
      --storage.tsdb.path=/var/lib/prometheus \
      --web.listen-address=127.0.0.1:9090 \
      --web.config.file=/etc/prometheus/web.yml"

# /etc/prometheus/web.yml
tls_server_config:
  cert_file: /etc/prometheus/server.crt
  key_file: /etc/prometheus/server.key
  min_version: TLS12

The certificate and key are read once at start. After a certificate rotation, Prometheus must be restarted for the new certificate to be loaded; SIGHUP does not reload tls_server_config.

Server-side TLS with mTLS

For installations where every client of the Prometheus API authenticates itself with a client certificate.

# /etc/prometheus/web.yml
tls_server_config:
  cert_file: /etc/prometheus/server.crt
  key_file: /etc/prometheus/server.key
  client_ca_file: /etc/prometheus/client-ca.crt
  client_auth_type: RequireAndVerifyClientCert
  min_version: TLS12

Clients (Grafana, the on-call engineer’s browser) must present a certificate signed by client-ca.crt. This is the right shape for a high-trust environment; for most production installations the operational cost (every caller needs a client cert) outweighs the security benefit.

Remote write with TLS

# /etc/prometheus/prometheus.yml
remote_write:
  - url: 'https://prometheus-remote.internal/api/v1/write'
    basic_auth:
      username: remote-write
      password_file: /etc/prometheus/secrets/remote-write.pass
    tls_config:
      ca_file: /etc/prometheus/ca/internal-ca.crt
      server_name: prometheus-remote.internal
      min_version: TLS12

How to validate it

# READ-ONLY: confirm the TLS handshake succeeds against a target.
openssl s_client -connect app-svc.internal:9100 \
  -servername app-svc.internal \
  -CAfile /etc/prometheus/ca/internal-ca.crt \
  </dev/null 2>&1 | grep -E 'Verify return code|subject=|issuer='
# subject=CN = app-svc.internal
# issuer=O = Internal CA, CN = Internal CA Root
# Verify return code: 0 (ok)

# READ-ONLY: confirm Prometheus agrees.
curl -fsS https://app-svc.internal:9100/metrics | head -3
# # HELP go_gc_duration_seconds A summary of the GC invocation durations.
# # TYPE go_gc_duration_seconds summary
# go_gc_duration_seconds{quantile="0"} 1.23e-05

# READ-ONLY: confirm the negotiated TLS version and cipher.
openssl s_client -connect app-svc.internal:9100 \
  -tls1_2 </dev/null 2>&1 | grep -E 'Protocol|Cipher'
# Protocol  : TLSv1.2
# Cipher    : ECDHE-ECDSA-AES256-GCM-SHA384

# READ-ONLY: confirm Prometheus's own HTTPS listener (when enabled).
curl -fsSI https://prometheus.example.com/api/v1/targets
# HTTP/2 200
# strict-transport-security: max-age=...

# READ-ONLY: confirm the negotiated protocol on the Prometheus listener.
openssl s_client -connect prometheus.example.com:443 \
  -servername prometheus.example.com \
  </dev/null 2>&1 | grep -E 'Protocol|Verify return code'
# Protocol  : TLSv1.3
# Verify return code: 0 (ok)

# CONFIGURATION: validate the web config file before restarting.
promtool check web-config /etc/prometheus/web.yml
# /etc/prometheus/web.yml: VALID

A clean validation: every TLS handshake returns Verify return code: 0 (ok), the negotiated protocol is at least TLS 1.2, the configured min_version is reflected in the negotiation, and promtool check web-config confirms the YAML.

How it can fail

The five TLS failure modes that show up most often in production.

1. Server-cert expired, scrape fails with x509: certificate has expired. The targets page shows every affected target as up = 0. The fix is to renew the certificate at the target, not to add insecure_skip_verify. The wrong fix has been observed in real incidents and survived in the config for years.
2. ca_file references an old CA after a CA rotation. Prometheus validates against the old bundle, every target presents a cert from the new CA, every scrape fails. The fix is to update ca_file to the new bundle (or to the concatenation of old and new during a transition window) and reload Prometheus.
3. server_name mismatches the certificate SAN. The certificate is valid, the CA bundle is correct, but the SAN on the certificate is for app-svc.internal and the scrape URL is app-svc-1.internal. The fix is to either issue a certificate with multiple SANs or to scrape the name that appears in the SAN.
4. min_version: "TLS10" left in from a debugging session. The scrape works, the target accepts TLS 1.0. A compliance scan flags the configuration. The fix is to raise to TLS12 or omit the field (the default).
5. client_ca_file expires for the mTLS server side. Every client presents a certificate signed by the expired CA; every handshake fails with x509: certificate signed by unknown authority. The fix is to rotate the CA and reissue client certificates.

How to troubleshoot it

Diagnostic order: which side (server or client cert), which file is wrong, what does the error actually say.

1. Read the error string. x509: certificate signed by unknown authority means the CA bundle is wrong. x509: certificate is valid for app-svc.internal, not app-svc-1.internal means the SAN is wrong. tls: handshake failure with no further detail usually means the negotiated cipher suites do not overlap.
2. Reproduce with openssl s_client. The command-line openssl s_client -connect host:port -servername host -CAfile /path/to/ca.crt (the tool reads from stdin by default; close stdin with Ctrl-D when ready) shows the full TLS negotiation chain and the Verify return code. This is the fastest way to distinguish a certificate problem from a cipher problem from a name mismatch.
3. Inspect the certificate itself.

   openssl x509 -in /etc/prometheus/server.crt -noout \
     -subject -issuer -dates -ext subjectAltName
   # subject=CN = prometheus.example.com
   # issuer=O = Internal CA
   # notBefore=Aug 14 00:00:00 2026 GMT
   # notAfter=Aug 14 00:00:00 2027 GMT
   # X509v3 Subject Alternative Name:
   #     DNS:prometheus.example.com

4. Validate the web config. promtool check web-config /etc/prometheus/web.yml catches missing fields and bad paths before the restart.
5. Reload safely. A SIGHUP reloads the scrape config (including tls_config blocks). Changes to web.yml require a full restart. (CONFIGURATION.)

Security implications

TLS in Prometheus is not a single control. It is three controls layered:

• Confidentiality. Encryption of the metrics on the wire. Without TLS, an attacker on the same network reads the response of every scrape and the response of every query.
• Integrity. A TLS connection with server-cert validation prevents an attacker from rewriting the response. Without validation, the encryption is against an unknown peer.
• Authentication. mTLS (client certificates) binds the identity of the client to the connection. Without it, server-cert validation only binds the server to the identity.

The most common production mistake is to enable TLS but skip validation. The encryption is correct. The authentication is absent. The attacker who intercepts the connection can still rewrite the response if they hold a valid certificate for the target’s hostname.

Verification

You should now be able to answer:

• What does the tls_config block in a scrape config control?
• What does tls_server_config in the web configuration file control?
• Why is insecure_skip_verify: true not a fix for an expired certificate?
• What is the difference between NoClientCert, RequireClientCert, and RequireAndVerifyClientCert?

Quiz