TLS Across the Platform

A Loki ingest cluster has been running for fourteen months. The TLS certificates were generated by an internal CA at install time; the certificates expire in thirty days. The on-call engineer opens the renewal calendar entry on a Monday morning. The internal CA certificate expired a week ago; the renewal script that the previous operator wrote was a cron job that read a file from a network share that no longer exists. The cron job has been logging “ERROR: cannot renew certificate” for seven days. The certificates will expire in twenty-three days. The fix is not yet in flight.

This is what the word TLS means in an observability context: the encryption that protects traffic between every component. The lesson is about the right posture per component, and the certificate lifecycle that keeps the posture from decaying.

What it is

TLS (Transport Layer Security) is the protocol that encrypts traffic between two endpoints. In the observability stack, four TLS shapes appear:

   Shape               | What it does                  | Where it fits
   --------------------+-------------------------------+-----------------------
   One-way TLS         | Server presents a certificate | Internal data plane
   (server cert only)  | Client trusts the CA that     | (Prometheus, Loki,
                       | signed it                     | Tempo, OTel Collector)
                       |                               |
   Mutual TLS          | Server and client each        | Service-to-service
   (client + server    | present a certificate         | where certificates
    certificates)      | Both sides validate the       | already exist
                       | peer's certificate            |
                       |                               |
   Reverse-proxy TLS   | A TLS-terminating proxy       | User-facing UI
                       | speaks TLS to the client and  | (Grafana)
                       | cleartext to the backend      |
                       |                               |
   No TLS              | Cleartext traffic             | Loopback only
                       |                               |

The right shape is different for every connection. The user-facing surface uses reverse-proxy TLS. The internal data plane uses one-way TLS or mutual TLS. The loopback uses no TLS.

Why a sysadmin cares

Three production failure modes map directly to wrong TLS choices.

1. An expired certificate on the data plane. The data plane TLS certificates are rotated by a script that no one has looked at in a year. The certificate expires. Every scrape fails with x509: certificate has expired. The blast radius is “every dashboard goes blank at the same time.”
2. A self-signed certificate on the user-facing UI. The reverse proxy is configured with a self-signed certificate. Every browser shows a certificate warning. Users learn to click through the warning. The blast radius is “every user has been trained to ignore certificate warnings.”
3. A hostname mismatch between the certificate and the endpoint. The certificate is for loki.internal.example.com but the endpoint is reached as loki. The connection succeeds with curl -k and fails with curl without the flag. The blast radius is “every legitimate client rejects the connection.”

How it works

Every component reads a certificate and a private key from disk, presents the certificate during the TLS handshake, and validates the peer’s certificate against a CA bundle. The shape of the validation depends on the TLS shape.

   Client                                Server
   ------                                ------
     |  -- TLS ClientHello --------------->  |
     |                                       |
     |  <-- TLS ServerHello + Certificate - |
     |                                       |
     |  -- Client validates server cert ---  |
     |     against trusted CA bundle         |
     |                                       |
     |  -- TLS Finished --------------------> |
     |                                       |
     |  -- HTTP request over TLS -----------> |
     |                                       |
     |  <-- HTTP response over TLS --------- |

For mutual TLS, the server presents a certificate authority bundle to the client; the client presents its own certificate; the server validates it.

How to configure it

Reverse-proxy TLS for Grafana (the user-facing UI)

# /etc/nginx/sites-available/grafana.conf
server {
  listen 443 ssl;
  listen [::]:443 ssl;
  http2 on;
  server_name grafana.example.com;

  ssl_certificate     /etc/letsencrypt/live/grafana.example.com/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/grafana.example.com/privkey.pem;

  ssl_protocols TLSv1.2 TLSv1.3;
  ssl_ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305;
  ssl_prefer_server_ciphers off;
  ssl_session_cache shared:SSL:10m;
  ssl_session_timeout 1d;
  ssl_session_tickets off;

  add_header Strict-Transport-Security "max-age=15768000" always;

  location / {
    proxy_set_header Host $host;
    proxy_pass http://127.0.0.1:3000;
  }
}

server {
  listen 80;
  listen [::]:80;
  server_name grafana.example.com;
  return 301 https://$host$request_uri;
}

# /etc/grafana/grafana.ini
[server]
protocol = http
cert_file =
key_file =

Grafana speaks cleartext on loopback; the reverse proxy terminates TLS. The certificate lifecycle is managed by Let’s Encrypt (or an internal CA) and the renewal is a cron job.

One-way TLS for Prometheus

# /etc/prometheus/web_config.yml
basic_auth_users:
  admin: $2y$10$bcrypt-hash-of-password

tls_server_config:
  cert_file: /etc/prometheus/certs/prometheus.crt
  key_file: /etc/prometheus/certs/prometheus.key

# Optional: require client certificates (mTLS)
tls_client_config:
  cert_file: /etc/prometheus/certs/prometheus-ca.crt

# /etc/systemd/system/prometheus.service
[Service]
ExecStart=/usr/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --web.config.file=/etc/prometheus/web_config.yml

One-way TLS for Loki

# /etc/loki/loki-config.yaml
server:
  http_listen_address: 10.0.10.6:3100
  grpc_listen_address: 10.0.10.6:9096
  http_tls_config:
    cert_file: /etc/loki/certs/loki.crt
    key_file: /etc/loki/certs/loki.key
  grpc_tls_config:
    cert_file: /etc/loki/certs/loki.crt
    key_file: /etc/loki/certs/loki.key

One-way TLS for Tempo

# /etc/tempo/tempo.yaml
server:
  http_listen_address: 10.0.10.7:3200
  grpc_listen_address: 10.0.10.7:9095
  http_tls_config:
    cert_file: /etc/tempo/certs/tempo.crt
    key_file: /etc/tempo/certs/tempo.key

Mutual TLS for OpenTelemetry Collector

# /etc/otelcol/config.yaml
extensions:
  tls:
    insecure: false
    cert_file: /etc/otelcol/certs/otelcol.crt
    key_file: /etc/otelcol/certs/otelcol.key

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 10.0.10.20:4317
        tls:
          cert_file: /etc/otelcol/certs/otelcol.crt
          key_file: /etc/otelcol/certs/otelcol.key
          client_ca_file: /etc/otelcol/certs/client-ca.crt

exporters:
  otlp/tempo:
    endpoint: tempo.internal.example.com:4317
    tls:
      cert_file: /etc/otelcol/certs/otelcol.crt
      key_file: /etc/otelcol/certs/otelcol.key
      insecure_skip_verify: false

service:
  extensions: [tls]
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlp/tempo]

The OpenTelemetry Collector extension tls provides mTLS configuration for both the receiver and the exporter. The client CA bundle validates the client certificate; the server certificate is presented to the client.

Mutual TLS for Grafana Alloy

# /etc/alloy/config.alloy
tls "default" {
  cert_file = "/etc/alloy/certs/alloy.crt"
  key_file  = "/etc/alloy/certs/alloy.key"
}

prometheus.remote_write "default" {
  endpoint {
    url = "https://prometheus.internal.example.com:9090/api/v1/write"
    tls_config {
      ca_file   = "/etc/alloy/certs/ca.crt"
      cert_file = "/etc/alloy/certs/alloy.crt"
      key_file  = "/etc/alloy/certs/alloy.key"
    }
  }
}

How to validate it

# READ-ONLY: confirm the Grafana reverse proxy presents a valid certificate.
curl -fsSI https://grafana.example.com/api/health
# HTTP/2 200
# strict-transport-security: max-age=15768000

# READ-ONLY: confirm Prometheus TLS is in place.
openssl s_client -connect prometheus.internal.example.com:9090 \
  -servername prometheus.internal.example.com </dev/null 2>&1 | grep -E 'subject|issuer|verify return'
# subject=CN = prometheus.internal.example.com
# issuer=CN = Internal CA
# verify return: 1

# READ-ONLY: confirm Loki TLS is in place.
openssl s_client -connect loki.internal.example.com:3100 \
  -servername loki.internal.example.com </dev/null 2>&1 | grep -E 'subject|verify return'
# subject=CN = loki.internal.example.com
# verify return: 1

# READ-ONLY: confirm the OpenTelemetry Collector requires a client certificate.
openssl s_client -connect otel.internal.example.com:4317 \
  -cert /etc/otelcol/certs/client.crt \
  -key /etc/otelcol/certs/client.key </dev/null 2>&1 | grep 'Verify return code'
# Verify return code: 0 (ok)

# READ-ONLY: confirm a TLS handshake without a client certificate is rejected.
openssl s_client -connect otel.internal.example.com:4317 </dev/null 2>&1 | grep 'Verify return code'
# Verify return code: 21 (unable to verify the first certificate)
# ... or a handshake failure with no certificate presented

# READ-ONLY: confirm certificate expiry is monitored.
promtool tsdb analyze /var/lib/prometheus | head -5
# (Prometheus's own certificate expiry is not exposed via promtool;
#  the right check is a blackbox-exporter probe against the TLS port.)

# CONFIGURATION: rotate a certificate with cert-manager (Kubernetes).
kubectl annotate certificate prometheus-tls cert-manager.io/issue-temporary-certificate="true"

A clean validation: every TLS endpoint presents a valid certificate, the certificate matches the hostname, the OpenTelemetry Collector rejects connections without a client certificate, and the certificate expiry is monitored with a blackbox-exporter probe.

How it can fail

The high-frequency TLS failure modes from real incidents.

1. Expired certificate. The certificate was issued for one year; the renewal cron job failed; the certificate expired. The visible symptom is curl returning x509: certificate has expired and every scrape failing at the same time.
2. Hostname mismatch. The certificate is for loki.internal.example.com but the client connects to loki. The visible symptom is curl returning x509: certificate is not valid for loki.
3. Self-signed certificate on the user-facing UI. The reverse proxy is configured with a self-signed certificate. The visible symptom is every browser showing a certificate warning.
4. Untrusted CA. The client does not trust the CA that signed the server certificate. The visible symptom is curl returning x509: certificate signed by unknown authority.
5. Insecure skip verify enabled. A client is configured with insecure_skip_verify: true because the test environment uses a self-signed certificate. The setting is copied to production. The visible symptom is a successful connection to a server that should have been rejected.
6. Mixed TLS / cleartext on the same endpoint. The component accepts both http:// and https:// requests on the same port. The visible symptom is a successful cleartext scrape against a port that should be TLS-only.

How to troubleshoot it

The diagnostic order is “is the certificate valid?”, “does the certificate match the hostname?”, “does the client trust the CA?”, “is the certificate about to expire?”.

1. Inspect the certificate with openssl s_client -connect <host>:<port> -servername <hostname>. The output includes the subject, the issuer, the expiry, and the verify return code.
2. Inspect the chain with openssl s_client -showcerts. A missing intermediate certificate produces a chain that does not reach the trusted root.
3. Check the expiry with openssl x509 -enddate -noout -in <cert>. A certificate that expires in less than thirty days is a finding.
4. Check the hostname with openssl x509 -text -noout -in <cert> | grep DNS. A certificate that does not include the hostname the client uses is a finding.
5. Check the CA trust on the client with openssl verify -CAfile <ca-bundle> <cert>. A certificate that fails verification is a finding.
6. Inspect the certificate rotation log (cert-manager, Vault PKI, or the external CA). A rotation that failed is a finding.

Security implications

• TLS is the boundary that decides whether traffic is readable by an attacker who can see the network. The right posture is TLS on every connection that leaves a host.
• Certificate management is the lifecycle that keeps the posture from decaying. A certificate that expires is a finding that the renewal process caught. A certificate that does not expire is a finding that the renewal process is silent.
• Hostname validation is the boundary that decides whether the right server received the traffic. A certificate that does not match the hostname is a finding.
• Mutual TLS is the strongest authentication on the service-to-service plane. A client certificate is bound to the host; the cost is certificate management.

Performance implications

• TLS handshake cost is on the proxy. Reusing connections via keepalive and ssl_session_cache cuts the per-request handshake cost to almost zero.
• Mutual TLS handshake is two to three times slower than one-way TLS. The cost is on connection establishment, not on every request.
• TLS encryption cost on modern hardware is negligible. AES-NI makes bulk encryption a small fraction of the request path.

Production guidance

• Reverse-proxy TLS for the user-facing UI (Grafana). The proxy terminates TLS; the Grafana process speaks cleartext on loopback.
• One-way TLS for the internal data plane (Prometheus, Loki, Tempo). The server presents a certificate; the client validates it.
• Mutual TLS for the service-to-service plane where certificates already exist (a service mesh, a Kubernetes PKI, a Vault PKI engine).
• Manage certificates with cert-manager, Vault PKI, or an external CA. The operator does not write the renewal cron job.
• Monitor certificate expiry with a blackbox-exporter probe. A certificate that expires in less than thirty days is a finding.
• Audit the certificate chain at least once per quarter. A certificate that includes the wrong hostname is a finding.

Verification

You should now be able to answer:

• What is the right TLS shape for the user-facing UI (Grafana), and why is it different from the data plane?
• What is the difference between one-way TLS and mutual TLS, and where is each the right choice?
• Why is certificate management a lifecycle discipline, and what is the role of cert-manager or Vault PKI?
• What is the failure shape of an expired certificate, and how do you detect it before it fires?
• Why is insecure_skip_verify: true the most expensive setting in any TLS configuration, and what is the right replacement?

Quiz