TLS and Authentication

A team deploys Loki behind a Grafana data source URL of http://loki.internal:3100. The Grafana server sits behind a VPN. The data source works. Six months later, a new joiner brings a laptop with a permissive corp-network egress policy that routes through a transparent proxy. The proxy rewrites the cleartext traffic to a third-party observability vendor that happens to run a Loki-compatible endpoint. The proxy starts returning a 200 OK with the same shape as Loki. Grafana accepts the responses; the dashboards render with attacker-controlled data. The team’s customer-impact numbers are now influenced by data the team did not ask for.

TLS is not decoration on the observability platform. Every hop in the topology that crosses a trust boundary — every workload-to-collector, every collector-to-backend, every Grafana-to-data-source — must be authenticated, encrypted, or both. The lesson is about which boundary gets which treatment and why.

What TLS and authentication are

Two orthogonal properties:

• TLS is encryption and server authentication. The client confirms the server is who it claims to be (via certificate chain) and the wire is encrypted. Optional client authentication (mTLS) adds the server confirming the client is who it claims to be.
• Authentication is identity verification at the application layer. A bearer token, a basic-auth pair, an OIDC-derived header, or a tenant ID. TLS does not imply authentication; an encrypted channel can carry unauthenticated data.

The topology has three classes of hop:

   Internet   --TLS-->   Reverse proxy   --mTLS-->   Grafana
                                                  (server auth)
                                                  --mTLS--> data
                                                  sources (server
                                                  auth)

   Workload  --mTLS-->   OTel Collector   --mTLS-->   Tempo / Loki
   / SDK     (client auth)               (server auth)

The class of authentication on a hop is a function of who is on each side of the hop and what the threat model is.

Why a sysadmin cares

Three failure shapes appear when TLS or authentication is under-applied.

1. The cleartext Loki. A team runs Loki on port 3100 with no TLS. A new SRE opens a laptop on a coffee-shop WiFi; the VPN drops; the laptop retries Loki over the public DNS. The 3100 port is open because it is a backend service. Symptom: log lines leave the network in cleartext. The fix is to bind Loki to a private subnet and place a TLS-terminating proxy in front.
2. The bearer token in a Helm values file. A team puts LOKI_BEARER_TOKEN: super-secret-string in a values file committed to git. The token grants write access to one tenant’s logs. Symptom: the token is in the git history; rotation requires every consumer to be updated. The fix is a secret store (Vault, sealed-secrets) read at pod start.
3. The wildcard cert that covered staging and production. A wildcard *.observability.example.com covers both environments. A misconfigured DNS record points production traffic at a staging backend. TLS still succeeds because the wildcard matches. Symptom: production dashboards render staging data without any TLS warning. The fix is per- environment certificates issued by a per-environment CA.

How it works

The trust model on the observability platform has four boundaries, each with a different answer.

Boundary 1: workload to collector

The workload is on the same host as the collector agent (or the same pod in sidecar mode). The hop is loopback. The authentication is “anyone on the host can write to the agent.” The threat model is “an attacker on the same host can already do anything the application can do,” which is the threat model for every host anyway.

The right answer for the loopback hop is: bind the OTLP receiver to 127.0.0.1, no authentication, no TLS. The boundary is the host boundary, not the protocol.

Boundary 2: collector to backend

The collector fans out to Loki, Tempo, and Prometheus. The hop is the cluster network. The threat model is “a compromised pod on the cluster can attempt to write telemetry to the backends.” The right answer is mTLS: the backend presents a server certificate, the collector presents a client certificate, and the backends authorise writers by the client certificate CN or SPIFFE ID.

Bearer tokens are an alternative. They are simpler to operate but they have one failure mode mTLS does not: the token is a secret that must be rotated, distributed, and revoked. A collector that leaks the token (in a debug log, in an environment dump) leaks the entire tenant.

Boundary 3: Grafana to data source

Grafana queries the data sources on behalf of users. The hop is the cluster network. The right answer depends on the data source:

• Prometheus. A bearer token in the data source config (issued by the auth proxy in front of Prometheus, or by the Prometheus native auth if enabled). mTLS is overkill here.
• Loki. A bearer token in the data source config (issued by Loki’s auth proxy or by the Loki native auth). Tenant separation is encoded in the X-Scope-OrgID header which the data source forwards on every query.
• Tempo. Same shape as Loki. Tenant separation via X-Scope-OrgID.

Boundary 4: user to Grafana

The user reaches Grafana over HTTPS, terminated at the reverse proxy. The proxy passes the request to Grafana over loopback cleartext (Grafana binds to 127.0.0.1). Authentication is at the Grafana layer: OIDC, SAML, LDAP, or local users. TLS does not authenticate the user; it only authenticates the proxy. The lesson on reverse proxies covers this hop in detail.

How to configure it

The configuration for mTLS between the collector and Tempo:

# /etc/otelcol/gateway.yaml
exporters:
  otlp/tempo:
    endpoint: tempo.observability.svc:4317
    tls:
      insecure: false
      ca_file: /etc/otelcol/certs/ca.crt
      cert_file: /etc/otelcol/certs/client.crt
      key_file: /etc/otelcol/certs/client.key
      min_version: "1.3"
    sending_queue:
      enabled: true
      storage: file_storage

The ca_file is the CA that signed both the Tempo server cert and the collector client cert. The collector refuses to connect to any server whose certificate is not signed by this CA; the Tempo server refuses any client whose certificate is not signed by this CA.

The configuration for Tempo’s mTLS receiver:

# /etc/tempo/tempo.yaml
server:
  http_listen_port: 3200
  grpc_listen_port: 4317
  tls:
    enabled: true
    cert_file: /etc/tempo/certs/server.crt
    key_file: /etc/tempo/certs/server.key
    client_ca_file: /etc/tempo/certs/ca.crt
    client_auth_type: RequireAndVerifyClientCert
    min_version: "1.3"

auth_enabled: true

RequireAndVerifyClientCert is the strict mTLS mode: Tempo requires the client to present a certificate, validates it against the CA, and refuses the connection if the certificate is missing or invalid.

The configuration for Loki’s bearer-token authentication:

# /etc/loki/loki-config.yaml
auth_enabled: true

server:
  http_listen_port: 3100
  grpc_listen_port: 9095
  tls:
    enabled: true
    cert_file: /etc/loki/certs/server.crt
    key_file: /etc/loki/certs/server.key
    min_version: "1.3"

# A separate auth proxy (e.g. nginx with LUA) issues the
# X-Scope-OrgID header from the bearer token.

The Grafana data source that uses the bearer token:

# /etc/grafana/provisioning/datasources/loki.yaml
apiVersion: 1
datasources:
  - name: Loki
    type: loki
    url: https://loki.observability.svc:3100
    access: proxy
    jsonData:
      tlsAuth: true
      tlsAuthWithCACert: true
      tlsCACert: |
        -----BEGIN CERTIFICATE-----
        ...
        -----END CERTIFICATE-----
    secureJsonData:
      tlsClientCert: |
        -----BEGIN CERTIFICATE-----
        ...
        -----BEGIN CERTIFICATE-----
      tlsClientKey: |
        -----BEGIN EC PRIVATE KEY-----
        ...
        -----END EC PRIVATE KEY-----

Grafana’s mTLS data source uses tlsClientCert / tlsClientKey from secureJsonData (which is encrypted at rest in the Grafana database). The CA cert lives in jsonData because it is not secret.

How to validate it

# READ-ONLY: TLS handshake and certificate chain.
openssl s_client -connect tempo.observability.svc:4317 \
  -cert /etc/otelcol/certs/client.crt \
  -key /etc/otelcol/certs/client.key \
  -CAfile /etc/otelcol/certs/ca.crt \
  -verify_return_error </dev/null
# verify return: 0
# Verification: OK

# READ-ONLY: certificate expiry.
openssl x509 -in /etc/otelcol/certs/client.crt \
  -noout -enddate -subject -issuer
# notAfter=Sep 13 10:00:00 2026 GMT
# subject=CN = otel-collector-7d4f
# issuer=CN = observability-ca

# READ-ONLY: the server requires a client cert (mTLS strict).
curl -fsS https://tempo:4317/ready
# curl: (56) OpenSSL SSL_read: error:140890B2:SSL routines:...
# (connection refused because no client cert was provided)

# CONFIGURATION: the right client cert opens the connection.
curl --cert /etc/otelcol/certs/client.crt \
     --key /etc/otelcol/certs/client.key \
     --cacert /etc/otelcol/certs/ca.crt \
     -fsS https://tempo:4317/ready
# ready

# READ-ONLY: the data source health check passes in Grafana.
curl -fsS -u admin:admin \
  http://grafana:3000/api/datasources/uid/${LOKI_DS_UID}/health
# {"message":"ok","status":"success"}

# CONFIGURATION: validate the Grafana data source auth mode.
curl -fsS -u admin:admin \
  http://grafana:3000/api/datasources/uid/${LOKI_DS_UID} | jq .secureJsonFields
# {"tlsClientCert":1,"tlsClientKey":1}

A clean validation: the TLS handshake succeeds only with the correct client cert, the server cert is signed by the trusted CA, the cert is not within 30 days of expiry, and the data source health check passes from Grafana.

How it can fail

The four most common TLS failure modes, in order of how often they appear in incident reviews.

1. Expired certificate. A certificate issued for 365 days expires on day 366 with no rotation in place. Symptom: every TLS handshake fails with certificate has expired; the collector cannot connect to the backend; metrics, logs, and traces stop flowing.
2. Hostname mismatch. A certificate issued for tempo.internal is presented by tempo.observability.svc. The TLS handshake fails at hostname verification. Symptom: x509: certificate is valid for tempo.internal, not tempo.observability.svc. The fix is SAN entries that include both names, or a wildcard for the cluster domain.
3. Untrusted CA. The backend’s certificate is signed by the team’s internal CA; the collector’s ca_file is not updated to include the new CA after a CA rotation. Symptom: x509: certificate signed by unknown authority.
4. Missing client certificate. The collector is configured for mTLS but the client cert is not mounted (typo in the secret, missing volume mount). Symptom: the backend’s RequireAndVerifyClientCert rejects the connection; the collector logs tls: client didn't provide a certificate.
5. Bearer token in plaintext secret. A team’s loki-token.txt is committed to git. Symptom: the token is leaked to every developer with repo read; a malicious actor writes logs to the team’s tenant. The fix is rotation and a secret store.
6. Self-signed cert on a public endpoint. A team runs Loki with a self-signed certificate on a hostname in the public DNS. Browsers refuse to load Grafana; the data source health check fails. Symptom: x509: certificate signed by unknown authority from every curl.

How to troubleshoot it

The diagnostic order is “is the TLS handshake completing at all?”, “is the cert chain valid?”, “is the hostname correct?”, “is the client cert being presented?”.

1. Run openssl s_client against the failing endpoint with the expected client cert. The output names the failure class: “certificate verify failed,” “certificate has expired,” “no peer certificate available,” “hostname mismatch.”
2. Inspect the cert. openssl x509 -in <cert> -noout -text shows the subject, issuer, validity dates, and SAN entries. The first 30 seconds of any TLS incident is reading the cert.
3. Walk the trust chain. openssl verify -CAfile ca.crt server.crt confirms the chain. A “self-signed certificate in certificate chain” error usually means the CA bundle is missing an intermediate.
4. Check the client cert path. On the collector host, ls -la /etc/otelcol/certs/ shows whether the cert files are present. A missing file means a deployment bug, not a TLS bug.
5. Check the cert expiry. A scheduled query against the cert expiry metric (exposed by the blackbox exporter’s probe_ssl_earliest_cert_expiry) catches expired certs before they page.
6. Reproduce from inside the trust zone. From a host that has the right CA and the right client cert, the handshake should succeed. If it does, the failure is at the network hop or at the secret mount, not at the protocol.

Security implications

• TLS 1.2 is the floor, TLS 1.3 is the baseline. TLS 1.0 and 1.1 are deprecated; their cipher suites are weak. A modern profile is TLS 1.2 with ECDHE-ECDSA-AES256-GCM-SHA384 and TLS 1.3 with TLS_AES_256_GCM_SHA384 plus TLS_CHACHA20_POLY1305_SHA256.
• mTLS is the right default for collector-to-backend. Bearer tokens are an acceptable alternative but they are a secret that must be rotated. mTLS is a property of the certificate, not a secret in the wire.
• Certificates are short-lived. A 90-day certificate is the modern baseline. A 365-day certificate is a year-old mistake. Short-lived certificates limit blast radius on key compromise.
• The secret store is part of the topology. HashiCorp Vault, the cloud provider’s secret store, or sealed-secrets belongs in the topology diagram. A certificate that lives in a /etc/otelcol/certs/ directory managed by Ansible is a certificate that is rotated by hand, which is to say, not rotated.

Performance implications

• mTLS adds ~1 ms to the first handshake. Subsequent handshakes (after session resumption) are sub-millisecond.
• Session resumption matters at the fan-out. A gateway that fronts 500 agents and reschedules TLS sessions every minute pays the handshake cost 500 times per minute. TLS session tickets on the gateway, with the tickets rotated daily, amortise the cost.
• OCSP stapling avoids the per-handshake OCSP fetch. The gateway fetches the OCSP response once per session and staples it to the handshake; the client never makes a separate OCSP request.

Production guidance

• mTLS on every collector-to-backend hop. The collector presents a client cert; the backend validates it.
• Bearer token + mTLS for Grafana-to-data-source. The data source authenticates the user via the token; the token is encrypted at rest in the Grafana database.
• TLS 1.2 minimum, TLS 1.3 preferred. Pin the version; pin the cipher list; verify with a scan from the public internet perspective.
• 90-day certificate lifetime, automated rotation. The CA issues a new certificate every 60 days; the consuming service reloads without restart. Cert-manager for Kubernetes; Vault PKI for everything else.
• Monitor probe_ssl_earliest_cert_expiry. Alert at 30 days; page at 7 days; rotate at 14 days.

Verification

You should now be able to answer:

• Why is mTLS the right default for collector-to-backend hops and bearer tokens the right default for Grafana-to-data-source hops?
• What is the first command to run when a TLS handshake is failing?
• What is the right certificate lifetime and how is rotation automated?
• Where does the secret store belong in the topology?

Quiz