Tempo TLS

A Tempo cluster’s OTLP gRPC port is reachable from the Kubernetes pod network because the service has a ClusterIP but no NetworkPolicy. A developer runs nc -z tempo 4317 from a test pod; the connection succeeds. The same developer runs otel-cli span export --endpoint tempo:4317; the export succeeds without TLS. The cluster has been accepting unauthenticated spans from any pod in the cluster for months. A noisy multitenancy cluster fires hundreds of test pods that all push traces; the distributor’s per-tenant budget exhausts in seconds. The data is debug noise, but the data is also plaintext on the wire; a tcpdump on the OTLP port reveals that any pod with cluster network access can capture the spans.

TLS at the receiver boundary is the difference between a secure pipeline and an open one. The lesson is about making the boundary actually close.

What Tempo TLS means

Tempo terminates TLS at the receiver boundary on the distributor. The configuration knob is the tls sub-block inside each protocol stanza:

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          tls:
            cert_file:       /etc/tempo/tls/tempo.crt
            key_file:        /etc/tempo/tls/tempo.key
            client_ca_file:  /etc/tempo/tls/internal-ca.crt

Three things happen with this configuration:

• Server identity. Tempo presents a certificate to the client. The client must trust the issuer (CA) or the hostname (SAN) for the connection to succeed.
• Encryption in transit. The OTLP payload is encrypted between the collector and Tempo. A passive observer of the wire sees only ciphertext.
• Optional mTLS. With client_ca_file, Tempo validates the client certificate against the CA. A client without a certificate signed by the CA is refused.

The same shape applies to the OTLP HTTP receiver and to the Jaeger gRPC receiver. Jaeger Thrift UDP and Zipkin HTTP do not have a TLS sub-block in current Tempo releases — UDP cannot carry TLS, and the Zipkin legacy path is plain HTTP.

Why a sysadmin cares

Two reasons.

1. The wire is the last place secrets can leak. A span payload on the wire includes the resource attributes, the span attributes (which the SDK has emitted), the trace ID, and the parent span ID. With attributes/redact at the collector, the span payload reaching Tempo is reasonably redacted. Without TLS, the payload is observable in transit between the collector and Tempo.
2. The OTLP port is a high-risk service-internal endpoint. It accepts frames from any client that reaches the port. Without mTLS, a misconfigured network policy lets any client on the pod network push traces into the cluster.

How it works — the cert shape

A tempo TLS configuration needs three files:

• Server certificate. Tempo’s identity. Issued by a CA the clients trust. The cert’s Subject Alternative Name (SAN) includes the DNS name the collector uses to reach Tempo: tempo.observability.svc.cluster.local for in-cluster, or a fully-qualified name for external.
• Server private key. Tempo’s key. Permissions 0640, owned by the Tempo service user. Never committed to source control.
• Client CA bundle (optional, for mTLS). The CA that signs client certificates. Tempo will refuse a client certificate that does not chain to this CA.

The cert rotation cadence is operational. With cert-manager on Kubernetes, the cert is rotated by the cert-manager controller; Tempo picks the rotated cert on the next reload.

A typical production shape uses service-mesh-issued certs:

• Service-mesh CA signs both client and server certs.
• Tempo serves a cert whose SAN includes the Tempo service DNS name.
• The collector presents a cert whose URI includes the collector service-account name.
• Both sides pin the service-mesh CA as the trust anchor.

This shape gives mTLS without per-deployment cert-management code. The service mesh rotates the leaf certs on a short cadence (hours), and the CA bundle is the only long- lived secret.

How to configure it

Tempo: server TLS only

# /etc/tempo/tempo.yaml
auth_enabled: true

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
          tls:
            cert_file: /etc/tempo/tls/tempo.crt
            key_file:  /etc/tempo/tls/tempo.key
        http:
          endpoint: 0.0.0.0:4318
          tls:
            cert_file: /etc/tempo/tls/tempo.crt
            key_file:  /etc/tempo/tls/tempo.key

The collector’s exporter pins the Tempo CA as trusted:

# /etc/otelcol/config.yaml
exporters:
  otlp/tempo:
    endpoint: tempo.observability.svc:4317
    tls:
      ca_file: /etc/tempo/tls/internal-ca.crt
      cert_file: /etc/otelcol/tls/collector.crt
      key_file: /etc/otelcol/tls/collector.key

Severity: CONFIGURATION. Restart Tempo and reload the collector to apply.

Tempo: mTLS

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
          tls:
            cert_file:      /etc/tempo/tls/tempo.crt
            key_file:       /etc/tempo/tls/tempo.key
            client_ca_file: /etc/tempo/tls/internal-ca.crt

The client_ca_file enables mTLS. Every client that does not present a cert signed by the CA is refused at the handshake. Plaintext connections are also refused; the receiver does not fall through to unauthenticated.

Fronting proxy: TLS terminates at nginx

The right alternative to TLS-at-receiver is to terminate TLS at a reverse proxy in front of Tempo. Tempo loopback-binds the OTLP port; the proxy holds the cert:

# /etc/nginx/sites-available/tempo.conf
upstream tempo_upstream {
  server 127.0.0.1:4317 max_fails=3 fail_timeout=10s;
  keepalive 32;
}

server {
  listen 443 ssl;
  server_name tempo.example.com;

  ssl_certificate     /etc/letsencrypt/live/tempo.example.com/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/tempo.example.com/privkey.pem;

  # Reject clients that present no cert (mTLS).
  ssl_verify_client      optional_no_ca;
  ssl_client_certificate /etc/nginx/tls/internal-ca.crt;

  location / {
    proxy_pass http://tempo_upstream;
    proxy_set_header X-Scope-OrgID $http_x_scope_orgid;
  }
}

Tempo binds 0.0.0.0:4317 with tls: undefined; only the proxy reaches it. The proxy enforces mTLS; Tempo does not.

File permissions and key handling

# Tempo's TLS directory.
ls -la /etc/tempo/tls
# -rw-r--r-- 1 tempo tempo  5671  tempo.crt
# -rw-r----- 1 tempo tempo  3243  tempo.key         # 0640
# -rw-r--r-- 1 tempo tempo  2049  internal-ca.crt

# The cert and CA are world-readable; the key is tempo-readable.
# The CA bundle is the trust anchor; it does not need
# confidentiality, only integrity.

How to validate it

Severity: READ-ONLY.

# 1. The handshake.
openssl s_client -connect tempo:4317 -servername tempo \
  -CAfile /etc/tempo/tls/internal-ca.crt < /dev/null 2>&1 \
  | grep -E 'subject=|issuer=|Verification|Protocol'
# subject=CN = tempo.observability.svc.cluster.local
# issuer=CN = Internal CA G2
# Verification: OK
# Protocol  : TLSv1.3

# 2. Cipher and protocol.
openssl s_client -connect tempo:4317 -tls1_2 \
  -CAfile /etc/tempo/tls/internal-ca.crt < /dev/null 2>&1 \
  | grep -E 'Protocol|Cipher'
# Protocol  : TLSv1.2
# Cipher    : ECDHE-RSA-AES256-GCM-SHA384

# 3. Client cert validation (mTLS).
openssl s_client -connect tempo:4317 \
  -cert /etc/otelcol/tls/collector.crt \
  -key  /etc/otelcol/tls/collector.key \
  -CAfile /etc/tempo/tls/internal-ca.crt < /dev/null 2>&1 \
  | grep -E 'Verification|Verify return code'
# Verification: OK

# 4. mTLS rejection: a cert that does not chain to the CA.
openssl s_client -connect tempo:4317 \
  -cert /tmp/untrusted.crt -key /tmp/untrusted.key \
  -CAfile /etc/tempo/tls/internal-ca.crt < /dev/null 2>&1 \
  | grep -E 'verify|Verify return code'
# verify error:num=27:certificate verify failed
# Verify return code: 21 (unable to verify the first certificate)

# 5. Plaintext rejection.
grpc_cli ls tempo:4317  # (with grpcurl)
# Fails: connection closed by remote.

# 6. End-to-end: the OTLP exporter over TLS.
otel-cli span export --endpoint tempo:4317 \
  --tls --ca-file /etc/otelcol/tls/internal-ca.crt \
  --service-name validate-tls --name "synthetic tls check"
# Reported trace_id; the previous step works.

A clean validation: openssl s_client succeeds against the Tempo receiver with the expected SAN; an untrusted client cert is rejected; a plaintext connection is refused; the OTLP exporter ships a span over TLS and the trace is queryable.

How it can fail

Five recurring shapes from real audits.

1. Cert path is wrong. A copy-paste of the config from another service references /etc/tempo/cert.pem; the actual path is /etc/tempo/tls/tempo.crt. Symptom: Tempo fails to start, log shows “open /etc/tempo/cert.pem: no such file or directory”. Fix: correct the path; the log message names the expected file.
2. Cert SAN does not include the collector’s DNS name. The cert is for tempo.example.com; the collector connects to tempo.observability.svc.cluster.local. Symptom: the client reports x509: certificate is valid for tempo.example.com, not for tempo.observability.svc.cluster.local. Fix: reissue the cert with the in-cluster SAN, or have the collector connect to the FQDN.
3. client_ca_file configured but clients use self-signed certs. The CA bundle does not include the clients’ CA. Symptom: every client handshake fails. Fix: add the client CA to the bundle; do not disable mTLS to fix it.
4. Cert expired. The cert-manager controller is down; the cert expired. Symptom: the OTLP exporter’s otelcol_exporter_sent_spans_total stays flat; the wire log shows handshake errors with x509: certificate has expired. Fix: alert on certs_not_after seconds for the Tempo cert; rotate it via cert-manager.
5. TLS configured on the gRPC receiver but not on the HTTP receiver. A service running in mixed mode can connect to 4318 over plaintext. Symptom: a tcpdump on 4318 shows OTLP HTTP bodies; Tempo has no way to know they were transmitted unencrypted. Fix: always pair TLS on both protocols.

How to troubleshoot it

The diagnostic order:

1. What is the Tempo log for the receiver? Restart with -config.file=tempo.yaml; the receiver stanza errors surface at startup.
2. Does the cert chain resolve locally? openssl crl2pkcs7 -nocrl -certfile tempo.crt | openssl pkcs7 -print_certs | grep -E '^subject=|^issuer='. A mismatch between issuer and the trust CA is the root cause.
3. Does the SAN match the endpoint? openssl x509 -in tempo.crt -noout -text | grep -A 1 "Subject Alternative Name". The DNS entries must include the connect target.
4. Can the process read the key file? stat /etc/tempo/ tls/tempo.key and confirm the user under which Tempo runs. A permission denied is the cause of a silent TLS-disabled startup.
5. Does the test path actually go through TLS? grpcurl -server-name tempo.example.com tempo:4317 list succeeds. A connection closed is TLS rejecting the plaintext; a connection refused is the port not bound.

Security implications

• The wire is observable. A tcpdump on the OTLP port sees every span that crosses. With TLS, the dump shows only ciphertext; without, it sees the payload.
• mTLS validates both sides. Server-only TLS proves the server to the client; mTLS proves the client to the server. Service-mesh-backed collector pods are the standard shape for mTLS.
• Cert rotation is a perimeter control. An expired cert is a forced outage; an alert on not_after is a perimeter alarm.
• TLS at the proxy is the alternative. The same TLS profile at nginx in front of loopback-bound Tempo. The cert lifecycle still belongs to the platform team.
• Pinned certificate authorities. A Tempo that trusts only the internal CA bundle is harder for an attacker to impersonate than one that trusts the system root store.

Performance implications

• Handshake cost. A TLS handshake is two round trips in TLS 1.3, three in TLS 1.2. With connection reuse (HTTP/2, gRPC keepalive), the per-request cost is negligible.
• CPU on the receiver. AES-GCM at typical span sizes is in the single-digit nanoseconds per byte. The receiver CPU is rarely the bottleneck.
• CPU on the client. Same: the SDK CPU is rarely the bottleneck.

Production guidance

• Always TLS. Never ship a Tempo OTLP port on plaintext even on a private network. The blast radius of a leaked packet dump of trace data is the same regardless of the packet’s path.
• mTLS when the collector identity is service-mesh- issued. Cert-manager for the server cert; mesh CA for the client cert.
• Monitor the cert expiry. A Prometheus alert on the cert’s not_after minus now with a 14-day warning and 1-hour page.
• Loopback-bind when TLS is at the proxy. Tempo with the tls: stanza undefined on OTLP, listening on 127.0.0.1; the proxy holds the cert and the policy.
• Pair TLS on gRPC and HTTP together. Half-encrypted boundaries are a finding in their own right.

Verification

You should now be able to answer:

• Which three files are required for TLS on a Tempo receiver, and what does each one do?
• What is the difference between server-only TLS and mTLS, and when is mTLS appropriate?
• How do you confirm a hand-rolled Tempo certificate chain with openssl?
• Why is TLS appropriate even on a private pod network?
• When is the right TLS termination Tempo, and when is it the reverse proxy?

Quiz