Tempo Receivers

A team adds OpenTelemetry instrumentation to a payment service and points the OTLP exporter at tempo:4317. The exporter reports ExporterExportingFailedPermanent after a few seconds. The Tempo logs show no traffic. The exporter was sending correctly-shaped protobuf frames to a port that nobody had bound. The Tempo binary was running in single-binary mode but the distributor.receivers block had not been enabled. The exporter saw a connection, but the connection was on a different process.

Receivers are the front door of Tempo. The shape of that front door (which protocols are accepted, on which ports, with what TLS) determines whether the rest of the system ever sees a span.

What it is

A Tempo receiver is a protocol listener inside the distributor that accepts spans from clients. Tempo currently supports four receiver kinds:

• OTLP — OpenTelemetry Protocol, the current default for new services. Available over gRPC (port 4317) and HTTP/protobuf (port 4318).
• Jaeger — Thrift over UDP (ports 6831/6832) and gRPC (port 14250). Still widely used by services that pre-date the OpenTelemetry SDK migration.
• Zipkin — Thrift and JSON over HTTP (port 9411). Legacy.
• Kafka — not a push receiver in the same sense; spans are read from a Kafka topic. Discussed in the Kafka ingress lesson; not configured under distributor.receivers.

Each receiver is a stanza under distributor.receivers in tempo.yaml. A receiver can be enabled or disabled independently of the others.

Why a sysadmin cares

The receiver choice is a fleet decision, not a per-service one. A service sending Zipkin Thrift on UDP cannot be asked to send OTLP/gRPC without code change. The receiver set on Tempo decides which client libraries can send data without an SDK migration.

It is also the boundary in the network. Receivers are the only components that accept unauthenticated traffic from application hosts (unless an auth proxy is in front). Misconfiguration here exposes the cluster to spam, data injection, and resource exhaustion.

How it works

   OpenTelemetry SDK                  Jaeger SDK               Zipkin SDK
   (OTLP gRPC :4317)                  (Thrift UDP :6831)        (HTTP :9411)
        |                                  |                        |
        v                                  v                        v
   +-----------+                     +-----------+             +-----------+
   | OTLP gRPC |                     | Jaeger    |             | Zipkin    |
   | receiver  |                     | receiver  |             | receiver  |
   +-----------+                     +-----------+             +-----------+
        |                                  |                        |
        +---------------+------------------+------------------------+
                        v
                  Tempo distributor
                  (in-memory trace buffers)
                        |
                        v
                  Ingester ring

The distributor sits in front of the ingester ring. Receivers deserialize the wire protocol into Tempo’s internal span format and push the spans onto the ingester. Authentication, if any, happens at the receiver boundary via the auth_context block.

OTLP

OTLP has two encodings on two ports:

• gRPC. Port 4317. Protobuf over HTTP/2. The default for service meshes and Kubernetes sidecars.
• HTTP. Port 4318. Protobuf over HTTP/1.1. Used by browser exporters and by services that cannot speak gRPC.

Jaeger

Three sub-receivers exist inside the Jaeger stanza:

• Thrift compact over UDP on port 6831. The original Jaeger client default. UDP means spans are dropped under load; not recommended for new services.
• Thrift binary over UDP on port 6832. Same caveat.
• gRPC on port 14250. The migration target for Jaeger SDK users; same wire efficiency as OTLP.

Zipkin

A single HTTP receiver on port 9411 accepts both Thrift and JSON spans. Use only for legacy migration.

Under the hood

How to configure it

OTLP gRPC + HTTP

# /etc/tempo/tempo.yaml — distributor receivers
distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
          max_recv_msg_size_mib: 16
        http:
          endpoint: 0.0.0.0:4318

  # Optional authentication. Tokens issued by an upstream auth
  # service are passed in via gRPC metadata.
  auth_context:
    extractors:
      - name: org-id
        from: header
        key: X-Scope-OrgID

Severity: CONFIGURATION. Restart the distributor to apply.

The max_recv_msg_size_mib default is 16 MiB. Services emitting large spans (e.g. spans with thousands of attributes) may hit resource exhausted. Raise it.

Jaeger

distributor:
  receivers:
    jaeger:
      protocols:
        grpc:
          endpoint: 0.0.0.0:14250
        thrift_compact:
          endpoint: 0.0.0.0:6831
        thrift_binary:
          endpoint: 0.0.0.0:6832

Severity: SERVICE-IMPACT. UDP receivers do not honour max_recv_msg_size_mib in the same way; tune the kernel UDP buffer (net.core.rmem_max) for high-volume UDP traffic.

Zipkin

distributor:
  receivers:
    zipkin:
      endpoint: 0.0.0.0:9411

TLS on receivers

Tempo terminates TLS at the receiver for the OTLP HTTP and gRPC receivers. The Jaeger gRPC receiver accepts TLS the same way.

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
          tls:
            cert_file: /etc/tempo/tls/tempo.crt
            key_file: /etc/tempo/tls/tempo.key
            client_ca_file: /etc/tempo/tls/ca.crt   # mTLS

Severity: CONFIGURATION. The cert and key paths must be readable by the Tempo process; permissions of 0640 owned by tempo:tempo are typical.

How to validate it

Severity: READ-ONLY.

# Substitute your own value before running:
TRACE_ID=4bf92f3577b34da6a3ce929d0e0e4736

# 1. Confirm the listener is bound (OTLP gRPC)
ss -tlnp | grep -E '4317|4318|14250|9411|6831|6832'

# 2. Confirm the Tempo distributor is registered
curl -s http://tempo:3200/ready

# 3. Send a synthetic OTLP gRPC span with otel-cli (or any SDK)
otel-cli span export \
  --endpoint tempo:4317 \
  --service-name validate-receiver \
  --name "synthetic check" \
  --kind server

# 4. Confirm the trace by ID was accepted
curl -s "http://tempo:3200/api/traces/$TRACE_ID" \
  | jq -r '.resourceSpans[0].scopeSpans[0].spans[0].name'

Real output:

$ ss -tlnp | grep 4317
LISTEN 0  128  0.0.0.0:4317  0.0.0.0:*  users:(("tempo",pid=8421,fd=12))

$ curl -s http://tempo:3200/ready
ready

$ otel-cli span export --endpoint tempo:4317 --service-name validate-receiver --name "synthetic check"
trace_id=4bf92f3577b34da6a3ce929d0e0e4736

$ curl -s http://tempo:3200/api/traces/4bf92f3577b34da6a3ce929d0e0e4736 \
    | jq -r '.resourceSpans[0].scopeSpans[0].spans[0].name'
"synthetic check"

# Validate the TLS handshake with openssl
openssl s_client -connect tempo:4317 -servername tempo \
  -CAfile /etc/tempo/tls/ca.crt < /dev/null 2>&1 \
  | grep -E 'subject=|issuer=|Verification'

Real output:

subject=CN = tempo.observability.svc
issuer=CN = Internal CA G2
Verification: OK

How it can fail

1. The OTLP exporter sends to port 4317 but the receiver only has HTTP enabled. The connection opens (the OS accepts it), but the gRPC handshake fails with UNIMPLEMENTED. The exporter retries; metrics show tempo_distributor_dropped_spans_total rising and the last_error is “method not implemented”. Symptom: client log spam; zero spans in storage.

2. Jaeger Thrift over UDP cannot keep up at peak. The kernel UDP buffer fills; new spans are silently dropped. Symptom: dmesg shows UDP: bad checksum or kernel: ... dropwatch warnings; trace UI shows sparse data during peak hours but full data during off-peak.

3. TLS enabled but client_ca_file path is wrong. Tempo starts but the gRPC handshake fails with x509: certificate signed by unknown authority at the client. The Tempo log shows transport: authentication handshake failed. Symptom: all TLS clients rejected; plain-text clients still accepted.

4. max_recv_msg_size_mib too low for a service with rich attributes. The client sees rpc error: code = ResourceExhausted desc = grpc: received message larger than max. Symptom: a single noisy service drops 100% of spans; everything else is fine.

5. Receiver enabled on a querier-only process. In microservices mode, the receiver is configured on the distributor. A copy-paste of the same tempo.yaml to the querier pod does not break the querier, but the gRPC ports are not bound there. Symptom: clients connecting to the querier pod get connection refused; connections to the distributor pod succeed.

6. Auth context extractor typo. A header of X-Scope-OrgID is requested but the upstream proxy sends X-Scope-OrgId (a different case). Symptom: all spans land in the anonymous tenant, mixing production data with every other tenant.

How to troubleshoot it

Order of diagnostics, cheapest first:

1. Is the listener bound? ss -tlnp | grep 4317. If the port is not in the list, the receiver stanza did not parse, or the process is not the distributor.
2. Did the receiver stanza parse? journalctl -u tempo | grep -i receivers or the Tempo stdout for unknown receivers key. A misspelling like otlp vs otlpgrpc silently ignores the stanza.
3. Is the connection reaching Tempo? curl -v http://tempo:4318/v1/traces -d '\{\}' -H 'Content-Type: application/x-protobuf'. A 400 from Tempo means the receiver is alive but the payload was rejected.
4. Is the ingester accepting? Look at tempo_distributor_spans_received_total minus tempo_distributor_dropped_spans_total. A non-zero delta confirms the receiver is forwarding.

Security implications

• Receiver surface. Every enabled receiver is an unauthenticated ingress unless an auth context or upstream proxy is in place. Disable receivers that the fleet does not need.
• UDP receivers. Jaeger Thrift UDP accepts any payload from any source. An attacker who can reach the receiver can flood the ingester with synthetic spans and exhaust the max_traces_per_user budget. Block UDP at the network layer if not in use.
• TLS. OTLP gRPC supports TLS natively. The receiver stanza must reference a certificate and key. mTLS (client_ca_file) is appropriate for service-mesh deployments where client identity is enforced.

Performance implications

• gRPC vs HTTP/2. OTLP gRPC uses HTTP/2 with header compression; OTLP HTTP uses HTTP/1.1. gRPC is materially cheaper at high volumes.
• UDP. UDP receivers offload the connection state to the kernel. At 100k spans/sec, the kernel UDP buffer is the bottleneck, not Tempo. Raise net.core.rmem_max and net.core.rmem_default.
• TLS. TLS adds ~5-10% CPU at typical span sizes. The receiver CPU is rarely the bottleneck; the ingester is.

Production guidance

• Enable OTLP gRPC and HTTP. Disable Jaeger and Zipkin once migration is complete — every active receiver is a load and attack surface.
• Set max_recv_msg_size_mib to 16 (default) and raise only on per-service evidence; large messages slow the distributor.
• Run receivers only on the distributor. A common misconfiguration is to copy tempo.yaml to every pod; receivers on the querier are wasted ports.

Verification

You should now be able to answer:

• Which port does OTLP gRPC use by default, and which port does OTLP HTTP use?
• Where in tempo.yaml are receivers configured?
• What is the difference between protocols.grpc and protocols.thrift_compact under the Jaeger stanza?
• Which two settings control the TLS handshake for the OTLP receiver?
• What does ResourceExhausted mean when an OTLP client sees it from Tempo?

Quiz