Receiver Compatibility

A team runs Tempo 2.6 with Jaeger and OTLP receivers enabled on the distributor. The team schedules an upgrade to 2.7 and reads the release notes section titled “Breaking changes”. The note is one line: Jaeger receivers removed. The team upgrades anyway because the Jaeger protocol counter in Grafana shows zero. Two weeks later, a legacy service that uses an unmaintained Jaeger client pushes its first span of the month. The span never reaches Tempo. The team finds the regression by reading the new bind addresses on the distributor pod and seeing that 14250 and 14268 are not listening. The fix is to migrate the client to OTLP and add a regression test that pushes one Jaeger-shaped span through the upgrade pipeline every day.

Receiver removals are silent when the receiver was already unused. They are loud when a single neglected client still depends on them. This lesson covers the receiver surface across Tempo versions and the diagnostic that catches a dependency that has outlived its receiver.

What it is

A Tempo receiver is the named configuration block inside distributor.receivers that binds a transport and protocol pair to a listen address. Tempo has shipped the following receivers across its version history:

• OTLP gRPC. otlp.protocols.grpc.endpoint. Default port 4317. Available in every Tempo release.
• OTLP HTTP. otlp.protocols.http.endpoint. Default port 4318. Available in every Tempo release.
• Jaeger Thrift compact (UDP). jaeger.protocols.thrift_compact. Default port 6831. Deprecated in Tempo 2.6 and removed in Tempo 2.7.
• Jaeger Thrift binary (UDP). jaeger.protocols.thrift_binary. Default port 6832. Deprecated in Tempo 2.6 and removed in Tempo 2.7.
• Jaeger Thrift HTTP. jaeger.protocols.thrift_http. Default port 14268. Deprecated in Tempo 2.6 and removed in Tempo 2.7.
• Jaeger Protobuf gRPC. jaeger.protocols.grpc. Default port 14250. Deprecated in Tempo 2.6 and removed in Tempo 2.7.
• Zipkin. zipkin.endpoint. Default port 9411. Available in every Tempo release.
• Kafka. kafka. Available from Tempo 2.1 onwards. Reads batched spans from a Kafka topic.

The migration path between receivers was always OTLP. Jaeger itself shifted its SDK default to OTLP before the Tempo 2.7 release, so the cutover is twofold: client SDK default plus receiver surface.

Why a sysadmin cares

Three production scenarios apply:

1. Ingest goes dark for the Jaeger fleet. A migration to Tempo 2.7 leaves the Jaeger clients without a target. Symptom: tempo_distributor_spans_received_total{protocol="jaeger"} drops to zero (because the metric stops being emitted, not because the spans stop arriving). The first signal of the regression is the absence of a signal.
2. OTLP over gRPC rejected by the load balancer. A network policy that whitelists by protocol name misses the new gRPC protocol. Symptom: tempo-distributor returns connection refused on 4317. The fix is to renew the policy before the upgrade.
3. Zipkin push behind a proxy. The Zipkin client pushes JSON. A proxy that does not allow HTTP POST on the right path is silent. Symptom: zipkin on the distributor log shows 404 repeatedly.

The cost of a missed receiver migration is the slow discovery of a fleet that pushes spans the cluster no longer accepts.

How it works

The receiver surface is a YAML block the distributor parses at start. Each protocol either binds a port (when enabled) or stays absent (when removed).

   Tempo distributor (start)
        |
        v
   +-------------------+
   | Parse             |
   | distributors.     |  look up "otlp", "jaeger",
   | receivers map     |  "zipkin", "kafka"
   +-------------------+
        |
        v
   +-------------------+   each receiver binds
   | For each entry:   |   its endpoint on the
   | bind address      |   distributor pod
   +-------------------+
        |
        v
   +-------------------+   parse the per-protocol
   | Map per-receiver  |   transports (gRPC, HTTP,
   | decoder           |   Thrift, JSON)
   +-------------------+
        |
        v
   Receivers ready on
   4317, 4318, 9411,
   optional jaeger ports,
   optional kafka topic

The map is the source of truth. A receiver key that disappears from the supported set produces a warning at start and the binary proceeds without the receiver. There is no crash.

Under the hood

How to configure it

A distributor config that supports every receiver available in Tempo 2.6. After the 2.7 cutover the Jaeger block stays in the YAML but is silently ignored.

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: '0.0.0.0:4317'
          max_recv_msg_size_mib: 16
        http:
          endpoint: '0.0.0.0:4318'
    jaeger:
      protocols:
        thrift_compact: { endpoint: '0.0.0.0:6831' }     # removed 2.7
        thrift_binary:  { endpoint: '0.0.0.0:6832' }     # removed 2.7
        thrift_http:    { endpoint: '0.0.0.0:14268' }    # removed 2.7
        grpc:           { endpoint: '0.0.0.0:14250' }    # removed 2.7
    zipkin:
      endpoint: '0.0.0.0:9411'

kafka:
  enabled: true
  topic: tempo-traces
  brokers: kafka-bootstrap.observability.svc:9092
  consumer_group: tempo

Three details to call out:

• The receivers block on the distributor process. In microservices mode each distributor pod exposes all enabled receivers; a Service load-balances the ports.
• The Kafka block on top level (not under distributors). The Kafka receiver writes spans directly to the ingester, bypassing the distributor entirely.
• max_recv_msg_size_mib matters for OTLP. A default of 16 rejects traces larger than 16 MiB with resource_exhausted. Bump it for clients that emit wide traces.

How to validate it

Severity: READ-ONLY.

1. Confirm the receivers bound the right ports in the new version. The pod log line at start lists every bound endpoint:

kubectl -n observability logs deploy/tempo-distributor --tail=50 \
  | grep -E "listening on|bound"
# level=info msg="listening on" address=0.0.0.0:4317 protocol=otlp/grpc
# level=info msg="listening on" address=0.0.0.0:4318 protocol=otlp/http
# level=info msg="listening on" address=0.0.0.0:9411 protocol=zipkin

2. Confirm OTLP gRPC accepts a span from a synthetic client:

TRACE=$(uuidgen)
otel-cli span export --service synth --name probe --trace-id "$TRACE" \
  --endpoint tempo-distributor.observability.svc:4317 \
  --protocol otlp/grpc
sleep 5
curl -sG http://tempo-querier:3200/api/traces/$TRACE | jq '.batches | length'
# 1

3. Confirm Zipkin accepts a span if any client still depends on it:

TRACE=$(uuidgen)
curl -sX POST "http://tempo-distributor:9411/api/v2/spans" \
  -H 'content-type: application/json' \
  -d "$(jq -n --arg id "$TRACE" '
      [{
        "traceId": $id,
        "id": "abc123",
        "name": "synthetic",
        "timestamp": (now | floor),
        "duration": 1000,
        "kind": "CLIENT",
        "localEndpoint": { "serviceName": "synth" }
      }]')"
sleep 5
curl -sG http://tempo-querier:3200/api/traces/$TRACE | jq '.batches | length'
# 1

4. Confirm the receiver-side counters move after the upgrade:

curl -s http://tempo-distributor:3200/metrics \
  | awk '/^tempo_distributor_spans_received_total/ {print}'
# tempo_distributor_spans_received_total{protocol="otlp/grpc"} 18423
# tempo_distributor_spans_received_total{protocol="zipkin"} 4892

5. Confirm a removed receiver stops emitting metric labels. A Jaeger label that disappears after 2.7 is the proof that the receiver was removed:

curl -s http://tempo-distributor:3200/metrics \
  | grep '^tempo_distributor_spans_received_total{protocol="jaeger' \
  | wc -l
# 0

How it can fail

Six shapes appear in production receiver migrations:

1. Jaeger fleet silent after 2.7. Tempo 2.7 removed the Jaeger receivers. A fleet that pushes Jaeger spans never sees them again. Symptom: the metric label protocol="jaeger" is absent after the upgrade but the fleet log shows no error — the UDP push is fire-and-forget.
2. OTLP gRPC rejected by the load balancer. A service mesh policy that was scoped to tcp:4317 is unchanged but the new binary expects the OTLP gRPC content-type. Symptom: 403 from the proxy with no obvious link to the upgrade.
3. Zipkin protocol mismatch. A Zipkin client that pushes protobuf on the JSON endpoint receives a parse error. Symptom: tempo_distributor_spans_dropped_total{protocol="zipkin"} rises. The fix is to either move the client to JSON or enable Zipkin protobuf at the distributor.
4. Kafka consumer group lag. The Kafka receiver reads the tempo-traces topic at the offset stored in the consumer group. A new consumer group starts from the latest offset and misses hours of historical spans. Symptom: the bucket suddenly grows after the upgrade while the metric tempo_kafka_consumer_lag reads at zero.
5. NetworkPolicy blocks the new port. Tempo 2.7 adds a new protocol sub-block on a port the cluster network policy does not allow. Symptom: distributor logs show bind: address already in use or the new port never binds.
6. max_recv_msg_size_mib too small for wide traces. OTLP rejects traces wider than the configured size with resource_exhausted. Symptom: client-side ExportedTooLarge errors appear after the upgrade.

How to troubleshoot it

The diagnostic order matters. Each step rules out one failure mode:

1. Did the receivers bind? Start with the distributor log file. The list of bound ports is the first thing the binary emits.
2. Is the metric counter moving per protocol? Per-protocol metrics confirm that each receiver is alive and accepting traffic.
3. Can a synthetic client reach each port? A 2-line curl call per protocol catches most regressions without needing the production fleet.
4. Are there per-receiver errors? tempo_distributor_spans_dropped_total{protocol=...} rises only when the distributor sees traffic and rejects it.
5. Is the network policy still matching? A policy diff that missed a new port is the most common answer.

Security implications

Every receiver is a network surface. Removing a receiver shrinks the surface; adding one grows it.

• Per-port authentication. OTLP accepts optional authentication blocks for client cert / API key / bearer token checks. A receiver added without auth becomes a public ingest endpoint.
• UDP attack surface. The Jaeger thrift_compact and thrift_binary receivers used UDP. UDP traffic is amplified by reflection attacks. Removing those receivers is also a security improvement.
• Zipkin JSON parsing. The Zipkin receiver parses JSON bodies without an authentication layer by default. A network policy or a TLS termination in front of the receiver is the recommended posture.
• Kafka consumer credentials. The Kafka receiver reads spans from a topic. SASL/PLAIN or SASL/SCRAM on the broker is required for any non-trivial deployment.

Performance implications

The receiver is the first stage of the ingest path. A misconfigured receiver costs you at the front door:

• OTLP gRPC. CPU-bound on the codec. Tune max_recv_msg_size_mib to the largest span size the clients emit. Avoid unbounded receives.
• OTLP HTTP. Same codec; HTTP/2 framing pulls a goroutine per stream. Bound with http_idle_timeout.
• Kafka. Network-bound on the partition fetch. Tune kafka.consume_lag_threshold and kafka.fetch_max_bytes to match the expected partition size.
• Jaeger (when still enabled). UDP is memoryless; a burst of UDP packets before the rate limiter catches them can spike memory.

Production guidance

• OTLP gRPC and OTLP HTTP are the only receivers guaranteed to remain across versions. Migrate the fleet to OTLP before any Jaeger removal.
• Read the release notes entry for distributor.receivers per version. The note is one line and easy to miss.
• Add a synthetic span push for every receiver you care about, on a schedule. The metric counter falling to zero is the first regression signal.
• Pin max_recv_msg_size_mib for OTLP. A misconfigured proxy payload limit is a common silent regression on wide traces.

Verification

You should now be able to answer:

• Which Tempo version removed the Jaeger receivers, and which receivers were deprecated in the version before?
• What is the default port for OTLP gRPC, OTLP HTTP, Zipkin, and Kafka receivers?
• Where does the Kafka receiver live in the tempo.yaml, and why is it not under distributor.receivers?
• What is the first diagnostic that confirms a removed receiver is no longer accepting traffic?
• How do you detect a fleet that is still pushing Jaeger spans after the receivers were removed?

Quiz