The Distributor

A team pushes a new SDK release that over-instruments spans by an order of magnitude. Within minutes the Tempo distributor starts returning 429. The on-call engineer opens Grafana, sees a single tenant called team-checkout consuming every allowed byte per second, and rate-limits that tenant. The rest of the platform keeps accepting traces. Without the distributor, that single runaway service would have either crashed the cluster or filled the ingester disk.

This lesson describes the Tempo distributor: where it sits, what it validates, how it rate-limits, and how it forwards spans to the ingester ring.

What it is

The Tempo distributor is a stateless service that fronts the ingester pool. Its job is to receive spans over OTLP, Jaeger, or Zipkin, validate them, enforce per-tenant ingestion limits, and forward the spans to the correct ingester using a consistent hash on the trace ID.

The distributor is the only component of Tempo that talks directly to clients. It is also the only component that enforces ingestion policy. An ingester that accepts the same span directly would have no rate-limit protection.

Why a sysadmin cares

The distributor is the buffer between untrusted upstream telemetry and the stateful ingesters. Three operational concerns make it production-critical:

1. A bad client can overwhelm the cluster. A single SDK release that double-emits spans, a collector that loops, or a misconfigured sampling policy can push more spans per second than the ingester pool can drain. Without per-tenant limits, one team takes the cluster down for everyone.
2. A bad client can poison the data. Traces without a trace ID, batches with timestamps from the year 2030, payloads with ten-megabyte attribute values. The distributor validates these before they cost the ingester CPU.
3. A bad client can probe the service. The OTLP, Jaeger, and Zipkin receivers all accept unauthenticated traffic by default. Anything that can reach those ports can send spans. Treating the receiver ports as service-internal is part of the security posture.

How it works

The distributor is a single pipeline of four stages:

  Client / Collector
        |
        v
  +------------------+
  |   Receivers      |  OTLP gRPC/HTTP, Jaeger thrift/gRPC, Zipkin
  +--------+---------+
           |
           v
  +------------------+
  |   Validation     |  Trace ID present, span name, timestamp sane,
  +--------+---------+  max span size
           |
           v
  +------------------+
  |   Rate Limiting  |  Per-tenant ingestion bytes/sec,
  +--------+---------+  spans/sec, max traces/sec
           |
           v
  +------------------+
  |   Fan-out        |  hash(trace_id) → ingester in the ring
  +--------+---------+
           |
           v
      Ingester pool

Each stage maps to a config block. Receivers are under distributor.receivers.*. Validation rules are under distributor.receivers.<protocol>.max_recv_msg_size and a global limit at distributor.global.max_idle_connections. Rate limiting is under distributor.limits and per-tenant overrides under distributor.limits_overrides. The fan-out uses the ingester ring configured at ingester.lifecycler.ring.

How to configure it

A production distributor config defines the receivers, the global limits, and a per-tenant override for the noisiest tenant:

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: '0.0.0.0:4317'
          max_recv_msg_size: 16777216      # 16 MiB; OTLP default is 4 MiB
        http:
          endpoint: '0.0.0.0:4318'
    jaeger:
      protocols:
        grpc:
          endpoint: '0.0.0.0:14250'
        thrift_http:
          endpoint: '0.0.0.0:14268'
    zipkin:
      endpoint: '0.0.0.0:9411'

  # Global ingestion rate limits. The cluster-wide cap. Per-tenant
  # overrides go under overrides below.
  global:
    ingestion_rate_limit_bytes: 26214400   # 25 MiB/s
    ingestion_burst_size_bytes: 36700160   # 35 MiB burst
    max_traces_per_user: 50000
    max_global_traces_per_user: 0           # 0 = disabled

  # Per-tenant overrides. Each tenant can have its own byte and span
  # budget. Use it to cap the noisiest tenant without affecting others.
  overrides:
    tenant_id: team-checkout
    ingestion_rate_limit_bytes: 5242880    # 5 MiB/s
    ingestion_burst_size_bytes: 8388608    # 8 MiB burst
    max_traces_per_user: 10000

Two production details to call out:

• max_recv_msg_size: 16777216 lets OTLP batches up to 16 MiB through. The OTLP default is 4 MiB and rejects anything larger with ResourceExhausted. Bump it when clients legitimately batch large traces.
• The per-tenant overrides: block keys on the X-Scope-OrgID header the client sends. The Alloy or collector pipeline must stamp that header for the override to apply. Without it the tenant name is anonymous and gets the global limits.

How to validate it

Five checks confirm the distributor is doing its job:

1. Confirm the process is up and accepting the configured receivers:

curl -s http://tempo.internal:3200/distributor/ready
# ready
curl -s http://tempo.internal:3200/status | jq '.distributor'

2. Confirm spans are arriving. The most reliable signal is the per-tenant span counter:

curl -s http://tempo.internal:3200/metrics \
  | grep tempo_distributor_spans_received_total | head
# tempo_distributor_spans_received_total{tenant="team-checkout"}  4821

3. Confirm the fan-out reaches the ingesters. A successful write shows up in the distributor side as tempo_distributor_bytes_received_total and on the ingester side as tempo_ingester_bytes_received_total. The two counters should move in lockstep under steady load.

4. Confirm a rate limit is actually enforced. Send a flood and watch the 429 counter:

# Quickly push many OTLP spans using otel-cli in a loop
for i in $(seq 1 5000); do
  otel-cli span export --endpoint tempo.internal:4317 \
    --service flood --name probe --attrs i=$i >/dev/null 2>&1
done
curl -s http://tempo.internal:3200/metrics \
  | grep tempo_distributor_ingester_append_failures_total
# tempo_distributor_ingester_append_failures_total{ingester="..."}  1

The non-zero append_failures_total is the rate limiter at work.

5. Confirm TraceQL can find the traces the distributor accepted:

curl -sG http://tempo.internal:3200/api/search \
  --data-urlencode 'q={ resource.service.name = "flood" }' \
  --data-urlencode 'limit=5' | jq '.traces | length'

How it can fail

Six shapes appear repeatedly:

1. Per-tenant rate limit misconfigured. A tenant that should have a 10 MiB/s limit is given the global cap. Under load the tenant’s spans are dropped. Symptom is the tempo_distributor_ingester_append_failures_total counter rising for that tenant.
2. Wrong header for tenant identification. Alloy stamps X-Scope-OrgID differently from what the distributor expects. The per-tenant override never matches; every tenant is treated as anonymous and gets the global limits. Symptom is tempo_distributor_spans_received_total{tenant="anonymous"} high.
3. Receiver not bound. The distributor config has distributor.receivers.otlp.protocols.grpc.endpoint but the bind fails because the port is already in use. Symptom is the process exiting on startup or tempo_distributor_reachable returning false for the missing protocol.
4. OTLP batch size too small. A client sends 32 MiB OTLP batches; the distributor rejects them with ResourceExhausted. Symptom is tempo_distributor_dropped_spans_total rising alongside the max_recv_msg_size default.
5. Ring unhealthy. The distributor cannot reach the KV store. The fan-out cannot find an ingester; writes are rejected with 503. Symptom is tempo_distributor_rings_healthy returning 0.
6. Auth disabled. auth_enabled: false means anyone who can reach the receiver ports can submit spans. Symptom is a unexpected spike in tempo_distributor_spans_received_total from an unknown source.

How to troubleshoot it

The diagnostic order:

1. Is the distributor process up? Check pod status, systemd unit, or process listing. A crashed distributor shows up as connection refused from the client.
2. Is the receiver listening? Use ss -tlnp or netstat to confirm the OTLP, Jaeger, and Zipkin ports are bound. A config error leaves the port unbound silently if the process treats it as non-fatal.
3. Is the distributor reaching the KV store? The tempo_distributor_rings_healthy counter reports 1 when the ring is reachable. If it is 0, the distributor cannot route spans.
4. Are spans arriving? The tempo_distributor_spans_received_total counter is the ground truth. If it is flat, the problem is on the client side, not the distributor.
5. Is a tenant being rate-limited? The tempo_distributor_ingester_append_failures_total counter breaks down by tenant. A tenant with rising failures is hitting its limit.
6. Is the limit configured correctly? Diff the YAML on disk against the running config. Tempo logs the effective config on startup; cross-reference it.

Security implications

The distributor is the largest attack surface in Tempo. Three controls matter:

• Authenticate the receivers. Tempo does not implement authentication itself. Put the OTLP / Jaeger / Zipkin ports behind a reverse proxy that enforces JWT or mTLS, or run them on a private network where only the collectors can reach them.
• Validate tenant identity. The X-Scope-OrgID header is the tenant identifier. An attacker who can spoof the header can bill spans to another tenant or evade per-tenant limits. Strip the header at the reverse proxy and re-add it after authentication.
• Limit the maximum message size. A 16 MiB OTLP batch is large but not abusive. A 1 GiB batch is a denial of service. Set max_recv_msg_size to the largest legitimate batch and no larger.

Performance implications

The distributor is CPU-bound on protocol decoding and ring lookups:

• CPU. OTLP over HTTP is JSON-decoded per span. A sustained 20 MiB/s of OTLP traffic consumes roughly one CPU core per distributor pod on a modern x86.
• Memory. The per-tenant rate-limiter stores a small token bucket per tenant. A thousand tenants consume a few hundred kilobytes.
• Network. The distributor fans out to replication_factor ingesters. A 25 MiB/s ingest becomes a 75 MiB/s internal stream at replication_factor: 3. The in-cluster network must absorb this.
• Hot tenants. A single noisy tenant can saturate a distributor. Per-tenant rate limits spread the load; they do not eliminate it.

Production guidance

• Run at least two distributor pods behind a load balancer. One pod is a single point of failure; two are not.
• Set max_recv_msg_size based on the largest legitimate OTLP batch, not on a guess. 16 MiB is a reasonable starting point.
• Define per-tenant overrides for every team that emits more than 10% of cluster volume.
• Do not expose the OTLP, Jaeger, or Zipkin ports on a public network. Treat them as service-internal.

Verification

You should now be able to answer:

• What is the distributor’s role in the Tempo write path?
• What four stages does every span pass through before it reaches an ingester?
• How is the per-tenant rate limit keyed?
• What happens when a tenant exceeds its rate limit?
• Why must the OTLP and Jaeger receiver ports never be on a public network?

Quiz