Tempo Deployment Modes

A 12:00 incident: the trace UI shows “no recent data” while the metrics and logs panels are healthy. The on-call engineer restarts the Tempo container. Spans resume. Two hours later, the same condition appears. The container was already running. What is actually wrong is that a single Tempo process was accepting trace writes, indexing them in memory, and serving queries, and the single ingester queue filled because nobody had sized it for the current load. The container was fine; the deployment mode was wrong for the load.

The mode you choose decides which Tempo components run in one process, which scale independently, and how an outage of one role affects the rest.

What it is

Tempo offers three deployment modes, selected with the top-level target key in tempo.yaml:

                 Single binary             Simple scalable           Microservices
                 ---------------           ----------------          -------------
Processes        1                         2-3 (per role)            8+ (one per role)
target value     all                       scalablesingleprocess     (one per service)
                 (default)                 (formerly scale-out)
Roles collapsed  distributor+ingester+     distributor, ingester,     each role runs
                 querier+compactor+        querier (compactor        on its own host
                 metrics-generator         optional)                  and StatefulSet
Best for         dev, lab, small prod      small/mid prod, single-   large prod, multi-
                                            tenant, 1-2 node HA       tenant, multi-AZ
Failover         restart the binary        restart replicas          per-role, no
                                                                    cross-impact

The three modes are not “small, medium, large” buckets. They are different shapes with different operational costs.

Why a sysadmin cares

The mode determines:

• Blast radius. A crash of the ingester in single-binary mode stops writes and queries. In microservices mode, only the ingester role stops, and reads continue against compacted blocks.
• Operational surface. Microservices mode needs eight or more StatefulSets / Deployments, plus service-to-service auth, plus a per-role SLO. A two-person team does not have that surface.
• Cost ceiling. Single binary caps at one host. When the host is full, the only path forward is a mode change, not a config change. Mode selection is a one-way door for a year or two.

How it works

Single binary

target: all runs every Tempo component inside one process. The distributor, ingester, querier, compactor, and metrics-generator share the same memory and the same lifecycle. Configuration keys that reference other components (e.g. the ingester address in the querier block) become localhost.

+---------------------------------------+
| tempo process (target: all)           |
|  distributor --> ingester --> wal/    |
|     |                       |         |
|     v                       v         |
|  metrics-gen            compactor    |
|     |                       |         |
|     +-------> querier <-----+         |
|              |                        |
+---------|-----|----------------------+
          v     v
         S3    query UI

Best for: development, lab exercises, single-tenant deployments processing under a few hundred thousand spans per second.

Simple scalable

target: scalablesingleprocess (formerly scale-out) runs the distributor, ingester, and querier as independent replicas. The compactor and metrics-generator remain optional but run in their own process if enabled. Components talk over gRPC on the loopback interface by default.

+--------------------+        +--------------------+
| distributor x N    |  --->  | ingester x N       | ---> S3
| accepts OTLP /     |        | writes WAL, flushes |
| Jaeger / Zipkin    |        | blocks every 15 m  |
+--------------------+        +--------------------+
                                       |
                                       v
                              +--------------------+
                              | querier x N        |
                              | serves queries     |
                              +--------------------+
                                       ^
                                       |
                              +--------------------+
                              | compactor          |
                              | (one replica)      |
                              +--------------------+

Best for: small to mid production, one or two AZs, single tenant, teams that want HA without running eight StatefulSets.

Microservices

Each role is its own process and its own deployment unit. The distributor, ingester, querier, query-frontend, compactor, metrics-generator, ingester-ring (if externalised), and the optional usage-report shipper all run independently.

   OTLP ---> distributor (k8s Deployment)
                  |
                  v
             ingester (StatefulSet, ring-stored)
                  |
                  v
               S3/GCS/Azure
                  ^
                  |
   query-frontend ---> querier ---> ingester (tail blocks)
                  |
                  v
              compactor (one)
              metrics-generator (sidecar on querier)

Best for: large production, multi-tenant, multi-AZ, hundreds of thousands of spans per second, teams that can operate it.

Under the hood

How to configure it

Single binary

# /etc/tempo/tempo.yaml — single binary
server:
  http_listen_port: 3200

distributor:
  receivers:
    otlp:
      protocols:
        grpc: {}
        http: {}

ingester:
  trace_idle_period: 10s
  max_block_duration: 5m

compactor:
  compaction:
    block_retention: 48h

storage:
  trace:
    backend: s3
    s3:
      bucket: tempo-traces
      endpoint: minio.storage.svc:9000
      access_key: tempo
      secret_key: change-me
      insecure: true

Severity: CONFIGURATION. Reload is not supported — restart the process to apply changes. The first cut for a lab; do not grow this past a single 16-CPU host.

Simple scalable with Docker Compose

The official tempo-distributed repo ships a Compose file. The key is the same binary image with different target values per service:

# docker-compose.yml (abridged)
services:
  tempo-distributor:
    image: grafana/tempo:latest
    command: ["-target=scalablesingleprocess", "-config.file=/etc/tempo.yaml"]
    ports: ["4317:4317", "4318:4318"]

  tempo-ingester:
    image: grafana/tempo:latest
    command: ["-target=scalablesingleprocess", "-config.file=/etc/tempo.yaml"]

  tempo-querier:
    image: grafana/tempo:latest
    command: ["-target=scalablesingleprocess", "-config.file=/etc/tempo.yaml"]

  tempo-compactor:
    image: grafana/tempo:latest
    command: ["-target=compactor", "-config.file=/etc/tempo.yaml"]

All services share one tempo.yaml. The querier block tells each component where to find the ingester:

querier:
  frontend_worker:
    frontend_address: tempo-query-frontend:9095

Severity: SERVICE-IMPACT. Restarting the distributor in Compose drops in-flight writes. Restarting the ingester flushes the in-memory trace buffer to S3 before exiting (the WAL does this, but the rollover takes a few seconds).

Microservices with Helm

helm repo add grafana https://grafana.github.io/helm-charts
helm install tempo grafana/tempo-distributed \
  --namespace observability --create-namespace \
  --set search.maxConcurrentQueriers=4 \
  --set ingester.replicas=3 \
  --set compactor.replicas=1

Severity: CLUSTER-WIDE. Helm diff first; apply second.

How to validate it

Severity: READ-ONLY.

# 1. Confirm the running target
curl -s http://tempo:3200/status | jq .target

# 2. Confirm each role is healthy in the relevant process
curl -s http://tempo-distributor:3200/ready
curl -s http://tempo-ingester:3200/ready
curl -s http://tempo-querier:3200/ready

# 3. In microservices, list the ingester ring members
curl -s http://tempo-query-frontend:3200/api/echo | jq .

Real output:

$ curl -s http://tempo:3200/status | jq .target
"all"

$ curl -s http://tempo-distributor:3200/ready
ready

A 404 from any role endpoint means the role is not registered in that process — likely a wrong target value in the Compose/Helm file.

How it can fail

1. Single-binary process OOMs at peak. The ingester holds in-flight traces in memory until the WAL flushes. A traffic spike that exceeds max_block_duration causes the heap to grow. Symptom: process restarted by container runtime; missing traces during the restart window.

2. Simple-scalable distributor accepts writes but ingester ring has no healthy members. The distributor cannot forward to the ingester ring. Symptom: 5xx responses at the OTLP endpoint; distributor_dropped_spans_total rising; zero new blocks in S3.

3. Microservices ingester StatefulSet stuck on one replica. A ReadWriteOnce PVC bound to a deleted pod stalls the new pod’s start. Symptom: ingester pods Pending, ring unhealthy.

4. Compactor missing in microservices. Blocks accumulate beyond their block_retention window. Symptom: storage bill rises; old data still returned but not deleted. The compactor is mandatory in microservices mode; the binary tolerates its absence, the storage bill does not.

5. Component address misconfiguration between roles. After a mode change from single-binary to simple-scalable, the querier.frontend_worker.frontend_address stays at localhost:9095 on each pod, which is unreachable. Symptom: query-frontend 503s; trace UI loads but trace lookup returns “trace not found”.

6. Helm chart version pinned to an old microservices layout. Tempo has renamed targets and moved blocks. An old chart on a new binary starts, accepts writes, and produces blocks in a layout the new compactor cannot compact. Symptom: compactor logs unknown block format.

How to troubleshoot it

Order of diagnostics, cheapest first:

1. Which mode is running? curl /status returns the target. If the value does not match the deployment plan, the mode change did not take.
2. Which process is alive? curl /ready per role. Any 404 points to the wrong target in the unit file or Compose.
3. What is the ring state? In any mode that uses an ingester ring, the distributor logs the healthy ingester set on every trace push. Empty set == ring is unhealthy; check the ingester pods / StatefulSet first.
4. Where is data landing? A quick aws s3 ls s3://tempo-traces/ --recursive | head confirms the block layout is what the YAML expects. A different prefix is a config drift, not a Tempo problem.

Security implications

Each mode exposes the same set of ports by default. The microservices mode exposes more ports because each role has its own service. The risk is not the port count; it is that the role addresses are stable DNS names and become a richer target for internal reconnaissance. Two mitigations:

• Bind the HTTP listen address to a private interface in microservices mode (server.http_listen_port and the per-role gRPC listen address).
• Use mTLS between roles. Simple-scalable on Compose often skips this because all services share a private network; microservices on a cluster with shared nodes should not skip it.

Performance implications

• Single binary. Limited by host CPU and memory. A 16 vCPU / 32 GiB host handles roughly 50k spans/sec at typical span size.
• Simple scalable. Distributor and querier scale horizontally; the ingester is bounded by per-pod memory. Add ingester pods for more ingest, querier pods for more concurrent queries.
• Microservices. Each role scales on its own metric. Watch ingester_blocks_flushed_total, querier_query_seconds, compactor_blocks_compacted_total.

Capacity rule of thumb: plan for 2x current load before adding replicas. The migration cost (drain, restart, warm WAL) swallows the headroom gain if you size for current.

Production guidance

• Start with single binary in dev, simple scalable in staging, microservices in prod once ingest exceeds ~50k spans/sec or the team grows past two operators.
• Use the Helm chart rather than hand-rolled Compose for microservices. The chart encodes the role wiring and is tested against new releases.
• Run the compactor as its own StatefulSet, not as a sidecar. A stuck compactor should not take down the ingester or querier.

Verification

You should now be able to answer:

• Which target value runs every Tempo component in one process?
• Which mode separates the ingester onto its own StatefulSet?
• Where does the compactor run in single-binary mode, and what happens if it stops?
• What is the operational ceiling that should trigger a move from single-binary to simple-scalable?
• Which deployment mode gives the smallest blast radius for an ingester crash?

Quiz