Tempo Ingester Scaling

At 11:42 a Tempo deployment was running five ingester pods with replication_factor: 3. An operator, asked to save cost, scaled the StatefulSet to three. Writes continued for a few minutes, then the distributor started returning 503. The remaining three pods had not yet absorbed the ring rebalance; two spans arrived at the same time, hit the same ingester, and the write quorum for one of them returned only two out of three required replicas before the new ring state was loaded. The fix was to roll forward to five, wait for the ring to settle, and then drain one pod at a time.

This lesson is the discipline of sizing the ingester StatefulSet in microservices mode: replication factor, ring churn, WAL placement, and the right number of replicas.

What it is

The Tempo ingester is the only component that holds spans in memory before they reach object storage. It is therefore the only component that can lose data on a single-host failure, and the only component whose scaling must respect state.

Scaling the ingester is the practice of choosing:

• The replication factor (replication_factor).
• The replica count of the StatefulSet.
• The placement of the WAL on local disk.
• The ring KV backend (memberlist or etcd).
• The lifecycle timings that decide how long a rebalance takes.

Each choice has a production trade-off. The wrong combination produces write outages during planned maintenance.

Why a sysadmin cares

Three operational pains are specific to ingester scaling:

1. Under-replicated ring. A replication_factor that exceeds the replica count at any moment produces write failures that look like a Tempo bug. They are a sizing bug.
2. Ring churn during rolling restarts. A misconfigured join_after or heartbeat_timeout makes each restart take minutes instead of seconds and produces 503 responses throughout the rollout.
3. WAL on slow disk. The WAL is fsync-heavy under sustained write load. Network-attached storage does not have the IOPS profile. The symptom is rising tempo_ingester_wal_write_bytes and tempo_ingester_failed_flushes_total together.

The ingester is the most expensive component to run and the most expensive to misconfigure.

How it works

The distributor hashes the trace ID to a position on the ring. Each ingester owns a contiguous range. For replication_factor: 3, the distributor forwards each span to the three ingesters that own the position and the two ranges adjacent to it.

  Distributor
      |
      |  hash(trace_id) = 17
      |
      v
  +----------------------------------------------+
  |         Ingester ring (logical view)         |
  |                                              |
  |   ingester-0   owns tokens 0-31, 96-127       |
  |   ingester-1   owns tokens 32-63, 128-159     |
  |   ingester-2   owns tokens 64-95              |
  |                                              |
  |   For trace_id at position 17, RF=3:         |
  |     primary   = ingester-0                   |
  |     replica 1 = ingester-2  (next clockwise) |
  |     replica 2 = ingester-1  (next clockwise) |
  |                                              |
  |   Quorum = RF/2 + 1 = 2 acks needed          |
  +----------------------------------------------+

The ring KV (memberlist gossip or etcd consensus) is the authoritative source for which ingester owns which token. The distributor reads the ring on every push. When the ring changes, spans in flight on the old ring may go to ingesters that no longer own the trace’s hash; the distributor retries against the new ring.

The WAL is local to each ingester pod. The PVC bound to the StatefulSet member must be on local SSD, not NFS or a remote block device.

How to configure it

A production ingester config pins the ring, the WAL, and the flush cadence. The values below are the production defaults for a mid-scale cluster:

ingester:
  trace_idle_period: 10s
  max_block_duration: 30m
  flush_check_period: 5s
  max_block_bytes: 524288000    # 500 MiB cap per block

  lifecycler:
    ring:
      kvstore:
        store: memberlist
      replication_factor: 3
      heartbeat_timeout: 1m
    heartbeat_period: 5s
    join_after: 10s
    observe_period: 10s
    final_sleep: 0s

storage:
  trace:
    backend: s3
    s3:
      bucket_name: tempo-traces-prod
      region: eu-west-1
    wal:
      path: /var/tempo/wal

Three details to call out:

• replication_factor: 3 requires at least three ingester pods. Two is not enough for the write quorum (RF/2 + 1 = 2) to succeed under a single-pod loss.
• heartbeat_timeout: 1m and heartbeat_period: 5s together decide how long a dead pod takes to be evicted from the ring. Too short: false evictions during a slow restart. Too long: the distributor forwards spans to a dead pod for too long.
• wal.path must point at a directory backed by local SSD. The StatefulSet must mount a hostPath or a local PV, not a network PVC.

When the deployment uses Kubernetes, the matching StatefulSet fragment:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: tempo-ingester
spec:
  replicas: 5                  # 5 > RF=3 absorbs one pod + one rolling
  serviceName: tempo-ingester
  selector:
    matchLabels:
      app: tempo-ingester
  template:
    metadata:
      labels:
        app: tempo-ingester
    spec:
      containers:
      - name: tempo
        image: grafana/tempo:latest
        args:
        - '-target=ingester'
        - '-config.file=/etc/tempo/tempo.yaml'
        volumeMounts:
        - name: wal
          mountPath: /var/tempo/wal
      volumes:
      - name: wal
        hostPath:
          path: /var/tempo/wal   # local SSD on the node

Severity: CONFIGURATION. Reloading the YAML requires a process restart. The StatefulSet replica count is a runtime edit that triggers a rolling restart.

How to validate it

Severity: READ-ONLY.

1. Confirm the ingester ring membership and state:

curl -s http://tempo-ingester:3200/ingester/ring | jq '
  { members: .members | length,
    active: [.members[] | select(.state == "ACTIVE")] | length }
'
# { "members": 5, "active": 5 }

2. Confirm replication_factor matches the live replica count:

curl -s http://tempo-ingester:3200/ingester/ring \
  | jq '.replication_factor'
# 3

kubectl get statefulset tempo-ingester \
  -o jsonpath='{.spec.replicas}'
# 5

3. Confirm the WAL is being drained:

curl -s http://tempo-ingester:3200/metrics \
  | grep tempo_ingester_local_checkpoint_manager_last_saved_timestamp
# tempo_ingester_local_checkpoint_manager_last_saved_timestamp  1723655400

The timestamp should be within max_block_duration of now. A value older than that means flushes have stalled.

4. Confirm the WAL disk has headroom:

kubectl exec tempo-ingester-0 -- df -h /var/tempo/wal
# Filesystem      Size  Used Avail Use% Mounted on
# /dev/nvme0n1    100G   12G   88G  12% /

5. Confirm replication is working by sending a single span and checking it on each ingester:

otel-cli span export --endpoint tempo-distributor:4317 \
  --service checkout --name POST /charge
TRACE_ID=$(otel-cli span ls --limit 1 --format json | jq -r '.[0].TraceId')

for i in 0 1 2 3 4; do
  echo -n "ingester-$i: "
  curl -s "http://tempo-ingester-$i:3200/api/traces/${TRACE_ID}" \
    | jq '.batches | length'
done
# ingester-0: 1
# ingester-1: 1
# ingester-2: 1
# ingester-3: 1
# ingester-4: 1

A non-1 count from one pod means the ring KV is reporting the pod as a primary for that trace, but the span has not arrived there yet. This is normal during a ring rebalance.

How it can fail

Six shapes appear repeatedly:

1. Replica count below replication_factor. A scale-down leaves the ring with too few owners; quorum writes fail. Symptom is 503 from the distributor and tempo_distributor_dropped_spans_total rising.
2. Ring churn during a rolling restart. A misconfigured join_after: 30s and heartbeat_timeout: 10s together cause the new pod to be marked dead before it has finished joining. Symptom is tempo_ingester_lifecycler_ring_inconsistencies_total rising and per-pod state flipping between ACTIVE and JOINING.
3. WAL on network-attached storage. NFS or a remote block device cannot sustain the fsync rate under sustained load. Symptom is tempo_ingester_wal_write_bytes rising and tempo_ingester_failed_flushes_total rising in lockstep.
4. replication_factor too low for the loss profile. A replication_factor: 1 deployment loses data on every pod restart. Symptom is missing traces during planned maintenance.
5. Tokens distributed unevenly across replicas. Older versions of Tempo used a fixed token scheme; newer versions use lifecycler.tokens_per_local_ring (default 128). A misconfiguration that drops the token count produces hot pods. Symptom is one pod’s tempo_ingester_local_blocks far above the others.
6. WAL full because flushes have stalled. Object storage credentials have rotated or the bucket is throttling. The ingester keeps accepting spans, the WAL grows, the disk fills. Symptom is df showing the WAL path near 100% and tempo_ingester_failed_flushes_total rising.

How to troubleshoot it

The diagnostic order, cheapest first:

1. How many ingester pods are healthy? kubectl get statefulset tempo-ingester and /ingester/ring. The two must agree.
2. Is the ring KV reachable? memberlist is gossip; a pod that cannot reach any other pod will not see the ring. etcd is consensus; a quorum loss stalls the ring.
3. Is the WAL draining? Check the checkpoint timestamp.
4. Are flushes failing? Check tempo_ingester_failed_flushes_total and the bucket credentials.
5. Is disk filling? df -h on the WAL path.
6. Is memory under pressure? Check go_memstats_heap_inuse_bytes and the OOM killer logs.

Security implications

The ingester is local-network-only:

• WAL on local disk. The WAL contains span payloads. Restrict the WAL path to root or to a dedicated service user. Anyone with read access to the WAL path can read every span that arrived in the last flush window.
• No inbound network. The ingester accepts traffic from the distributor over the in-cluster network. The distributor must not be exposed publicly; the ingester inherits that protection.
• Tracing sensitive data. If traces contain PII or secrets, the WAL and the bucket both contain them. Tempo does not redact at ingest; redaction is the responsibility of the SDK or the collector.

Performance implications

The ingester is the most expensive component to run:

• Memory. Each open head block holds all spans for one trace ID. A burst of long-lived traces can exhaust heap memory.
• Disk. The WAL writes one record per span. A 20 MiB/s ingest writes roughly 20 MiB/s to the WAL until the next checkpoint.
• Network. Each span is replicated replication_factor times inside the ring (the distributor fans out). At replication_factor: 3 a 20 MiB/s ingest becomes a 60 MiB/s internal stream.
• CPU. Span decoding and WAL encoding consume CPU. A sustained 30 MiB/s ingest saturates one CPU core per ingester pod.

Sizing rule of thumb: one ingester pod handles roughly 20k spans/sec at typical span size on a 4-vCPU / 8 GiB host. Add pods (not CPU) for more ingest.

Production guidance

• Pin the WAL to local SSD. Verify the disk with fio before promoting an ingester to production.
• Run at least replication_factor + 2 ingester pods so one pod can be in a rolling restart without dropping below RF.
• Set replication_factor to the smallest number that tolerates your expected loss profile. 3 is the production default. 2 is acceptable for dev only.
• Alert on tempo_ingester_local_checkpoint_manager_last_saved_timestamp age and on df of the WAL path.
• Use memberlist for a single Kubernetes cluster. Switch to etcd when the ring spans clusters or when the cluster has more than a few hundred members.

Verification

You should now be able to answer:

• What does replication_factor decide, and what is the smallest value that tolerates a single ingester failure?
• Why must the WAL live on local SSD?
• What does join_after control, and what is the failure shape if it is too low?
• How do you validate that replication is working?
• Why is scaling down below replication_factor a write outage?

Quiz