Skip to main content
RunBook Academy

← All checklists in Observability

Before deploymenttempo-readiness

Tempo Production Readiness

25 items ·19 critical ·6 warn ·0 info

Run this before Tempo takes production traces, and again after any change to the deployment mode, the receivers, the bucket, or the identity Tempo uses to reach it.

It overlaps the collector readiness gate deliberately. That gate covers the pipeline that produces spans — what is sampled, what is redacted, what is batched and where it is sent. This one covers everything on the receiving side, starting at the port the exporter connects to.

Why Tempo needs a different kind of check

Two properties shape this list.

The first is that Tempo’s parser is permissive and its readiness probe is honest about the wrong thing. Unknown configuration keys are ignored rather than rejected, so a misspelling does not stop the process — the affected subsystem never starts, the role reports ready, and everything looks correct until somebody needs a trace. Several items below exist only because a green probe is not evidence.

The second is that the bucket is the trace store. Tempo is an index-less query engine sitting on top of object storage, with no durable local copy of the data. That means the bucket, the identity used to reach it, and the lifecycle rules attached to it are production dependencies of the query path, not storage-team detail. Most incidents that arrive as “Tempo is broken” are one of those three.

Where the numbers come from

Most items are one call to a component’s config, ring or metrics endpoint, one shell command on a component host, or one object-store API call. Substitute your own addresses, bucket names and role names; the ones below are the shapes used throughout this course, not an estate that exists.

Access this needs

Read access to each Tempo role’s HTTP endpoint, shell access on a component host for the listener and disk items, read-only credentials for the object store, and a client host on the network the applications use for the reachability check. Two items — the propagation demonstration and the sampling strategy — are attested by a person. The synthetic trace is the only thing on this list that writes, and what it writes is one span.

Sign-off

  • Reviewer: ________________ Date: ___________
  • Platform owner: ___________ Date: ___________
  • Largest producing team: ____ Date: ___________

Critical19 items

  1. curl -s http://tempo:3200/status | jq .target
  2. for role in distributor ingester querier compactor query-frontend metrics-generator; do printf "%s: " "$role"; curl -s -o /dev/null -w "%{http_code}\n" "http://tempo-${role}:3200/ready"; done
  3. tempo-cli validate-config /etc/tempo/tempo.yaml
  4. ss -tlnp | grep -E '4317|4318|14250|9411|6831|6832'
  5. sed -n '/^distributor:/,/^ingester:/p' /etc/tempo/tempo.yaml
  6. nc -zv tempo 4317; nc -zv tempo 4318
  7. curl -s http://tempo:3200/config | grep -A8 'trace:'
  8. aws s3 ls s3://tempo-traces-prod/blocks/ --recursive --summarize | tail -3; aws s3api get-bucket-policy --bucket tempo-traces-prod | jq -r '.Policy' | jq '[.Statement[].Action] | flatten'
  9. aws s3api get-bucket-lifecycle-configuration --bucket tempo-traces-prod
  10. curl -s http://tempo:3200/config | grep -A5 'compaction:'
  11. curl -s http://tempo-compactor:3200/compactor/ring | jq '.members'; curl -s http://tempo-compactor:3200/metrics | grep -E '^tempo_compactor_(blocks_compacted_total|blocks_marked_for_deletion_total)'
  12. curl -s http://tempo-ingester:3200/ingester/ring | jq '.members'; df -h /var/tempo/wal
  13. curl -s http://tempo:3200/config | grep -A8 'limits'
  14. curl -s http://tempo-distributor:3200/metrics | grep -E '^tempo_distributor_(spans_received_total|dropped_spans_total)'
  15. curl -sG http://tempo:3200/api/search --data-urlencode 'query={ resource.service.name = "tempo-smoke" }' --data-urlencode 'limit=5' | jq '.traces | length'
  16. sed -n '/^processors:/,/^exporters:/p' /etc/otelcol/config.yaml
  17. curl -s http://tempo:3200/config | grep -E 'auth_enabled|X-Scope-OrgID'
  18. curl -s -G http://meta-prom:9090/api/v1/query --data-urlencode 'query=up{job=~"tempo.*"}' | jq -r '.data.result[] | "\(.metric.instance) \(.value[1])"'

Warning6 items

  1. curl -s http://tempo:3200/config | grep -A6 'otlp:'
  2. curl -s http://tempo:3200/config | grep -E 'endpoint|region|forcepathstyle|prefix'
  3. curl -s http://tempo:3200/config | grep -E 'region|endpoint'
  4. curl -s http://tempo:3200/config | grep -E 'max_query_length|max_concurrent_queries|query_timeout|max_parallelism'
  5. curl -s http://tempo-metrics-generator:3200/metrics | grep -E '^tempo_metrics_generator_spans_total' | head -3