Tempo Config Changes

A team upgrades Tempo from 2.5 to 2.6. The team uses the official Helm chart. The team does not run helm diff against the running values because the team trusts the chart to apply sensible defaults. The upgrade completes. The team notices that the metrics-generator is no longer producing service-graph metrics. The team investigates and finds the following diff:

-statefulSet: { }    # chart bumped from Deployment
                     # to StatefulSet for the metrics-generator
+ingester.lifecycler.join_after: 30s
                     # the default was tightened from 10s to 30s,
                     # which the running Helm values did not pin

The metrics-generator now runs as a StatefulSet with a stable identity, and the rest of the cluster has not been told. The Service routes traffic to a Service object that no longer matches the pod selector. The fix is to apply the diff manually to the running values, redeploy the chart, and restart the metrics-generator.

The lesson is not that the chart did the wrong thing. The lesson is that the team did the diffing in their head, and the diff in their head was incomplete.

What it is

A Tempo config migration is the process of updating a tempo values file (or a tempo.yaml) to match the syntax the target binary accepts. The migration is necessary because every Tempo minor release renames, restructures, removes, or replaces some YAML keys. A config that was valid in 2.5 may parse in 2.6 but behave differently.

The categories of config change between Tempo minor releases:

• Renames. A key was renamed. The old key is no longer recognised; the new key takes the same value. compactor.compaction was split into compactor.compaction.block_search_encoding and compactor.compaction.search_relevant_tag_values.
• Restructures. A block was reorganised. The values are the same; the nesting changed. query_frontend.search was extracted from the querier block to give the query-frontend its own search budget.
• Removals. A key was removed entirely. The value is no longer accepted. The Jaeger receivers under distributor.receivers.jaeger were removed in 2.7.
• New defaults. A key was introduced with a default that differs from the previous behaviour. The config does not need to change but the cluster behaves differently. The ingester.lifecycler.ring.heartbeat_timeout default was tightened across the 2.x series.
• Chart changes. The Helm chart controller changed. The values are the same but the workload kind was changed, the selector was broadened, or the Service object was added.

The Tempo binary does not include a migrate subcommand; the config is read once at start and the rename is a name change only, not a tool-driven transformation. The migration tool is the human review of the diff against the release notes.

Why a sysadmin cares

Three production scenarios apply:

1. Half-migrated config. A values file applied during the upgrade is the new chart plus the previous values. Some renames are accepted silently (the new key is honoured alongside the old key for one release) and some are not. The ones that are not produce a cluster that runs in an unintended default for the unrecognised key.
2. Storage layer reorganisation. The storage block has been reorganised several times across the 2.x series. A config that worked in the simple-scalable default may not work in the microservices default. The marker is the presence of a missing storage sub-section.
3. Workload kind change. The chart changes a workload from Deployment to StatefulSet or vice versa, and the Service selector no longer matches the new pod labels. Symptom: the metrics-generator or compactor Service points at nothing because the selector was not updated in the values.

The cost of a missed config migration is a cluster that runs, accepts traffic, and silently degrades. The metrics are still green; the dashboards still load; the saved panels show no data.

How it works

The config migration is a side-effect of the binary version change. The binary accepts a YAML map; the chart renders a tempo.yaml from the values. The cutover has three steps:

   Running values (2.5)
        |
        v
   +-------------------+   chart upgrade produces
   | helm diff         |   new values and new chart
   |                   |   defaults
   +-------------------+
        |
        v
   +-------------------+   review the diff line by line;
   | Read diff         |   flag any removed key that is
   |                   |   still in the values; flag any
   +-------------------+   renamed key with a new default
        |
        v
   +-------------------+   apply the new values file,
   | Apply diff        |   watch the roll, watch the
   |                   |   metrics
   +-------------------+
        |
        v
   +-------------------+   validation: every metric
   | Validate          |   bound by the previous config
   |                   |   still emits after the cutover
   +-------------------+

The diff is the input. The discipline is the human review.

Under the hood

How to configure it

The tempo.yaml below is a minimal microservices config that demonstrates the keys most affected by recent renames.

# tempo.yaml -- keys that move between minor versions
server:
  http_listen_port: 3200
  grpc_listen_port: 9095

distributor:
  receivers:
    otlp:
      protocols:
        grpc: { endpoint: '0.0.0.0:4317' }
        http: { endpoint: '0.0.0.0:4318' }
    zipkin:
      endpoint: '0.0.0.0:9411'

ingester:
  trace_idle_period: 10s
  max_block_duration: 30m
  lifecycler:
    ring:
      kvstore: { store: memberlist }
      replication_factor: 3
      heartbeat_timeout: 5m   # pinned; the default was tightened
    join_after: 30s           # pinned; was 10s in older versions

querier:
  frontend_worker:
    frontend_address: tempo-query-frontend:9095

query_frontend:
  max_concurrent_queries: 200
  results_cache:
    cache:
      embedded_cache:
        max_size_items: 1024
        ttl: 1h

compactor:
  compaction:
    block_retention: 168h
    compaction_window: 1h
    block_search_encoding: v2
    search_relevant_tag_values:
      - resource.service.name
      - span.http.method
      - span.http.status_code

storage:
  trace:
    backend: s3
    s3:
      bucket_name: tempo-traces-prod
      region: eu-west-1
    wal:
      path: /var/tempo/wal

metrics_generator:
  registry:
    external_labels:
      source: tempo
  # Service-graph generator enabled; service_graph is the
  # newer spelling (was servicegraph).
  service_graph:
    enabled: true

Five keys to call out:

• ingester.lifecycler.ring.heartbeat_timeout: 5m is pinned explicitly. The default was tightened across minor releases. Without pinning, a chart upgrade changes the running cluster silently.
• compactor.compaction.search_relevant_tag_values is the replacement for search_relevant_tag. The values file that still names the old key produces the chart default and the dashboard silently empty.
• metrics_generator.service_graph is the new spelling replacing the older servicegraph key. The values file that still uses the old name shows the metric receiving zero service_graph_total updates.
• query_frontend.results_cache.cache.embedded_cache was renamed from results_cache.cache_config. The old key continues to be accepted silently with a log warning.
• distributor.receivers.jaeger is removed in 2.7. The comment in the values file marks the removal explicitly so the diff is visible in the chart upgrade.

How to validate it

Severity: READ-ONLY.

1. Diff the running values against the target. The helm diff upgrade command shows every key that the chart defaults are about to override:

helm diff upgrade tempo grafana/tempo \
  --namespace observability \
  --values values-running.yaml \
  --values values-target.yaml \
  | tee /tmp/tempo-diff.txt

The diff output lists every default that changed. A line beginning with - is the previous value; a line beginning with + is the new value; the lines in between are the context.

2. Confirm the config loads after the cutover. The pod log lists the role that started and any unexpected key warnings:

kubectl -n observability logs deploy/tempo-ingester --tail=20 \
  | grep -E "unexpected|listening on|target"
# level=info msg="starting" target=ingester
# level=warn msg="unexpected config key" key=legacy_heartbeat

A unexpected config key line is a renamed or removed key that is still in the values file. The default applies.

3. Confirm per-component unexpected key warnings. A wholesale list is the cleanest way to confirm:

for role in distributor ingester querier query-frontend compactor metrics-generator; do
  echo -n "$role: "
  kubectl -n observability logs deploy/tempo-${role} --tail=50 \
    | grep -E "unexpected config key" \
    | awk '{print $NF}' \
    | sort -u \
    | paste -sd ','
done
# distributor:
# ingester:
# querier: legacy_heartbeat
# query-frontend:
# compactor:
# metrics-generator: servicegraph

4. Confirm the metric bound by the renamed key is still emitting with the expected label value:

curl -s http://tempo-ingester:3200/metrics \
  | grep '^tempo_ingester_lifecycler_ring_active_members'
# tempo_ingester_lifecycler_ring_active_members{state="ACTIVE"} 3

5. Confirm the renamed cache key applied. The query-frontend exposes the in-memory cache size as a metric:

curl -s http://tempo-query-frontend:3200/metrics \
  | grep '^tempo_query_frontend_results_cache'
# tempo_query_frontend_results_cache_items 412

How it can fail

Six shapes appear in production tempo config migrations:

1. Silently renamed key. The renamed key is no longer mapped; the new key uses the default. Symptom: behaviour change that matches the default, not the pinned value.
2. Restructured key at the wrong nesting. A key that lived under compactor.compaction is moved to compactor.compaction.sub_block. The values file still has the old nesting. Symptom: same as above, with an unexpected config key warning at start.
3. Removed receiver still in the block. A values file that still references distributor.receivers.jaeger after 2.7 loads but binds nothing. Symptom: distributor log shows unknown receiver and the metric label for Jaeger is absent.
4. Workload kind change. The chart bumps a Deployment to a StatefulSet; the Service selector was not migrated in the values. Symptom: the Service has no endpoints; the new pod identity has no DNS entry.
5. Workload selector changed. The chart bumps the pod label selector; the PodDisruptionBudget still matches the old label. Symptom: PDB blocks evictions that the chart wants to perform.
6. Cache key renamed. cache_config is now cache.embedded_cache. A values file that still has the old key shows unexpected config key and the in-memory cache size is the new default.

How to troubleshoot it

The diagnostic order matters. Each step rules out one failure mode:

1. Did the diff run? helm diff upgrade should produce an output of non-zero length for every chart upgrade. An empty diff means nothing changed.
2. Did the pod log unexpected config key warnings at start? The warnings list the renamed or removed keys that are still in the values file.
3. Are the metrics for the renamed key missing? A metric that was bound by a pinned value is the confirmation that the renamed key took effect.
4. Are the workload kinds matching the Service selectors? kubectl get endpoints on the Service returns the pod list. An empty endpoint list points at a workload-kind mismatch.
5. Are the renamed receivers bound? The pod log lists every bound port; a renamed receiver should appear under its new name.

Security implications

The config migration does not change the data, but it changes the surface:

• Receiver surface renames. A receiver that is renamed still binds the port; a receiver that is removed does not. The NetworkPolicy that whitelists a removed port is a stale policy.
• Cache authentication. The cache.cache block carries optional authentication (mTLS or SASL). A rename that swaps a config key without swapping the auth block leaves the cache unauthenticated.
• Bucket credentials. The storage block reorganised the credential storage scope. A migration that leaves the bucket credentials at the old YAML location produces a credentials not found log line at start and the cluster falls back to the IAM instance role.

Performance implications

• Default changes. A default that was tightened (e.g. heartbeat timeout) reduces the ingester ring churn budget. The cluster runs faster on paper but produces more roll failures during a normal restart.
• Cache defaults. The cache defaults were tightened across recent minors. A rename that applies the new default without honouring the previously-pinned TTL produces shorter cache validity and more querier fan-out.
• Block size defaults. The ingester.max_block_duration default was tightened in a recent minor. A migration that does not pin the value produces smaller blocks and a higher block count.

Production guidance

• Always run helm diff upgrade against a staging cluster or in --dry-run mode before the production upgrade. The diff output is the input to the review.
• Pin every value the chart default would override regardless of version. Diff against the pinned values, not against the chart defaults.
• Keep a copy of the previous values file in git. The rollback path is the previous values file plus the previous container image tag.
• After every upgrade, scan the pod logs for unexpected config key warnings. The warnings list is the proof that the rename was visible to the binary.

Verification

You should now be able to answer:

• Name the four categories of config change between Tempo minor releases.
• What is the role of helm diff upgrade in a Tempo config migration?
• What does the unexpected config key warning at start indicate about the values file?
• Which config key was moved from search_relevant_tag to search_relevant_tag_values, and which release line moved it?
• Why does a values file that still contains distributor.receivers.jaeger not bind the port after Tempo 2.7?

Quiz