The Metrics Generator

A platform team wants RED metrics for every service in the fleet, but every service team refuses to instrument Prometheus exposition endpoints. The team enables the Tempo metrics- generator. Within an hour the Grafana dashboard shows rate, errors, and duration per service derived from the spans that are already flowing. No code change in any service.

This lesson describes the Tempo metrics-generator: how it reads spans from the ingester pipeline, what metrics it derives, and how it ships them to Prometheus.

What it is

The Tempo metrics-generator is a stateless component that consumes spans from the ingester write path and produces Prometheus metrics. Two processors ship in the box:

1. Service graphs. Walks the parent/child relationships between spans and emits traces_service_graph_request_total and traces_service_graph_request_failed_total per (client, server) pair. The result is a topology of which services call which, plus failure rates per edge.
2. Span metrics. Aggregates every leaf span into traces_spanmetrics_calls_total, traces_spanmetrics_latency_bucket, and traces_spanmetrics_latency_sum per (service, span_name) pair. The result is the RED metrics for every service in the fleet.

Both processors push to a Prometheus remote-write endpoint. The result is a Grafana dashboard of RED metrics that required no SDK change in any service.

Why a sysadmin cares

Three operational pains are specific to the metrics-generator:

1. Cardinality explosion. A span_name that includes a path parameter (GET /users/12345) produces one series per distinct path. A /users/{id}/orders/{id} endpoint produces millions of series. The metrics-generator exposes the cardinality; Prometheus pays the storage cost.
2. Remote-write backpressure. The metrics-generator buffers in memory until remote-write accepts the batch. A slow Prometheus or a throttled network produces a buffer that grows until the metrics-generator OOMs.
3. Span-rate dependence. The metrics-generator sees only the spans Tempo receives. If a service over-samples or under- samples, the metrics are skewed. If a service drops a sampling decision in favour of “no traces”, the metrics are silent.

How it works

The metrics-generator runs in the ingester pipeline:

  Spans arrive at distributor
            |
            v
  Distributor forwards to ingesters
            |
            v
  +-------------------------------------------+
  |                  Ingester                 |
  |                                           |
  |   write span to head block                |
  |             |                             |
  |             v                             |
  |   metrics-generator registry list         |
  |             |                             |
  |   +---------+---------+                   |
  |   |                   |                   |
  |   v                   v                   |
  |   service-graphs    span-metrics          |
  |   processor         processor             |
  |   |                 |                     |
  |   v                 v                     |
  |   parent/child      leaf-span aggregation |
  |   edges             per service/span      |
  |   |                 |                     |
  |   +--------+--------+                     |
  |            v                              |
  |   remote-write batcher                     |
  |            |                              |
  |            v                              |
  |   Prometheus remote-write                  |
  +-------------------------------------------+

Two production details to call out:

• In-pipeline, not sidecar. The metrics-generator runs as part of the ingester process. It is not a separate collector. The cost is one process per ingester pod rather than a separate deployment.
• Aggregation interval. Both processors accumulate metrics in memory and flush them to Prometheus on metrics_generator.collection_interval (default 15 s). The remote-write endpoint receives the batch and persists it.

How to configure it

A production metrics-generator config defines the two processors, the remote-write endpoint, and the collection interval:

metrics_generator:
  registry:
    # How often each processor flushes its registry to remote-write.
    collection_interval: 30s

  # Where the metrics are pushed. Either a Prometheus remote-write
  # endpoint or a Prometheus Agent. The metrics-generator does not
  # expose Prometheus metrics over /metrics for these registries.
  remote_write:
    - url: 'http://prometheus.internal:9090/api/v1/write'

  # Service graphs: parent/child edges with failure counts.
  service_graphs:
    enabled: true
    # Wait this long for a server span to arrive before recording
    # the edge as failed. The default is too aggressive for services
    # with high tail latency; raise it for slow services.
    propagation_delay: 5s
    # Drop edges between services in the same process. Reduces
    # cardinality for monolithic apps.
    enable_in_db_processing: false

  # Span metrics: per-leaf RED metrics.
  span_metrics:
    enabled: true
    # Histogram buckets in seconds. Default is the OpenTelemetry
    # default; tune for your service's latency profile.
    latency_histogram_buckets: [0.002, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10]
    # Include the span kind as a label. Useful for filtering
    # SERVER vs CLIENT spans; doubles cardinality.
    span_label_key_filter:
      include:
        - http.method
        - http.status_code
        - http.url

Three production details to call out:

• collection_interval: 30s produces one scrape per minute per service. Shorter intervals produce more remote-write traffic and more accurate dashboards; longer intervals reduce cost and smooth over short-lived spikes.
• propagation_delay: 5s is the right default for fast services. For services with multi-second tails, raise it to avoid marking slow-but-successful edges as failed.
• latency_histogram_buckets defines the buckets for traces_spanmetrics_latency_bucket. The defaults are tuned for web services; adjust them for batch jobs or real-time systems.

How to validate it

Five checks confirm the metrics-generator is doing its job:

1. Confirm the metrics-generator is running. Its metrics appear on the ingester’s /metrics endpoint:

curl -s http://tempo.internal:3200/metrics \
  | grep -E '^tempo_metrics_generator_registry' | head
# tempo_metrics_generator_registry_active_series{registry="span-metrics",tenant="team-checkout"}  8412

2. Confirm service-graph metrics are arriving in Prometheus. The metric name is traces_service_graph_request_total:

curl -sG http://prometheus.internal:9090/api/v1/query \
  --data-urlencode 'query=traces_service_graph_request_total' \
  | jq '.data.result | length'
# 24

3. Confirm span-metrics RED metrics are arriving:

curl -sG http://prometheus.internal:9090/api/v1/query \
  --data-urlencode 'query=traces_spanmetrics_calls_total{service_name="checkout"}' \
  | jq '.data.result[0].value[1]'
# "18412"

4. Confirm the metrics-generator is consuming spans. The tempo_metrics_generator_registry_processed_spans_total counter should rise in lockstep with the ingester’s span counter:

curl -s http://tempo.internal:3200/metrics \
  | grep tempo_metrics_generator_registry_processed_spans_total
# tempo_metrics_generator_registry_processed_spans_total{registry="span-metrics",tenant="team-checkout"}  4821

5. Confirm the remote-write path is healthy. The tempo_metrics_generator_remote_write_sent_bytes_total counter should rise at the configured interval:

curl -s http://tempo.internal:3200/metrics \
  | grep tempo_metrics_generator_remote_write_sent_bytes_total
# tempo_metrics_generator_remote_write_sent_bytes_total  4823104

How it can fail

Five shapes appear repeatedly:

1. Cardinality explosion. A developer emits span.name = "GET /users/12345". The traces_spanmetrics_calls_total series count grows without bound. Symptom is the Prometheus TSDB size growing past plan and tempo_metrics_generator_registry_active_series reaching the configured limit.
2. Remote-write backpressure. The Prometheus remote-write endpoint is slow. The metrics-generator buffer grows until the ingester OOMs. Symptom is tempo_metrics_generator_remote_write_buffered_bytes rising alongside go_memstats_heap_inuse_bytes.
3. propagation_delay too short. A server that normally responds in 8 s gets marked as failed after 5 s. The service-graph failure rate is wrong. Symptom is traces_service_graph_request_failed_total rising while the application’s own metrics show success.
4. collection_interval too long. Dashboards show stale data. The 15-minute RED dashboard updates every 30 seconds instead of every 5. Symptom is user complaints about “the dashboard is lagging”.
5. Prometheus is down. The remote-write endpoint stops accepting batches. The metrics-generator buffers until the buffer cap is reached and drops metrics. Symptom is tempo_metrics_generator_remote_write_dropped_bytes_total rising.

How to troubleshoot it

The diagnostic order:

1. Is the metrics-generator enabled? Check the running config. A metrics-generator block missing from the YAML produces no metrics. The tempo_metrics_generator_registry metric series will be absent.
2. Are spans reaching the processors? Check tempo_metrics_generator_registry_processed_spans_total. A flat counter despite active ingester traffic means the processors are not consuming.
3. Is the registry cardinality bounded? Check tempo_metrics_generator_registry_active_series per tenant. A tenant whose active series count exceeds the plan is the source of the cost.
4. Is remote-write succeeding? Check tempo_metrics_generator_remote_write_sent_bytes_total. A flat counter despite processed spans means remote-write is failing.
5. Is Prometheus accepting the metrics? Query traces_spanmetrics_calls_total and traces_service_graph_request_total directly. If they are absent in Prometheus but present in remote_write_sent_bytes_total, Prometheus is rejecting the series.
6. Are dashboards stale? Check the time-since-last-scrape for the relevant dashboard panel. A panel whose last value is older than collection_interval points at a pipeline stall between metrics-generator and Grafana.

Security implications

The metrics-generator pushes metrics to a remote endpoint:

• Remote-write endpoint. The metrics-generator sends metrics over HTTP or gRPC. The endpoint URL should be internal; remote-write over the public internet is a leak vector.
• Tenant isolation. Each tenant’s metrics are pushed to a separate remote-write stream. Multi-tenant deployments must enforce that one tenant’s metrics cannot bleed into another tenant’s Prometheus.
• Sensitive labels. Span attributes can include secrets (URLs with tokens, query parameters with PII). The span_label_key_filter.include block is the discipline: the default is empty; including http.url produces a metrics series per distinct URL.

Performance implications

The metrics-generator is memory- and network-bound:

• Memory. Each registry holds one counter and one histogram per distinct label combination. A tenant with 10,000 distinct (service, span_name) pairs consumes tens of megabytes of memory.
• CPU. Counter and histogram updates are cheap. The expensive part is the remote-write encoding at collection time.
• Network. Each collection interval sends the registry to Prometheus. A 100 KiB registry and 30 s collection interval produce roughly 3 KiB/s of remote-write traffic per tenant.
• Prometheus. A tenant with 10,000 series adds 10,000 series to the TSDB. A platform with 50 such tenants adds half a million series. Storage and query cost scale linearly.

Production guidance

• Start with collection_interval: 30s. Lower intervals are more accurate but more expensive.
• Bound cardinality with span_label_key_filter.include and a short latency_histogram_buckets list. The defaults are conservative; widen them only when a specific dashboard needs it.
• Use a separate Prometheus for metrics-generator output. The derived metrics have a different shape than the application metrics; mixing them makes alerts harder to reason about.
• Alert on tempo_metrics_generator_remote_write_dropped_bytes_total. A non-zero value means remote-write is failing.

Verification

You should now be able to answer:

• What two processors ship in the Tempo metrics-generator?
• What is the difference between service-graph metrics and span metrics?
• How does the metrics-generator push metrics to Prometheus?
• What is the cardinality risk of including http.url as a span label?
• What happens to the tracing pipeline when the metrics- generator is down?

Quiz