Tempo as a Data Source

An engineer opens a trace in Tempo, clicks the slowest span, and the panel shows the span with no log lines attached. The span carries a traceID field. Loki has the lines. The link is not configured. The engineer copies the trace ID, opens Loki in a new tab, and reconstructs the correlation by hand. This is what the trace-to-logs pivot exists to prevent.

A second engineer opens a service map panel. The map is empty. Tempo is up; the spans are in storage; the metrics are in Prometheus. The tracesToMetricsV1 block is missing the Prometheus UID. The service map is a wall of grey boxes with no edges. The lesson this time is about the Tempo data source: the YAML that declares it, the trace-to-logs pivot that ties a span to its log lines, the service map that derives the dependency graph from span metrics, and the search-recent-traces query path that powers ad-hoc investigation.

What it is

A Tempo data source in Grafana is a named, configured client of the Tempo HTTP API. The provisioning file declares the data source’s URL, type, credentials, TLS posture, and a small set of Tempo-specific integration options. Once provisioned, Grafana runs every Trace panel and Explore query through a server-side proxy at /api/datasources/proxy/uid/<uid>/.... The credentials live in secureJsonData on the server.

In Grafana 11.x the canonical data source type is tempo. TraceQL is the query language. Two adjacent integrations are declared on the Tempo data source: the trace-to-logs pivot (tracesToLogsV1) and the trace-to-metrics pivot (tracesToMetricsV1). The service map is the most prominent user of the latter. Tempo also exposes a search block that controls the recent-traces Explore path.

Why a sysadmin cares

Tempo data source misconfigurations are the most common cause of “we have traces but cannot use them” complaints. Three production shapes appear repeatedly:

• Trace-to-logs pivot not configured. Every investigation starts in Tempo, hits a slow span, and ends with the engineer opening Loki in a separate tab to find the corresponding log lines. The cost is paid per span per incident.
• Service map empty. The dependency graph that Grafana renders from Tempo’s span metrics depends on a Prometheus that exposes the traces_spanmetrics_* series. A Tempo data source without a configured Prometheus UID renders an empty graph and the team concludes “we do not have a service map”.
• TraceID format mismatch. Tempo ingests spans with W3C traceparent headers (32-hex IDs); a legacy OpenTelemetry SDK emits 16-hex IDs. The Tempo data source has a tracesToLogsV1.traceID format field that controls the link template; a mismatch means the link returns “trace not found” from Tempo.

How it works: the request path

   browser        grafana-server           tempo
   -------        --------------           -----
      |                |                     |
      |--TraceQL----->|                     |
      |                |--GET /api/search   |
      |                |  /api/traces/<id>  |
      |                |  BasicAuth + TLS    |
      |                |                     |
      |                |<--streams-----------|
      |<--panel data---|                     |

Three observations:

1. The browser never sees the credentials. Every trace query from the browser is rewritten by Grafana into a server-side call to /api/datasources/proxy/uid/<uid>/api/.... The credentials live in secureJsonData.
2. The trace-to-logs pivot is a client-side link, not a server-side query. Grafana renders the trace, parses the span attributes, and emits a clickable link whose URL references a Loki data source. The pivot is a configuration shape, not a query path.
3. The service map is a derived metric view. Tempo does not return a service graph; Tempo returns span metrics through a Prometheus that ingests them. Grafana queries that Prometheus for traces_spanmetrics_calls_total (and related series) and renders the dependency graph.

How to configure it

# /etc/grafana/provisioning/datasources/tempo.yml
apiVersion: 1

datasources:
  - name: tempo-prod-eu
    uid: tempo-prod-eu
    type: tempo
    access: proxy
    orgId: 1
    url: https://tempo-prod-eu.internal:3200
    isDefault: false
    editable: false

    basicAuth: true
    basicAuthUser: grafana-reader
    jsonData:
      tlsAuth: false
      tlsAuthWithCACert: true
      tlsSkipVerify: false

      # TraceID format. "w3c" emits 32-hex IDs (the OpenTelemetry
      # default); "jaeger" emits 16-hex IDs from the legacy
      # Jaeger SDK. Mismatch produces "trace not found" links.
      tracesToLogsV1:
        datasourceUid: loki-prod-eu
        tags: ['job', 'service.name', 'service.namespace']
        # Template that runs against each span to find the
        # corresponding log lines. ${__span.traceId} is the
        # span's trace ID; ${__var.<tag>} is a span attribute.
        spanStartTimeShift: '10m'
        spanEndTimeShift: '10m'
        query: 'method="${__span.tags.method}"'
        refId: 'tempo-traces-to-logs'

      # Service map: a Prometheus that holds traces_spanmetrics_*.
      # The UID must match a Prometheus data source in Grafana.
      tracesToMetricsV1:
        datasourceUid: prom-prod-eu
        tags: [{ key: 'service.name', value: 'service' }]
        refId: 'tempo-traces-to-metrics'

      # Search-recent-traces: cap on the time window the
      # operator can ask for in Explore. 1h is a sensible
      # default; raise with care.
      search:
        maxDuration: '1h'
        defaultLimit: 20

      # StreamingAvailable: Tempo supports gRPC streaming;
      # Grafana uses it for the live-tail of recent traces.
      streamingAvailable: true

      # Span bar: enables the per-span duration histogram
      # overlay on the trace timeline.
      spanBar:
        type: 'None'
    secureJsonData:
      tlsCACert: |
        -----BEGIN CERTIFICATE-----
        MIIDazCCAlOgAwIBAgIUJx...
        -----END CERTIFICATE-----
      basicAuthPassword: ${TEMPO_PASSWORD}

A few production notes on the options:

• type: tempo. Anything else produces a data source that Grafana treats as a generic backend and the TraceQL editor does not appear.
• tracesToLogsV1.datasourceUid must point at a Loki data source in the same Grafana instance. A UID that references a missing or removed data source renders a link that fails at click time.
• tracesToLogsV1.spanStartTimeShift and spanEndTimeShift widen the log-query window around the span. The default is zero; a value of 10m widens the log search by ten minutes on either side of the span, which is the right shape for spans whose log lines arrive slightly out of order.
• tracesToMetricsV1.datasourceUid is the service map’s only dependency. A wrong UID produces an empty map without an error message.
• search.maxDuration is the upper bound on the time window the operator can ask for in Explore. A larger cap means longer queries and a slower UI; the default of 1h is conservative.
• streamingAvailable enables gRPC streaming for the trace-detail panel. Disable it only if the Grafana host is behind a proxy that does not forward gRPC frames.

How to validate it

# READ-ONLY: the data source is provisioned and reachable.
curl -fsS -u grafana-admin:$GRAFANA_ADMIN \
  http://grafana.internal:3000/api/datasources/uid/tempo-prod-eu
# {
#   "id": 3,
#   "uid": "tempo-prod-eu",
#   "name": "tempo-prod-eu",
#   "type": "tempo",
#   "url": "https://tempo-prod-eu.internal:3200",
#   ...
# }

# READ-ONLY: the health check returns one of four states.
curl -fsS -u grafana-admin:$GRAFANA_ADMIN \
  http://grafana.internal:3000/api/datasources/uid/tempo-prod-eu/health
# {"message":"Data source is working","status":"success"}

# READ-ONLY: search-recent-traces through the proxy.
curl -fsS -u grafana-admin:$GRAFANA_ADMIN \
  --data-urlencode 'q={ status = error }' \
  --data-urlencode 'limit=20' \
  --data-urlencode 'start=1700000000' \
  --data-urlencode 'end=1700003600' \
  http://grafana.internal:3000/api/datasources/proxy/uid/tempo-prod-eu/api/search
# {"traces":[...],"metrics":{...}}

# READ-ONLY: a single trace by ID.
curl -fsS -u grafana-admin:$GRAFANA_ADMIN \
  http://grafana.internal:3000/api/datasources/proxy/uid/tempo-prod-eu/api/traces/24f2b8a3c5d6e7f0
# {"batches":[...]}

# CONFIGURATION: reload Grafana provisioning.
sudo systemctl reload grafana-server

A clean validation: the UID is present, the health check is green, the proxy returns Tempo traces, and clicking a span in the UI opens Loki with the corresponding log lines. The service map renders edges between services.

How it can fail

The most expensive Tempo data source failure modes from real production incidents.

1. Trace-to-logs pivot not configured. A trace renders in the panel but no “Logs for this span” link appears. The engineer copies the trace ID by hand, opens Loki, and reconstructs the correlation manually. The symptom is a missing button on every span.
2. Service map empty. The dependency graph renders the services as boxes with no edges. The cause is a missing or wrong tracesToMetricsV1.datasourceUid, or a Prometheus that does not ingest the traces_spanmetrics_* series.
3. TraceID format mismatch. A legacy Jaeger-emitted span carries a 16-hex ID; the tracesToLogsV1 block uses the W3C template. The link returns “trace not found” from Tempo. The symptom is a link that opens Tempo but reports the trace does not exist.
4. Search-recent-traces times out. The search.maxDuration is left at the default 1h, but the operator asks for 24h in Explore. The query times out. The symptom is a search bar that hangs and returns context deadline exceeded.
5. Streaming gRPC blocked. A reverse proxy does not forward gRPC frames; the trace-detail panel fails to stream new spans as they arrive. The symptom is a trace panel that “loads” but never updates during a live-tail session.
6. CA bundle drift. The private CA that signs Tempo’s certificate rotated. Grafana’s tlsCACert is stale. The health check returns an x509: certificate signed by unknown authority error in the message field.

How to troubleshoot it

The diagnostic order is “is the data source provisioned?”, “is the proxy reachable?”, “is Tempo responding?”, “are the integrations pointed at the right UIDs?”.

1. Confirm the data source exists. GET /api/datasources/uid/<uid>. If 404, the provisioning file did not reload; check grafana.log for parse errors and confirm the type is tempo.
2. Confirm the health check. GET /api/datasources/uid/<uid>/health.
3. Reproduce the request through the proxy. curl against /api/datasources/proxy/uid/<uid>/api/search?q={status=error}&limit=20. This is the simplest search-recent-traces request.
4. Reproduce the request directly. curl against the upstream Tempo URL with the same credentials. A working direct curl that fails through the proxy isolates the problem to Grafana’s secureJsonData or tlsConfig.
5. Validate the trace-to-logs pivot. Click a span in the trace-detail panel and confirm the link opens Loki with the expected log lines. A wrong datasourceUid or a stale attribute template surfaces at click time.
6. Validate the service map. Open the service map panel and confirm the edges render. A wrong datasourceUid for tracesToMetricsV1 produces an empty map without an error.
7. Inspect Grafana’s logs. /var/log/grafana/grafana.log records every proxy call. A 404 Not Found from Tempo appears with the trace ID; a timeout appears as context deadline exceeded.

Security implications

• The proxy holds the credentials. A Tempo data source with access: direct exposes the password to every browser.
• The trace-to-logs pivot is a cross-data-source link. A link template that leaks a span attribute into the URL is a log of every query the operator runs. Audit the templates for sensitive attributes.
• search.maxDuration is a DoS knob. A Tempo data source without a cap can be driven into a slow-query storm by a single Explore session.
• tracesToMetricsV1 exposes the Prometheus UID. The service map’s Prometheus is reachable from the Grafana server; the same network ACL that protects the Loki UID applies here.

Performance implications

• Trace-detail streaming is gRPC. The reverse proxy must forward gRPC frames; otherwise the live-tail session polls every second instead of streaming.
• search.maxDuration is the upper bound. A larger cap means longer queries and a slower UI. The default of 1h is conservative for ad-hoc investigation; raise it only when the operator workflow demands it.
• The service map is bounded by the Prometheus query budget. A service map with thousands of services takes longer to render than one with fifty. Bound the map’s services explicitly in the panel.
• The trace-to-logs pivot’s spanStartTimeShift widens the log query. A 10-minute shift on either side doubles or triples the log-query workload per span click; raise only when out-of-order spans are common.

Production guidance

• Pin the uid. Renaming a Tempo data source is fine; changing the UID breaks every trace-to-logs and trace-to-metrics reference.
• Use access: proxy for every Tempo with credentials.
• Configure the trace-to-logs pivot for the trace ID and the request ID before the first incident that needs them. Add the rules in code review, not during a Sev 1.
• Bound the service map in the panel, not in the data source YAML. The data source declares the Prometheus UID; the panel decides which services to render.
• Periodically exercise the trace-to-logs and trace-to-metrics pivots. A pivot that has silently drifted is invisible until an engineer clicks it.
• Reload Grafana provisioning through the GitOps pipeline.

Verification

You should now be able to answer:

• What does the tracesToLogsV1 block do at click time, and which fields decide whether the link opens the correct Loki query?
• How does the service map in Tempo derive the dependency graph, and which UID is its only dependency?
• What is the difference between search.maxDuration and streamingAvailable in the Tempo data source, and which one affects the trace-detail panel rather than Explore?
• Why does a traceID format mismatch surface as “trace not found” rather than as a configuration error?

Quiz