OTel into Grafana and Tempo - the visual stack

Grafana is the visualisation layer for OTel data. Tempo is the trace store. This lesson covers the integration and how to correlate metrics, logs, and traces in a single dashboard.

Grafana data sources

For OTel data, configure Grafana data sources:

• Prometheus (metrics): scrape the OTel Collector Prometheus exporter.
• Loki (logs): ship to Loki, query via LogQL.
• Tempo (traces): store OTLP traces, query via TraceQL.

Grafana
├── Prometheus data source (http://prometheus:9090)
├── Loki data source (http://loki:3100)
└── Tempo data source (http://tempo:3200)

Tempo for traces

Tempo is Grafana’s trace store. It accepts OTLP traces:

# /etc/tempo/tempo.yaml
server:
  http_listen_port: 3200

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
        http:

storage:
  trace:
    backend: local
    local:
      path: /var/tempo/traces
    pool:
      max_workers: 100

ingester:
  max_block_duration: 5m

Tempo stores traces locally or in object storage (S3, GCS, Azure).

Correlate in Grafana

Grafana’s correlation feature ties metrics, logs, and traces:

1. Open a metric panel (e.g. p99 latency).
2. Click a data point.
3. Choose “Query traces” - Grafana shows the traces for that time range.
4. Click a trace - see the full request flow.

This is the power of OTel: one identifier (the trace ID) ties everything together.

Example dashboard

A typical request-flow dashboard:

• Top: request rate, error rate, latency p50/p99 (Prometheus).
• Middle: trace count by service, slow traces (Tempo).
• Bottom: error logs (Loki).

A single click on a slow trace shows the spans and the associated logs.

Query OTel data

In Grafana Explore:

# Tempo
{service.name = "checkout"} | duration > 500ms

# Loki
{service_name="checkout"} |= "error"

# Prometheus
histogram_quantile(0.99, rate(http_server_request_duration_seconds_bucket{service="checkout"}[5m]))

The query language differs per backend, but the identifiers are consistent.

Alert on OTel data

For Prometheus metrics (from OTel Collector):

# Latency p99 > 1 second
histogram_quantile(0.99, rate(http_server_request_duration_seconds_bucket{service="checkout"}[5m])) > 1

For Loki logs (from OTel log exporter):

# Alert when a service logs more than 10 errors in 5 minutes
sum by (service_name) (
  count_over_time({service_name="checkout"} |= "error" [5m])
) > 10

count_over_time is a range-aggregation function: it takes a log-range selector as its argument, so the range goes inside the call and the whole expression wraps the stream selector. Piping into it — {...} |= "error" | count_over_time([5m]) — is not LogQL syntax; | chains line and label filters, not aggregations, and Loki rejects the query at parse time.

Wrap the result in sum by (...) as well. Without an aggregation you get one series per distinct label set, so a > 10 comparison fires per stream rather than per service.

For Tempo traces:

{ service.name = "checkout" && status = error }

Note the language: that is TraceQL, which Tempo evaluates over spans. It shares braces and label syntax with LogQL and is a different query language with different functions.

Knowledge check