Jaeger and Tempo — the trace backends

Jaeger and Tempo are the trace backends. Jaeger is the traditional choice; Tempo is the Grafana-integrated choice. Both backend the OpenTelemetry traces. This lesson walks the architectures, the deployments, the storage, and the production patterns.

The Jaeger deployment

The Jaeger deployment:

helm repo add jaegertracing https://jaegertracing.github.io/helm-charts
helm install jaeger jaegertracing/jaeger \
  --namespace monitoring \
  --set storage.type=elasticsearch \
  --set storage.elasticsearch.serverList=http://elasticsearch:9200

The Helm chart deploys the Jaeger.

The Jaeger architecture

The Jaeger architecture:

flowchart LR
    A[Application] --> B[Jaeger agent]
    B --> C[Jaeger collector]
    C --> D[Storage]
    D --> E[Jaeger query]
    E --> F[Jaeger UI]

The Jaeger architecture is the trace path.

The Tempo deployment

The Tempo deployment:

helm repo add grafana https://grafana.github.io/helm-charts
helm install tempo grafana/tempo \
  --namespace monitoring \
  --values tempo-values.yaml

The Helm chart deploys the Tempo.

# tempo-values.yaml
storage:
  trace:
    backend: s3
    s3:
      bucket: my-tempo-bucket
      endpoint: s3.amazonaws.com
      access_key: <key>
      secret_key: <key>

The Tempo uses S3 for storage.

The Tempo architecture

The Tempo architecture:

flowchart LR
    A[Application] --> B[OTel Collector]
    B --> C[Tempo distributor]
    C --> D[Tempo ingester]
    D --> E[S3]
    F[Tempo querier] --> E
    G[Grafana] --> F

The Tempo architecture is the trace path.

The Jaeger vs Tempo

The Jaeger vs Tempo:

The choice is per cluster.

The Jaeger with Elasticsearch

The Jaeger with Elasticsearch:

# Jaeger config
storage:
  type: elasticsearch
  elasticsearch:
    serverUrls: http://elasticsearch:9200
    indexPrefix: jaeger

The Elasticsearch is the storage.

The Jaeger with Cassandra

The Jaeger with Cassandra:

# Jaeger config
storage:
  type: cassandra
  cassandra:
    servers: cassandra:9042
    keyspace: jaeger

The Cassandra is the storage.

The Grafana integration with Tempo

The Grafana integration with Tempo:

flowchart LR
    A[Grafana] --> B[Tempo datasource]
    B --> C[Tempo]
    C --> D[S3]

The Grafana integrates with Tempo via the Tempo datasource.

The Grafana integration with Jaeger

The Grafana integration with Jaeger:

flowchart LR
    A[Grafana] --> B[Jaeger datasource]
    B --> C[Jaeger]
    C --> D[Storage]

The Grafana integrates with Jaeger via the Jaeger datasource.

The TraceQL queries

The TraceQL queries:

# Find traces for a service
{ resource.service.name = "nginx" }

# Find traces with errors
{ resource.service.name = "nginx" && status = error }

# Find traces with duration > 1s
{ resource.service.name = "nginx" && duration > 1s }

# Aggregate
{ resource.service.name = "nginx" } | count()

The TraceQL is the query language.

The trace-based alerts

The trace-based alerts:

groups:
  - name: trace-alerts
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate(traces_spanmetrics_latency_count{status="error"}[5m])) > 1
        for: 5m
        labels:
          severity: warning

The trace-based alerts are the input for the Alertmanager.

The cross-course references

• The OpenTelemetry course covers the SDK.
• The Prometheus course covers the metrics.
• The Grafana course covers the dashboards.

Quiz

Production discipline

• Choose Jaeger or Tempo per cluster. The Grafana integration is the differentiator.
• Use the OTel Collector. The central pipeline.
• Configure the storage. S3 for Tempo; ES for Jaeger.
• Configure the Grafana datasource. The integration.
• Use the TraceQL queries. The dashboards and alerts.
• Document the deployment. The Helm values, the storage.

The Jaeger and Tempo are the trace backends. Operating it well is via the OTel Collector, with the storage, and the Grafana integration.