Skip to main content
RunBook Academy

KubernetesXC · Distributed TracingTracing

Tracing + metrics + logs — the three signals integrated

Advanced⏱ ~13 minkubectlopentelemetrylokiprometheusjaegergrafana

What you'll learn

  • Integrate the three signals
  • Use the trace IDs for correlation
  • Configure the OTel Collector as the central pipeline
  • Plan the production architecture

Prerequisites

Verified against Kubernetes 1.34.x · kubeadm 1.34.x · kubectl 1.34.x · etcd 3.6.x · CoreDNS 1.11.x · containerd 1.7.x / 2.x · 2026-08-16

Not yet marked complete on this device.

The three signals (traces, metrics, logs) are the observability pillars. The trace IDs are the correlation across signals. The OTel Collector is the central pipeline. The Grafana is the integration point. This lesson walks the integration, the correlation, the pipeline, and the production patterns.

The three signals

The three signals:

flowchart LR
    A[Application] --> B[Traces]
    A --> C[Metrics]
    A --> D[Logs]
    B --> E[Jaeger/Tempo]
    C --> F[Prometheus]
    D --> G[Loki]
    E --> H[Grafana]
    F --> H
    G --> H

The three signals are the observability pillars.

The correlation

The correlation:

{
  "timestamp": "2026-08-16T10:00:00.000Z",
  "level": "INFO",
  "message": "Request processed",
  "trace_id": "0af7651916cd43dd8448eb211c80319c",
  "span_id": "b7ad6b7169203331"
}

The trace IDs in the logs enable the correlation.

The OTel Collector as the central pipeline

The OTel Collector:

flowchart LR
    A[Application] --> B[OTel SDK]
    B --> C[OTel Collector]
    C --> D[Jaeger]
    C --> E[Prometheus]
    C --> F[Loki]

The OTel Collector is the central pipeline.

The Grafana integration

The Grafana integration:

flowchart LR
    A[Grafana] --> B[Prometheus datasource]
    A --> C[Loki datasource]
    A --> D[Jaeger/Tempo datasource]
    B --> E[Metrics]
    C --> F[Logs]
    D --> G[Traces]
    E --> H[Dashboard]
    F --> H
    G --> H

The Grafana is the integration point.

The cross-signal navigation

The cross-signal navigation:

flowchart LR
    A[Grafana dashboard] --> B[Prometheus metric]
    B --> C[See logs]
    C --> D[Loki logs]
    D --> E[See trace]
    E --> F[Jaeger trace]
    F --> G[See metrics]
    G --> B

The navigation is the cross-signal.

The OTel Collector exporters

The exporters:

exporters:
  prometheus:
    endpoint: 0.0.0.0:8889
  loki:
    endpoint: http://loki:3100/loki/api/v1/push
  jaeger:
    endpoint: jaeger:14250
    tls:
      insecure: true

The exporters send the signals to the backends.

The trace-based metrics

The trace-based metrics:

processors:
  spanmetrics:
    metrics_exporter: prometheus
    latency_histogram_buckets: [2ms, 10ms, 50ms, 100ms, 500ms, 1s, 5s]
    dimensions:
      - name: http.method
      - name: http.status_code

The spanmetrics processor generates metrics from the traces.

The correlation in logs

The correlation in logs:

import (
    "go.opentelemetry.io/otel/trace"
)

log.Info("Request processed",
    "trace_id", trace.SpanContextFromContext(ctx).TraceID().String(),
    "span_id", trace.SpanContextFromContext(ctx).SpanID().String(),
)

The trace IDs are added to the logs.

The SLO-driven observability

The SLO-driven observability:

flowchart LR
    A[Service] --> B[Metrics + SLO]
    A --> C[Traces for errors]
    A --> D[Logs for context]
    B --> E[Alert]
    C --> E
    D --> E
    E --> F[Runbook]

The SLOs are the integration point.

The production patterns

The production patterns:

flowchart LR
    A[Application] --> B[OTel SDK]
    B --> C[OTel Collector]
    C --> D[Prometheus]
    C --> E[Loki]
    C --> F[Jaeger]
    D --> G[Grafana]
    E --> G
    F --> G
    G --> H[Dashboard]
    G --> I[Alert]

The production pattern is the integration.

The cross-course references

The Observability course covers the integration in detail.

  • The OpenTelemetry course covers the SDK.
  • The Prometheus course covers the metrics.
  • The Loki course covers the logs.
  • The Jaeger course covers the traces.

Quiz

Knowledge check · 4 questions

  1. Q1. What is the role of the trace IDs in the three signals?

  2. Q2. The OTel Collector is the central pipeline for the three signals.

  3. Q3. Walk the integrated observability for a workload.

    Workload: HTTP API with 3 services. The team is configuring the integrated observability.

  4. Q4. What is the spanmetrics processor in the OTel Collector?

Passing score: 75%. Answers are checked in this browser.

Production discipline

  • Integrate the three signals. Traces, metrics, logs.
  • Use the trace IDs for correlation. The integration.
  • Use the OTel Collector as the central pipeline. The integration.
  • Configure the cross-signal navigation. The Grafana.
  • Use the spanmetrics processor. The trace-based metrics.
  • Document the integration. The SDK, the backends, the navigation.

The three signals integrated is the production observability. Operating it well is via the OTel Collector, with the trace IDs for correlation, and the Grafana for the integration.