Distributed tracing — the request journey

Distributed tracing is the discipline of tracking the request journey across services. The trace is the request; the span is the unit of work. The context propagation is the W3C traceparent header. The OpenTelemetry SDK is the standard. This lesson walks the tracing, the spans, the context, and the integration.

What is distributed tracing

The distributed tracing tracks the request journey:

sequenceDiagram
    participant C as Client
    participant A as API
    participant B as Backend
    participant D as Database
    C->>A: GET /api/users (trace_id: abc)
    A->>B: forward (span_id: def)
    B->>D: SELECT * (span_id: ghi)
    D-->>B: result
    B-->>A: response
    A-->>C: 200 OK

The trace is the journey; the spans are the segments.

The trace

The trace:

{
  "trace_id": "abc123",
  "spans": [
    {
      "span_id": "def",
      "name": "GET /api/users",
      "start_time": "2026-08-16T10:00:00.000Z",
      "duration_ms": 100,
      "attributes": {
        "http.method": "GET",
        "http.url": "/api/users"
      }
    }
  ]
}

The trace is the JSON of the request.

The span

The span:

{
  "span_id": "def",
  "trace_id": "abc123",
  "parent_span_id": "ghi",
  "name": "GET /api/users",
  "start_time": "2026-08-16T10:00:00.000Z",
  "duration_ms": 100,
  "attributes": {
    "http.method": "GET",
    "http.url": "/api/users"
  },
  "events": [
    {
      "name": "exception",
      "time": "2026-08-16T10:00:00.050Z",
      "attributes": {
        "exception.type": "DatabaseError",
        "exception.message": "connection refused"
      }
    }
  ]
}

The span is the unit of work.

The context propagation

The context propagation:

GET /api/users HTTP/1.1
Host: api.example.com
traceparent: 00-abc123-def456-01

The traceparent header is the W3C Trace Context. The header is propagated across services.

sequenceDiagram
    participant C as Client
    participant A as API
    participant B as Backend
    C->>A: GET /api/users with traceparent
    A->>B: forward with traceparent
    B-->>A: response
    A-->>C: 200 OK
    Note over C,B: trace_id is the same. span_id differs

The context propagation is the link.

The OpenTelemetry SDK

The OpenTelemetry SDK:

import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/trace"
)

tracer := otel.Tracer("my-app")
ctx, span := tracer.Start(ctx, "GET /api/users")
defer span.End()

// Add attributes
span.SetAttributes(
    attribute.String("http.method", "GET"),
    attribute.String("http.url", "/api/users"),
)

// Record an error
if err != nil {
    span.RecordError(err)
    span.SetStatus(codes.Error, err.Error())
}

The SDK is the standard.

The OpenTelemetry Collector

The OpenTelemetry Collector:

flowchart LR
    A[Application] --> B[OTel SDK]
    B --> C[OTel Collector]
    C --> D[Jaeger]
    C --> E[Tempo]
    C --> F[Prometheus]
    C --> G[Loki]

The Collector is the central pipeline.

The trace backend

The trace backend:

The backend is the trace storage.

The Jaeger architecture

The Jaeger architecture:

flowchart LR
    A[Application] --> B[Jaeger agent]
    B --> C[Jaeger collector]
    C --> D[Storage]
    D --> E[Jaeger query]
    E --> F[Jaeger UI]

The Jaeger architecture is the trace path.

The OpenTelemetry SDKs

The OpenTelemetry SDKs:

• Go: go.opentelemetry.io/otel
• Python: opentelemetry-python
• Java: io.opentelemetry:opentelemetry-api
• Node.js: @opentelemetry/api
• .NET: OpenTelemetry.Api

The SDKs are the language-specific libraries.

The trace sampling

The trace sampling:

import (
    "go.opentelemetry.io/otel/sdk/trace"
)

sampler := trace.TraceIDRatioBased(0.1)  # 10% sampling
tracerProvider := sdktrace.NewTracerProvider(
    sdktrace.WithSampler(sampler),
)

The sampling is the head-based sampling.

The trace context

The trace context:

{
  "trace_id": "abc123",
  "span_id": "def456",
  "trace_flags": "01",  # 01 = sampled, 00 = not sampled
  "trace_state": "vendor1=value1,vendor2=value2"
}

The trace context is the W3C standard.

The cross-course references

The Observability course covers the tracing in detail.

• The OpenTelemetry course covers the SDK and the Collector.
• The Jaeger course covers the trace backend.
• The Grafana course covers the dashboards.

Quiz

Production discipline

• Use the OpenTelemetry SDK. The standard.
• Configure the context propagation. The traceparent header.
• Configure the OTel Collector. The central pipeline.
• Use a trace backend. Jaeger, Tempo, Zipkin.
• Configure the sampling. 10% head-based for production.
• Document the tracing. The SDK, the backend, the sampling.

The distributed tracing is the cluster’s request journey. Operating it well is via the OpenTelemetry SDK, with the context propagation, and the trace backend.