OpenTelemetry — the SDK and the Collector

OpenTelemetry is the standard for observability. The SDK is the language-specific library; the Collector is the central pipeline. The auto-instrumentation is the zero-code integration. This lesson walks the SDK, the Collector, the auto-instrumentation, and the production patterns.

The OpenTelemetry architecture

The OpenTelemetry architecture:

flowchart LR
    A[Application] --> B[OTel SDK]
    B --> C[OTel Collector]
    C --> D[Jaeger]
    C --> E[Tempo]
    C --> F[Prometheus]
    C --> G[Loki]

The architecture is the pipeline.

The SDK

The SDK is the language-specific library:

import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace"
    "go.opentelemetry.io/otel/sdk/trace"
)

exporter, _ := otlptrace.New(ctx, otlptrace.WithInsecure())
provider := trace.NewTracerProvider(
    trace.WithBatcher(exporter),
)

otel.SetTracerProvider(provider)

The SDK is the API for the application.

The auto-instrumentation

The auto-instrumentation:

# Java
java -javaagent:opentelemetry-javaagent.jar -jar my-app.jar

# Python
opentelemetry-instrument python my-app.py

# Node.js
node --require @opentelemetry/auto-instrumentations-node my-app.js

The auto-instrumentation is the zero-code integration.

The OTel Collector

The OTel Collector:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 10s
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80

exporters:
  jaeger:
    endpoint: jaeger:14250
    tls:
      insecure: true
  prometheus:
    endpoint: 0.0.0.0:8889

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch, memory_limiter]
      exporters: [jaeger]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheus]

The Collector is the central pipeline.

The OTel Collector deployment

The OTel Collector deployment:

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm install otel-collector open-telemetry/opentelemetry-collector \
  --namespace monitoring \
  --values otel-collector-values.yaml

The Helm chart deploys the Collector.

The receivers

The receivers:

The receivers are the input of the Collector.

The processors

The processors:

The processors are the transformation.

The exporters

The exporters:

The exporters are the output of the Collector.

The OTel Collector metrics

The OTel Collector metrics:

otelcol_receiver_accepted_metric_points
otelcol_exporter_sent_metric_points
otelcol_processor_batch_batch_send_size
otelcol_exporter_queue_size

The metrics are the input for the alerts.

The OTel Collector in production

The OTel Collector in production:

flowchart LR
    A[Application 1] --> B[OTel Collector]
    C[Application 2] --> B
    D[Application 3] --> B
    B --> E[Jaeger]
    B --> F[Prometheus]
    B --> G[Loki]

The Collector is the central pipeline.

The cross-course references

• The Prometheus course covers the metrics exporter.
• The Loki course covers the logs exporter.
• The Jaeger course covers the traces exporter.

Quiz

Production discipline

• Use the OTel SDK. The standard.
• Use the auto-instrumentation. The zero-code integration.
• Deploy the OTel Collector. The central pipeline.
• Configure the receivers, processors, exporters. The pipeline.
• Monitor the OTel Collector metrics. The Prometheus exporter.
• Document the deployment. The Helm values, the configuration.

The OpenTelemetry is the observability standard. Operating it well is the SDK, the Collector, the auto-instrumentation, and the exporters.