OpenTelemetry Overview

A team adopts Prometheus in 2020, Jaeger in 2021, a vendor-hosted log service in 2022, and a third-party tracing product in 2023. By 2025 the application code carries four different instrumentation libraries and four different exporters. The vendor’s tracing SDK is end-of-life’d. The team faces a six-month migration to replace one library with another, across 200 services, in two languages, while keeping dashboards green. A single vendor-neutral framework would have avoided the migration entirely.

OpenTelemetry is that framework. It is a CNCF incubating project that defines a vendor-neutral way to generate, transmit, and receive telemetry. It is not a backend, not a vendor product, not a storage system. It is a specification plus the implementations of that specification. The specification is the contract; the SDKs and the Collector are the implementations.

What it is

OpenTelemetry is three artefacts published under one name:

1. The specification. A set of documents that define telemetry data types, the wire protocol, semantic conventions, and the expected behaviour of SDKs. The specification is the source of truth. Backends and SDKs conform to it; the specification does not conform to them.
2. The SDKs. Per-language libraries that implement the specification. Java, Python, Go, .NET, Node.js, Ruby, PHP, Rust, C++, and Swift each have an OTel SDK. The SDKs instrument application code, attach correlation metadata, batch telemetry, and export it.
3. The Collector. A vendor-neutral telemetry pipeline published by the project. A single binary that receives telemetry from any source, transforms and batches it, and exports it to any backend. The Collector is optional — SDKs can export directly — but is the recommended pattern in production.

These three artefacts solve the same problem from three angles: the specification prevents vendor lock-in at the wire level, the SDKs prevent lock-in at the application level, and the Collector prevents lock-in at the operations level.

The four signals

The specification defines four signal types. The first three have been stable for several releases; the fourth graduated more recently.

• Metrics — numeric values sampled over time. Counters, gauges, histograms. Cheap to store, good for alerting and dashboards. A single counter tells you how many or how long but does not identify the specific request.
• Logs — discrete events with structured or unstructured content. Captured per-service. Good for what happened, weak on which request unless the application stamps a trace ID.
• Traces — the journey of a single request through a distributed system, modelled as parent/child spans. Good for where did the latency come from and which dependency failed.
• Profiles — a 2024-2025 addition. Continuous runtime profiling of CPU and memory allocation, sampled per process. Good for which function is consuming the heap.

The four signals are not interchangeable. Each answers a question the others answer only with great effort. The course covers metrics, logs, and traces in production detail; profiles are covered at the boundary because the runtime support is still uneven across languages.

Why a sysadmin cares

Three failure shapes appear when a fleet’s telemetry is split across multiple vendor-specific pipelines.

1. The vendor exit. A vendor’s product is acquired, de-listed, or sunset. The team’s instrumentation now requires migration. A vendor-neutral layer (OTel) is portable; a vendor SDK is not.
2. The dashboard drift. A team runs three backends, each with its own dashboard grammar, its own retention knobs, its own query language. The on-call engineer pays the cognitive cost every incident.
3. The integration tax. Each new service speaks a new protocol. Prometheus must learn how to scrape it, the log pipeline must be re-taught, the tracing backend must accept a new SDK. The cost compounds with the service count.

OpenTelemetry pays back at fleet scale. A single OTel Collector can receive from every instrumented service in the fleet, regardless of the language, and route the four signals to four different backends (or to the same one). The backends become swappable; the SDKs become swappable; the on-call engineer deals with one telemetry vocabulary.

How it works

The mental model. A single application emits the four signals through a single vendor-neutral protocol. The Collector sits in the middle, fan-in from many services, fan-out to many backends.

   +-------------+   +-------------+   +-------------+
   | Service A   |   | Service B   |   | Service C   |
   | (Go)        |   | (Python)    |   | (Java)      |
   +------+------+   +------+------+   +------+------+
          |                |                |
          | OTLP over gRPC or HTTP/protobuf |
          +----------------+----------------+
                           |
                     +-----v-----+
                     | Collector |
                     | (gateway) |
                     +-----+-----+
                           |
        +------------------+------------------+
        |                  |                  |
   +----v----+       +-----v-----+      +-----v-----+
   | Mimir   |       | Loki      |      | Tempo     |
   | metrics |       | logs      |      | traces    |
   +---------+       +-----------+      +-----------+

The specification, in detail

The specification lives at opentelemetry.io/docs/specs/. It is organised by signal and by language. Each signal has:

• A data model. The shapes of the telemetry records — what a metric, a log, a span, a profile actually contains.
• A wire protocol. OTLP, defined as protobuf messages over either gRPC or HTTP.
• Semantic conventions. A vocabulary for the attribute keys and values that describe the entity producing the telemetry (the service.name, the HTTP method, the database statement). Semantic conventions are versioned and evolve independently of the wire protocol.

Each SDK has:

• An API. The surface the application calls into. Creating a span, recording a value, attaching an attribute. The API is no-op when no SDK is configured — that is the contract.
• An SDK. The implementation that registers the API calls, samples them, batches them, and exports them. The SDK is the part the operator configures; the API is the part the application calls.
• Instrumentation libraries. Reusable wrappers that record spans for well-known frameworks — HTTP servers, database drivers, RPC frameworks. The instrumentation libraries are the bulk of the value; the application rarely calls the API directly.

The split is deliberate. The API stays stable. The SDK can be swapped, the exporter can be swapped, the instrumentation libraries can be upgraded independently. The application code is the same before and after.

The Collector

The Collector is a Go binary that embeds component factories. Each receiver, processor, exporter, and extension is a Go type that implements a known interface. The YAML configuration names the factories; the runtime builds the components; the pipeline wires them.

The Collector is published as two distributions: otelcol (core, smaller, fewer components) and otelcol-contrib (everything in core plus the community components). Most production deployments use contrib because most production receivers and exporters (filelog, journald, loki, prometheus) live in contrib.

The Collector is the recommended fan-in point for two reasons. First, the application SDK can export to one endpoint (otelcol:4317) regardless of the destination backend, and the operator changes destinations in the Collector config without touching the application. Second, the Collector provides a single queue, single memory limiter, single batch processor per pipeline — properties that are awkward to coordinate across N applications.

How to configure it

A minimum viable Collector that receives all four signals and exports the first three to a Loki/Tempo/Mimir back-end, with the fourth (profiles) accepted but exported nowhere yet. Real config, annotated:

# /etc/otelcol/config.yaml

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  # memory_limiter MUST come before batch in every pipeline.
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25
  batch:
    timeout: 5s
    send_batch_size: 8192
  # resource detection stamps k8s pod info onto every record.
  resourcedetection:
    detectors: [env, system, k8s]
    timeout: 2s

exporters:
  otlp/mimir:
    endpoint: mimir.internal.example.com:4317
    tls:
      insecure: false
      ca_file: /etc/otelcol/ca.pem
  otlp/loki:
    # The Loki exporter speaks OTLP-to-Loki via the gateway tenant.
    endpoint: https://loki.internal.example.com/otlp
    headers:
      X-Scope-OrgID: prod
  otlp/tempo:
    endpoint: tempo.internal.example.com:4317
    tls:
      insecure: false
      ca_file: /etc/otelcol/ca.pem

service:
  telemetry:
    metrics:
      address: localhost:8888
    logs:
      level: info
  pipelines:
    metrics:
      receivers:  [otlp]
      processors: [memory_limiter, resourcedetection, batch]
      exporters:  [otlp/mimir]
    traces:
      receivers:  [otlp]
      processors: [memory_limiter, resourcedetection, batch]
      exporters:  [otlp/tempo]
    logs:
      receivers:  [otlp]
      processors: [memory_limiter, resourcedetection, batch]
      exporters:  [otlp/loki]
    profiles:
      receivers:  [otlp]
      processors: [memory_limiter, resourcedetection, batch]
      exporters:  []

Three pipelines, one exporter each. The profiles pipeline declares a receiver but no exporters — the application emits profiles, the Collector accepts them, and the queue drains to nothing until a profiles backend is wired. Removing the unused pipeline is wrong: the application will fail to export when the SDK expects a receiver for every signal.

How to validate it

Validate the config and the pipeline from local to remote.

# CONFIGURATION: parse-check.
otelcol validate --config=/etc/otelcol/config.yaml

# (no output on success; non-zero exit on error)

# READ-ONLY: list every component in the running binary.
otelcol components

receivers:
  - otlp
  - filelog
  - journald
  - prometheus
processors:
  - batch
  - memory_limiter
  - resourcedetection
exporters:
  - otlp
  - otlphttp
  - debug
  - loki
extensions:
  - health_check
  - pprof
  - zpages
  - bearertokenauth

# READ-IMPACT: apply the config.
systemctl reload otelcol

# READ-ONLY: confirm the receiver accepted data on every signal.
curl -s http://localhost:8888/metrics | grep otelcol_receiver_accepted

otelcol_receiver_accepted_metric_points{receiver="otlp",signal="metrics"} 4096
otelcol_receiver_accepted_spans{receiver="otlp",signal="traces"} 1024
otelcol_receiver_accepted_log_records{receiver="otlp",signal="logs"} 8128

Each signal type has its own counter. A missing line for a signal the application emits is the first sign the SDK is not configured for that signal — see lesson 06.

How it can fail

Five failure modes specific to OpenTelemetry deployments.

1. The distribution mismatch. The YAML names filelog, the binary is otelcol (core, no contrib). The collector refuses to start with component "filelog" not found in the binary. The fix is to switch the binary to otelcol-contrib, not to remove the component.
2. The signal without a pipeline. An application emits metrics, traces, and logs. The Collector config declares pipelines for traces and logs only. Symptom: the SDK logs failed to export metrics: no pipeline for signal; the metrics counter on the receiver never climbs.
3. The exporter without the tenant header. The loki exporter ships to the default tenant. Symptom: lines appear in Loki but in the wrong tenant; dashboards return nothing for the production tenant.
4. The OTLP port blocked. An egress firewall only allows port 443. The SDK exports to otelcol:4317 (gRPC) which is reachable. The Collector exports to tempo.internal:4317 which is not. Symptom: receiver counters climb, exporter counters do not, and the destination backend is empty.
5. The resource detector that did not detect. The resourcedetection processor is configured with detectors: [k8s] but the Collector runs as a process on a VM with no Kubernetes API. The detector fails silently; the service.namespace attribute is missing; the service.name falls back to the binary name (otelcol). Symptom: every backend sees one giant service called otelcol containing every fleet’s spans.

How to troubleshoot it

When the backends receive no data, the diagnostic order matters.

1. Is the Collector running? systemctl status otelcol and the agent log for the start banner. The first line of the log names the version and the loaded config path.
2. Is the config valid? otelcol validate. A parse error names the line and the argument. The first error is usually the only one.
3. Are the components in the binary? otelcol components for the names declared in the YAML. A missing component means a distribution mismatch.
4. Are the receivers accepting? otelcol_receiver_accepted_* for the signal type the application emits. Zero means the SDK is not exporting, or the network is blocked, or the port is wrong.
5. Are the exporters sending? otelcol_exporter_sent_* for the signal and the exporter name. Zero with non-zero receivers means a pipeline failure.
6. Is the backend reachable from the Collector host? curl -v https://tempo.internal:4317 (no path; gRPC responds with HTTP/2 + 415 to a plaintext GET, which is the expected answer). A connection refused means DNS or firewall.

Security implications

OpenTelemetry exposes the same surfaces as the rest of the observability stack, with OTel-specific names.

• The OTLP listener. 0.0.0.0:4317 (gRPC) and 0.0.0.0:4318 (HTTP) by default. In agent mode, bind to localhost or to a private interface. In gateway mode, restrict the listener with a NetworkPolicy or with an ingress controller that requires mTLS.
• The Collector metrics and debug endpoints. Defaults are localhost:8888 for /metrics, localhost:8889 for /debug, and localhost:13133 for the health check. All bind to localhost. Expose them on the cluster network only with authentication.
• TLS to the exporters. The otlp exporter accepts a tls block for CA bundles, client certificates, and insecure_skip_verify (development only). A stale CA bundle is the most common cause of silent shipping failure.
• Semantic-attribute leakage. Semantic conventions are public vocabulary; HTTP paths, database statements, and exception messages are common attributes. An application that records db.statement against the production database leaks query text into traces. The processor pipeline (see lesson 04-otel-collector-config) is where redaction lives.

Performance implications

The Collector is in the hot path on every host (agent mode) or behind the hot path (gateway mode). Three knobs dominate.

• Batching. The batch processor coalesces entries to reduce per-call cost. send_batch_size is the upper bound per request; timeout is the upper bound on latency. A larger batch with a longer timeout trades latency for throughput. The default timeout: 200ms is too aggressive for a gateway; five seconds is a more realistic starting point.
• Memory limiter. The memory_limiter processor refuses data when the process approaches a memory limit. It must be the first processor in the chain. A batch placed before memory_limiter lets the batch grow unbounded; the limiter never sees the queue.
• Disk queue. The file_storage extension backs the sending_queue on exporters. A disk queue lets the gateway survive a downstream outage without rejecting batches. The trade-off is durability against disk I/O.

CPU is rarely the bottleneck. RAM scales with the queue depth, which scales with the line rate and the network latency to the backend. Disk I/O spikes during buffer flushes; an SSD-backed host disk is appropriate.

Production guidance

• Use otelcol-contrib unless the deployment only needs core. The component set in contrib matches the production needs of almost every fleet.
• Place memory_limiter first in every pipeline. The chain is memory_limiter, batch, then the rest.
• Pin OTel SDK versions in the application manifest. The SDKs release on a monthly cadence; the spec evolves more slowly. Read the per-language release notes before bumping.
• Smoke test after every config change. Ship a known log line, a known metric increment, a known trace, and find each one in its destination within ten seconds.
• Document the rollback. The reload is kill -HUP $(pidof otelcol) or systemctl reload otelcol. The rollback is the same command with the previous config restored.

Verification

You should now be able to answer:

• What three artefacts does OpenTelemetry publish, and what is the role of each?
• Which of the four signals is the most recent to graduate, and what does it answer?
• What is the difference between the OTel API and the OTel SDK?
• Why is the Collector positioned as the recommended fan-in point?
• Where does the authoritative specification live, and where is its stability status declared?

Quiz