Signals

A checkout service emits 200 metrics, 50 spans per second, and 2,000 log lines per second. Grafana shows three dashboards, all green. The on-call engineer opens one to investigate a slow checkout and finds no way to move from the metric panel to the spans, from the spans to the logs, from the logs to the host metrics. The signals are present; the signals are not correlated. The investigation takes forty-five minutes that should have taken four.

OpenTelemetry solves this at the data-model level. The four signal types share a resource model, share a context propagation mechanism, and conform to a published set of semantic conventions. The result is that a single trace ID stamped by the service can pivot from metric to log to trace in Grafana without a manual correlation step. The lesson that follows names each signal, names the resource that joins them, and walks the configuration that gets correlation right.

What it is

A signal is a category of telemetry the OTel specification defines. There are four.

Each signal has a data model in the OTel specification. The data model is the contract; the SDKs produce records that match the contract; the Collector and the backends consume records that match the contract. Records that do not match the contract are either rejected at the wire boundary or coerced by the SDK into something lossy.

Metrics

A metric is a numeric value sampled at a point in time. The OTel specification defines six instrument kinds; the four in production use today are:

• Counter — monotonically increasing value. Request count, error count, bytes sent. Reset only on process restart.
• UpDownCounter — value that goes up and down. Active connections, queue depth, in-flight requests.
• Gauge — point-in-time value that may go up or down. CPU usage, memory usage, temperature.
• Histogram — distribution of values across buckets. Latency, request size. Histograms are aggregated across processes by the backend (sum, count, quantiles).

Two more instruments exist for exponential histograms and gauge histograms. They are useful for high-cardinality latency work but the runtime support is uneven.

Logs

A log is a discrete event. The OTel data model carries the timestamp, severity, body, attributes, and the resource. The body is the human-readable text; the attributes are the structured fields. A log record may also carry a trace_id and span_id, in which case it is bound to a trace and is queryable from the trace view in Grafana.

The OTel specification intentionally does not redefine log formats. A JSON line, a syslog line, a free-form string — all are valid bodies. The specification normalises the envelope, not the content.

Traces

A trace is the journey of a single request through a distributed system. A trace is a tree of spans. Each span has a name, a start time, a duration, attributes, and a parent. The root span is the entry point; child spans are the downstream operations.

Spans carry the trace context — a trace_id and a span_id — and the context propagates from caller to callee at the wire boundary. A span emitted by service A that calls service B carries the same trace_id; service B extracts the context from the inbound request, creates a child span, and emits the new span on its own OTLP connection. The chain is the trace.

Profiles

A profile is a sampled view of a running process. The OTel profile signal carries CPU samples and allocation samples, indexed by stack trace and timestamp. The granularity is the profile period (default 10 milliseconds for CPU); the storage shape is a flame-graph-compatible aggregation.

The runtime support for profiles is uneven. The Go SDK and the Java SDK (with a JFR collector) ship production-grade profile export. Python, Node.js, Ruby, PHP, and the .NET SDKs have experimental or partial support. Treat profile export as opt-in per language.

Why a sysadmin cares

Three failure shapes appear when the four signals are emitted independently without a shared resource or context model.

1. The unjoined signals. A metric says the checkout latency p99 is 4.2 seconds. A log says payment timed out. A span says payment-svc returned 503. None of them carries the trace ID of the specific slow request. The engineer reads three dashboards and joins them by hand, in their head, at 03:00.
2. The unowned service. A backend ingests metrics labelled service.name=otelcol because the resource detector failed and the Collector fell back to its binary name as service.name. Every fleet’s metrics live in the same service. Dashboards return nothing useful.
3. The semantic vocabulary drift. Service A records the HTTP method on attribute http.method. Service B records it on http.request.method. A dashboard that groups by the first attribute sees half the traffic; a dashboard that groups by the second sees the other half.

OpenTelemetry solves all three at the data-model level. Every record carries a Resource describing the entity that emitted it. Every record optionally carries the W3C Trace Context. Every record’s attributes follow the published semantic conventions (http.request.method, not http_method). The downstream backends inherit the model.

How it works

The mental model. A single service emits four signals. Every record carries the same Resource. A trace ID stamps every record that participates in a request. Semantic conventions give every attribute a known name.

    +---------------------------------------------+
    |              Service checkout               |
    |  Resource:                                  |
    |    service.name = checkout                  |
    |    service.version = 1.42.0                 |
    |    deployment.environment = prod            |
    |    host.name = checkout-7d-b8b9d6c8-x2k7q   |
    |    k8s.pod.uid = 7c1...                     |
    +---------------------+-----------------------+
                          |
        +-----------------+----------------+----------------+
        |                 |                |                |
    metrics            traces             logs           profiles
    (Counter,         (Span tree,        (LogRecord,    (CPU samples,
     Histogram)         trace_id)          trace_id)     allocations)
        |                 |                |                |
        +---- OTLP frames carrying the same Resource -------+
                          |
                          v
                  +------------------+
                  |     Collector    |
                  +------------------+
                          |
        +-----------------+----------------+----------------+
        |                 |                |                |
     Mimir/Prom       Tempo              Loki           Profile
                                                        backend

The Resource is the join key. The trace_id is the per-request join key. The attribute keys are the join vocabulary across services.

The Resource model

The Resource is a set of attributes that describe the entity producing the telemetry — the service, the host, the pod, the deployment. The Resource is attached to every signal type. The Collector uses the Resource to route, deduplicate, and stamp the destination backend.

The OTel specification defines a canonical set of Resource attributes:

• service.name — the name of the service as the operator sees it. Required. There is no default.
• service.version — the build version.
• deployment.environment — the deployment environment (prod, staging, dev).
• Host attributes (host.name, host.arch) — filled in by the system resource detector.
• Process attributes (process.pid, process.executable.path) — filled in by the SDK.
• Container / k8s attributes (k8s.pod.name, k8s.namespace.name) — filled in by the k8s resource detector when running in cluster.

A Resource is set once at SDK initialisation and attached to every record. The Collector can add or override Resource attributes via the resource processor.

Context propagation

The W3C Trace Context specification defines two headers: traceparent and tracestate. The OTel SDK stamps traceparent on every outbound HTTP and gRPC call; the SDK extracts traceparent on every inbound call and uses it to build a parent / child relationship.

The propagation is what turns N independent spans into one trace. The propagator must be configured on every service in the request path; one service without propagation breaks the trace across its boundary. The TraceQL course material in tempo covers this in detail; the present lesson names it because correlation is the reason the resource model exists.

Semantic conventions

Semantic conventions are the published attribute vocabulary. HTTP, database, RPC, messaging, and runtime frameworks all have a convention. The convention names the attribute key (http.request.method), the value type (string), and the allowed values (GET, POST, …).

The convention is versioned. The 1.x line is stable; the 2.x line is the current draft. Mixing the two in the same fleet is a known source of dashboard inconsistency. The OTel Collector can migrate one version to another via the semconv processor, but the disciplined answer is to pick one version and stick to it.

How to configure it

A Collector pipeline per signal type with the resourcedetection processor on each. The processors order matters: memory_limiter first, then resourcedetection, then batch. The exporters are per-signal.

# /etc/otelcol/config.yaml

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25
  resourcedetection:
    detectors: [env, system, k8s]
    timeout: 2s
    override: true
  batch:
    timeout: 5s
    send_batch_size: 8192

exporters:
  otlp/mimir:
    endpoint: mimir.internal.example.com:4317
    tls:
      insecure: false
      ca_file: /etc/otelcol/ca.pem
  otlp/tempo:
    endpoint: tempo.internal.example.com:4317
    tls:
      insecure: false
      ca_file: /etc/otelcol/ca.pem
  otlp/loki:
    endpoint: https://loki.internal.example.com/otlp
    headers:
      X-Scope-OrgID: prod

service:
  telemetry:
    metrics:
      address: localhost:8888
    logs:
      level: info
  pipelines:
    metrics:
      receivers:  [otlp]
      processors: [memory_limiter, resourcedetection, batch]
      exporters:  [otlp/mimir]
    traces:
      receivers:  [otlp]
      processors: [memory_limiter, resourcedetection, batch]
      exporters:  [otlp/tempo]
    logs:
      receivers:  [otlp]
      processors: [memory_limiter, resourcedetection, batch]
      exporters:  [otlp/loki]

The override: true on resourcedetection lets the container-aware detector replace any host.name the SDK already set. Without it, two detectors race to set the same attribute and the result depends on detector order.

How to validate it

Validate that each signal is being received, processed, and exported, and that the Resource attributes are present.

# CONFIGURATION: parse-check.
otelcol validate --config=/etc/otelcol/config.yaml

# READ-ONLY: confirm each signal type is accepted.
curl -s http://localhost:8888/metrics | grep otelcol_receiver_accepted

otelcol_receiver_accepted_metric_points{receiver="otlp",signal="metrics"} 4096
otelcol_receiver_accepted_spans{receiver="otlp",signal="traces"} 1024
otelcol_receiver_accepted_log_records{receiver="otlp",signal="logs"} 8128

# READ-ONLY: confirm each pipeline is exporting.
curl -s http://localhost:8888/metrics | grep otelcol_exporter_sent

otelcol_exporter_sent_metric_points{exporter="otlp/mimir"} 4096
otelcol_exporter_sent_spans{exporter="otlp/tempo"} 1024
otelcol_exporter_sent_log_records{exporter="otlp/loki"} 8128

# READ-ONLY: confirm Resource attributes are set on the records.
# (This query runs against the destination Tempo.)
traceql '{ resource.service.name = "checkout" }' | head -5

{ resource.service.name = "checkout", resource.deployment.environment = "prod",
  span.http.request.method = "POST", span.http.route = "/api/v1/checkout" }

The semantic-convention attribute names (http.request.method, http.route) appear with the span. prefix that Tempo applies. A missing service.name on every record is the first sign the SDK was initialised without a resource.

How it can fail

Five failure modes that arise when the signals are emitted without the resource model that joins them.

1. The metric with no service.name. The SDK was initialised before the resource was set. Every metric has service.name=unknown_service. Symptom: Mimir’s OTLP receiver rejects the batch with missing required attribute service.name; the metric counter on the receiver never climbs.
2. The trace context that stops at the boundary. Service A propagates the W3C traceparent; service B’s HTTP client uses a third-party library that strips the header. Symptom: spans are emitted but every trace has exactly one span. The investigation is “where did the call go” and the answer is “nowhere I can see”.
3. The log record with the wrong severity. A library records level=ERROR on a routine warning; the alerts flood; the operator learns to ignore level=ERROR. The severity scale (TRACE, DEBUG, INFO, WARN, ERROR, FATAL) is the contract; non-standard values break dashboards that filter on severity.
4. The cardinality blow-up. A metric instrumented against http.request.method and user.id per request. The label set multiplies by the number of users. Mimir rejects the batch with cardinality limit exceeded. Symptom: the metric counter is non-zero on the receiver but the backend rejects every batch.
5. The semantic-convention version drift. Service A uses semconv 1.x (http.method); service B uses semconv 2.x (http.request.method). The dashboard that groups by http.method sees only service A; the one that groups by http.request.method sees only service B. Symptom: every dashboard sums to roughly half of the expected total.

How to troubleshoot it

When the four signals do not correlate, the diagnostic order is from the model down.

1. Is service.name set? otelcol_receiver_accepted_* with a label match on service.name. Empty match means the resource was not set before the SDK emitted its first record. Fix at SDK initialisation.
2. Does the trace ID survive the service boundary? traceql for a known trace_id after a synthetic end-to-end test. If the trace contains a single span, the propagator was not installed on the client side of one of the services. The fix is to enable the W3C TraceContext propagator on every HTTP/gRPC client.
3. Are the attribute names the convention? otelcol processor.transform with a debug exporter on a single pipeline, dump the records, check the attribute names. http_method is a tell; http.request.method is the convention.
4. Is the cardinality bounded? mimir analyze for the active series per metric. A metric with more than 100k active series is suspect; more than 1M is the failure shape.
5. Are the semantic-convention versions aligned? otelcol_semconv_version (when exposed) or a manual inspection of the SDK source. The fix is to align the SDK versions across the fleet, or to add the semconv processor to migrate one version to another.

Security implications

The data model carries attributes that may be sensitive.

• Resource attributes. service.name, service.version, host.name, k8s.pod.name are all metadata that names internal hosts and services. A trace export to a third-party backend exposes the topology. The operator decides which attributes are exportable.
• Span attributes. HTTP path, query string, request body excerpts, database statement, exception message — common span attributes that frequently leak credentials or PII. The redaction layer lives in the Collector (attributes or transform processor), not in the SDK.
• Log body. A free-form string that may contain secrets. The discipline is to never log secrets; the realistic answer is to redact in the pipeline.
• Profiles. Stack frames often contain class names, file paths, and inline function arguments in interpreted languages. A profile export to a third party exposes the codebase shape. Treat profile export as a higher-sensitivity signal than metrics.

Performance implications

The data model and the pipelines are the performance surface.

• Resource detection cost. k8s and system detectors talk to the kubelet and the host. The cost is paid once per pipeline start, not per record. Cache the result.
• Attribute cardinality. Every distinct attribute value multiplies the active series count. A metric with five attributes each with 100 values has 10^10 possible combinations — Mimir will reject most of them. The discipline is to keep attribute cardinality bounded.
• Span volume. Every span emits per request. A high-traffic service can emit millions of spans per minute; the cost is on the wire, on the Collector, and on the backend. Sampling (head-based or tail-based) is the mitigation. Lesson 05 covers tail sampling in the Collector.
• Profile volume. A CPU profile sampled every 10 ms is a constant stream. The cost is per-process. Profile export is not for every service; profile for the services the operator is investigating.

Production guidance

• Set service.name at SDK initialisation. A missing service.name is the highest-cost silent failure in OTel deployments.
• Pick a semantic-convention version and stick to it. The 1.x line is stable; the 2.x line is the draft. Mixing is the failure shape.
• Bound attribute cardinality. Cap attribute cardinality per metric in the SDK or in the Collector. A cardinality budget is a real thing; lesson 04-cardinality-budget covers the discipline.
• Sample traces by policy, not by default. Head sampling with a sensible rate; tail sampling on the gateway for high-value traces.
• Verify the resource attributes on every release. A new SDK version can change the default resource detector. The smoke test should include a synthetic trace with a known service.name.

Verification

You should now be able to answer:

• Which question does each of the four signal types uniquely answer?
• What is the Resource model and what attributes does it carry?
• Why is context propagation the mechanism that turns spans into traces?
• What is the failure shape when service.name is missing on every record?
• How do semantic conventions prevent dashboard inconsistency across services?

Quiz