The Three Telemetry Signals

The three telemetry signals are not equivalent. They capture different things, they cost different amounts, they answer different questions. A platform that understands the distinction produces telemetry that supports investigation. A platform that does not collects numbers and strings that look right on a dashboard and fail when the production failure arrives.

This lesson defines each signal precisely, then walks through the question patterns that map to each.

What a metric is

A metric is a numeric time series. Prometheus models it as:

metric_name{label=value, label2=value2}  [timestamp]  value

A single time series is the unique combination of (metric_name, {label=value, ...}). Every value seen by Prometheus for that series is one sample. Two examples:

node_cpu_seconds_total{cpu=0, mode=idle, instance=web01}  12345.6
http_requests_total{method=GET, route=/checkout, status=200}  4096

What metrics measure. Counts (http_requests_total), gauges (node_memory_MemFree_bytes), distributions (http_request_duration_seconds_bucket), and summaries. They are sampled at a point in time. A counter never decreases except across a process restart; a gauge is whatever the source reports.

What metrics are good at. Aggregations: average, sum, percentile. Long retention: a metric sample is ~3 bytes; a million samples / second is feasible on modest hardware. Alerting: a metric is a single time series and can be tested cheaply against a threshold. Detection of trends over hours and days.

What metrics are not. A metric does not tell you which specific request was slow, which user saw an error, or what the log line said. A “high p99 latency” alert does not include the trace of the offending request unless an exemplar was added.

What a log is

A log is a discrete record of an event. Loki models it as a stream of lines attached to a labelled stream:

stream: {app="checkout", env="prod", host="web01"}
  2026-08-13T03:01:42Z INFO request completed status=200 duration_ms=124 route=/checkout request_id=req-8c43
  2026-08-13T03:01:42Z ERROR payment_svc returned 503 duration_ms=4800 route=/checkout request_id=req-8c43

The labels identify the stream; the lines carry the event content. Structured fields (status=200, duration_ms=124) are machine-parseable. Unstructured lines are searchable by full-text.

What logs measure. Discrete events. State changes. Error messages. Stack traces. Audit records. Any time a piece of code wanted to say “this happened,” it should have logged it.

What logs are good at. Forensic reconstruction: “what was the process doing at 03:01:42?” Structured logs enable filtering by field. Unstructured logs require regex or full-text. Logs are the canonical signal for understanding a specific event.

What logs are not. Cheap at scale. A 50-line verbose stack trace is 50 KB of log shipped; a million of those a day is 50 GB. Logs are also not a substitute for metrics: aggregating log content (count of status=500 per minute) is log-derived metrics, which is a useful fallback but should not replace native metrics for high-frequency counters.

What a trace is

A trace is the recorded journey of one request through a distributed system. Tempo stores them:

Trace: 7c91a4b1...
  Span: 7c91a4b1...
    name: "checkout-handler"
    parent: -
    service.name: checkout
    duration: 5.2 s
  Span: 7c91a4b1...a3f2
    name: "get_cart"
    parent: 7c91a4b1...
    service.name: cart
    duration: 80 ms
  Span: 7c91a4b1...c8d1
    name: "POST /payments"
    parent: 7c91a4b1...
    service.name: payment-svc
    duration: 4.8 s
    error: true

Each span has a name, a service, a duration, a parent (except the root), and attributes (e.g. http.route, http.status_code). Spans are stitched together by trace ID. Errors are a span status. Exemplars are a link from a metric datapoint to a trace.

What traces measure. Where the time went for one request. Which dependency was slow. Where the error originated. Critical path: the sequence of spans that account for end-to-end latency.

What traces are not. Cheap. A single trace can be hundreds of spans; a high-traffic service can produce millions per second. Production tracing therefore requires sampling — most requests are not traced, only a fraction. Traces are also not a substitute for logs: a trace shows duration and attributes, not event content.

What each signal uniquely answers

The matrix below is the framework for choosing the right signal. Match the question to the row that contains it.

When each signal is the wrong tool

A common production mistake is using a signal because it was cheaper to instrument, not because it answered the right question.

• Using metrics for what happened. A counter is a number, not a story. If the question is “what was the SQL?”, the answer is not in metrics. It is in the application’s structured log.
• Using logs for aggregate over time. A log-derived metric can substitute, but native metrics are cheaper. If the question is “did error rate increase?”, instrument a counter, not a log search.
• Using traces for long retention. Spans are large and sampled. If the question is “was this metric elevated a week ago?”, the answer is in metrics, not traces.
• Using any one signal alone. A symptom is rarely diagnosed from one signal. The course returns to correlation in Part LI.

What happens during a production failure

Three signals at the moment of failure:

• Metrics show aggregate symptoms (p99 latency, error rate).
• Logs show discrete events, often correlated by request_id.
• Traces show the call tree for a representative failing request.

Each signal independently would still leave the investigation incomplete. Together — and correlated — they cover the investigation tree. Part LI builds correlation workflows.

Verification

You should be able to answer:

• What is the unit of storage for each signal (a time series, a stream of lines, a set of spans)?
• Which signal is the right answer to “which dependency was slow?”
• Which signal is the right answer to “did error rate increase over the past hour?”
• Why does instrumenting every signal the same way miss the design?