Three Signals as Investigative Tools

A primary on-call engineer opens a P1: checkout p99 latency at 4.2 seconds, above the 800 ms SLO threshold. They have one Grafana instance and three data sources: Prometheus, Loki, Tempo. The dashboard opens fast. The decision of where to click first is the entire investigation.

This lesson is about that decision. The three signals are not interchangeable; each is a tool with a specific job in the loop. Used correctly the loop moves from phase 1 to phase 6 in fifteen minutes. Used incorrectly the loop either stalls on a metric panel that has no further information, or runs queries across gigabytes of logs that the metric pointed at but did not explain.

What each signal is a tool for

The three signals answer different classes of question. The loop assigns each class of question to a specific signal.

   +-------------------+
   | Phase 1 symptom   |  ---->  Metrics  (alert panel: which metric, which label, since when)
   +-------------------+
   |
   v
   +-------------------+
   | Phase 2 impact    |  ---->  Metrics  (error budget burn, business counter, SLO panel)
   +-------------------+
   |
   v
   +-------------------+
   | Phase 3 hypothesis|---->  Change log + dependency map  (operational history, not signals)
   +-------------------+
   |
   v
   +-------------------+
   | Phase 4 evidence  |  ---->  Trace of one failing request  (smoking gun: which span, which dep)
   |                   |        OR
   |                   |  ---->  Logs filtered by trace_id or service (what did this request do)
   +-------------------+
   |
   v
   +-------------------+
   | Phase 5 test      |  ---->  Cross-signal: metric AND trace AND log all agree
   +-------------------+
   |
   v
   +-------------------+
   | Phase 6 document  |  ---->  Runbook entry with links to each signal at each phase
   +-------------------+

A rule of thumb for the pivot points between signals:

• Metric to trace. A metric confirms “the system is slow on the checkout route”. The trace of one failing checkout confirms “the slow time is in the payment-svc span, not the cart-svc span”. The pivot needs a correlation ID: exemplars on the histogram bucket or trace IDs in the logs.
• Trace to log. A trace of one request confirms “this dependency call waited 4 seconds”. The logs for that dependency in the trace’s time window confirm “the dependency returned 503 after 4 seconds; the upstream gave up retrying”. The pivot needs the trace ID in the log stream.
• Log to metric. A log error spike at 02:47 confirms “the payment-svc error message appeared 200 times in 5 minutes”. A query against the log-derived metric (count_over_time({job="payment-svc"} |= "503" [5m])) confirms the spike aligned with the symptom window.

Why a sysadmin cares

Three concrete operational pains disappear when signals are used as tools inside the loop rather than as a wall of dashboards.

1. The dashboard of green panels. A common failure shape is a team that builds panels for every metric they can think of and uses none of them during an incident. Assigning a panel to each phase of the loop turns the dashboard into a navigation map.
2. The log-spike trap. A P0 with a log-rate alert pulls the on-call engineer to the log explorer before they have quantified impact. The log shows the symptom; it does not show impact. The methodology pulls the engineer to the metric first; logs come at phase 4.
3. The trace-without-context trap. A senior engineer opens a trace and identifies a slow dependency. Without the metric, the trace cannot quantify impact (“how many users are affected?”) and cannot show baseline (“is this normal?”). The trace is a tool at phase 4, not at phase 1 or 2.

How to configure it

The three signals are configured into one loop by:

1. Putting trace IDs into logs (and span IDs where the log line is per-span rather than per-request).
2. Putting exemplars onto latency histograms at the SDK or auto-instrumentation layer.
3. Linking dashboards to the data source each phase uses, so the on-call engineer navigates the loop by clicking.

A trimmed OpenTelemetry Collector pipeline that keeps the trace ID in every log line:

# /etc/otelcol/config.yaml
receivers:
  otlp:
    protocols:
      grpc: {}
      http: {}

processors:
  batch: {}
  resource:
    attributes:
      - key: deployment.environment
        value: production
        action: insert

exporters:
  prometheusremotewrite:
    endpoint: 'http://prometheus:9090/api/v1/write'
  loki:
    endpoint: 'http://loki:3100/loki/api/v1/push'
  otlp/tempo:
    endpoint: 'tempo:4317'
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch, resource]
      exporters: [otlp/tempo]
    metrics:
      receivers: [otlp]
      processors: [batch, resource]
      exporters: [prometheusremotewrite]
    logs:
      receivers: [otlp]
      processors: [batch, resource]
      exporters: [loki]

A Loki log line without the trace ID is searchable but uncorrelatable. The fix is at the SDK layer; on Java this is done by the Logback appender including the MDC entry, on Go by the slog handler carrying the trace context, and in Grafana Alloy via the loki.process stage with a traceID parser.

// /etc/alloy/config.alloy
loki.process "checkout" {
  stage.json {
    expressions = ["trace_id", "span_id", "level", "msg"]
  }
  stage.labels {
    values = {
      "trace_id" = "",
      "level"    = "",
    }
  }
}

The trace_id label is the correlation. After processing, the same log line can be queried alongside the matching trace in Tempo:

{service="checkout"} | json | trace_id="a3f2c1..."

How to validate it

Validate the loop by reproducing a Phase 1-5 walk on the same incident:

# SEVERITY: READ-ONLY
# 1. Confirm exemplars are emitted on the latency histogram.
curl -s 'http://prometheus:9090/api/v1/query?query=http_request_duration_seconds_bucket%7Bjob%3D%22checkout%22%7D' \
  | jq '.data.result[0].metric' \
  | head

Expected output:

{
  "job": "checkout",
  "le": "0.8",
  "route": "/checkout"
}

The presence of a route label and the bucket boundary confirm the histogram is shaped for exemplars.

# SEVERITY: READ-ONLY
# 2. Confirm trace IDs reach Loki.
logcli query --addr=http://loki:3100 \
  '{service="checkout"} |= "checkout p99" | json | trace_id=~".+"' \
  --since=15m --limit=1

Expected output:

2026-08-13T03:14:22Z {service="checkout"}  latency 4120 trace_id=a3f2c1d4e5f6 span_id=b1c2

# SEVERITY: READ-ONLY
# 3. Confirm the trace exists in Tempo with the same trace ID.
tracecli lookup --addr=http://tempo:3200 a3f2c1d4e5f6

Expected output:

TraceID: a3f2c1d4e5f6
  Service         Span                  Duration   Start
  checkout        POST /checkout        4120 ms    2026-08-13T03:14:18Z
  payment-svc     POST /charge          4060 ms    2026-08-13T03:14:18Z

The three checks together confirm that:

• The metric panel can show “which latency, which histogram, which bucket”.
• The trace ID is present in logs that reference the same request.
• The trace contains the slow dependency span.

The on-call engineer moves through the loop without copying IDs by hand.

How it can fail

Six failure shapes occur when signals are used as walls of charts rather than tools in a loop.

1. The dashboard-of-everything trap. The engineer opens every panel in sequence, looking for the smoking gun without a hypothesis. Each panel either confirms the last guess or is set aside; phase 5 is never reached. Symptom: the runbook entry is a stack of panel screenshots and no falsifier.
2. The log-spike trap (signal misordered). A log-rate alert pages the engineer; the engineer opens Loki before quantifying impact. The log shows the message but not how many users are affected. Phase 2 is skipped. Symptom: the runbook entry names a log pattern but not a number.
3. The trace-without-metric trap. A senior engineer opens a trace and identifies a slow dependency. Without the metric, the trace cannot quantify impact and cannot validate baseline. Phase 2 is missing. Symptom: the runbook entry names a span but not a percentage.
4. Cross-signal pivot missing. The metric shows latency on checkout. The trace confirms the slow span. The log shows an error message in the same window. None of the three carry the same identifier; the engineer correlates by hand and gets the wrong request. Symptom: the runbook entry names a span and a log line that came from different requests.
5. Single-signal tunnel vision. The on-call engineer follows one signal (usually the metric the page cited) and never opens the other two. The cause turns out to be visible only in a trace or only in a log; the engineer misses it. Symptom: the runbook entry cites only one signal; another engineer can complete the investigation in five minutes by reading the missing signal.
6. Trace sampled out. A tail-sampling decision dropped the failing trace at the collector because the error rate at the trace layer was below the keep-rate. The engineer has a metric anomaly and a log spike and no trace. Symptom: phase 4 has logs only; the smoking-gun span never appears.

How to troubleshoot it

When the engineer reports “stuck in dashboards”, the diagnostic order is:

1. Confirm the page includes a metric, not a log line. A page that cites only a log count is phase 4 evidence dressed up as a phase 1 alert. Re-tier and route.
2. Confirm exemplars are present on the failing metric. curl /api/v1/query?query=...exemplar_labels returns the available exemplars.
3. Confirm trace IDs reach Loki. A logcli query with | json | trace_id=~".+" returns at least one log line in the symptom window.
4. Confirm a trace exists with that trace ID. tracecli lookup or the Tempo UI returns the trace within 5 seconds.
5. Confirm the slowest span is named. A trace with Unnamed span points at instrumentation that is not shipping the operation name; the loop cannot read phase 4 from it.

Security implications

The three signals together expose more than any one of them individually. The pivot from metric to trace to log can reveal PII or secrets that none of them carries alone.

• A metric for http_requests_total{user_id="..."} is rare but exposes user identifiers in label values.
• A trace for a slow request may include the database statement in a span attribute, including user identifiers in the WHERE clause.
• A log filtered by trace ID may include request bodies that were not stripped at the SDK layer.

The investigation loop walks all three. Limit access to the on-call rotation; redact at the SDK layer (do not redact at Loki query time, which is too late); sample traces by outcome, not by user identifier.

Performance implications

Three signals, three cost profiles.

• Metrics: bounded by cardinality and retention. Cheap to store, cheap to query when recording rules exist.
• Logs: bounded by ingestion rate and retention. Log ingestion cost dominates the platform bill at high traffic.
• Traces: bounded by span count and retention. Trace ingestion cost is the largest single line in a typical observability bill.

The investigation loop concentrates queries during an incident. Phase 4 queries against hot TSDB and high-volume log streams during a P0 will be the most expensive minutes the platform ever serves. Recording rules, structured log search, and trace search by ID are the optimisations; ad-hoc high-cardinality queries during a page are the failure shape.

Production guidance

• Configure once for the pivot points. Exemplars on histograms, trace IDs in logs, span attributes on traces. Without these, the methodology degenerates to hand correlation.
• Alert on the signal the next phase needs. P1 alerts cite the metric; P2 alerts may cite a log line if the metric does not exist; P3 alerts may cite a trace.
• Document the dashboard per phase, not per signal. A “phase 1-2 dashboard” with the user-visible metrics; a “phase 4 dashboard” with traces-and-logs split-views. The on-call rotation navigates the loop by opening the right dashboard.
• Bound phase 4 by time and cardinality. A 30-minute window against a recording rule beats an ad-hoc query against a hot TSDB.

Verification

You should now be able to answer:

• Which phase of the methodology uses metrics, which uses traces, and which uses logs?
• What is the role of exemplars in the metric-to-trace pivot?
• Why does putting trace IDs into logs enable trace-to-log pivots that would otherwise be impossible?
• Which of the six failure shapes (dashboard-of-everything, log-spike trap, trace-without-metric, cross-signal pivot missing, single-signal tunnel vision, trace sampled out) applies when a P0 log alert pulls the engineer to Loki before impact is quantified?
• Why should traces be tail-sampled by outcome rather than uniformly sampled at production traffic?

Quiz