Correlation with metrics and logs — the joined query

The change-cause annotation marks the moment. The version label carries the runtime identity. The event log carries the actor and the build URL. None of the three answers the on-call engineer’s question directly. The question is “show me every metric, every log line, every trace that the suspect version produced in the incident window”. The answer is the joined query: the metric stream filtered by the version label, the log stream filtered by the OpenTelemetry attribute, the trace stream filtered by the same attribute, all anchored by the deploy event. The marker is the join key; the query is the postmortem view.

Filtering metrics by version

The version label propagates to the Prometheus metric labels through kube-state-metrics and through the workload SDK resource attributes. The investigator queries the error rate by version with a PromQL query that filters by the version label:

sum by (version) (
  rate(http_requests_total{service="api",status=~"5.."}[5m])
)

The query returns a time series per version. The investigator reads the series, identifies the version whose error rate spiked in the incident window, and filters further:

sum(
  rate(http_requests_total{service="api",version="v3.4.1",status=~"5.."}[5m])
)
/
sum(
  rate(http_requests_total{service="api",version="v3.4.1"}[5m])
)

The query returns the error rate for v3.4.1 specifically. The investigator compares the rate to the previous version’s rate and to the long-term baseline. The comparison produces the regression signal that identifies v3.4.1 as the suspect deploy.

flowchart LR
    A["Deployment label\napp.kubernetes.io/version=v3.4.1"] --> B["kube-state-metrics\nmetric label"]
    A --> C["Workload SDK\nOTEL_SERVICE_VERSION"]
    B --> D["Prometheus\nfiltered by version"]
    C --> E["Tempo / Loki\nfiltered by version"]
    D --> F["Joined query\npostmortem view"]
    E --> F
    G["Deploy event\ntimestamp + SHA"] --> F

Filtering logs by service.version

The OpenTelemetry service.version attribute propagates to every log line emitted by the workload. The investigator queries the log stream with a LogQL query that filters by the attribute:

{service="api"} | json | service_version="v3.4.1" | status="error"

The query returns only the error log lines emitted by v3.4.1 in the incident window. The investigator reads the lines, identifies the failing endpoint, and pivots to the trace stream to recover the full request context.

The log query is the second surface of the joined view. A team that has only the metric query sees the rate but not the cause; a team that has only the log query sees the lines but not the rate. The two together produce the postmortem view that compresses the investigation.

The trace-to-deploy pivot

The trace stream is the third surface of the joined view. The OpenTelemetry resource attributes propagate to every span; the investigator queries Tempo by resource.attributes.service.version:

{resource.attributes.service.version="v3.4.1" && span.status.code="error"}

The query returns only the failing spans emitted by v3.4.1. The investigator clicks an exemplar on the metric panel, opens the trace, walks the span tree, and identifies the failing service or the slow database query.

The trace-to-deploy pivot is the closure of the correlation loop. The metric shows the regression; the log line carries the error; the trace carries the request context; the deploy event carries the actor, the build URL, and the approval chain. The four signals are joined by the version reference and anchored by the deploy timestamp.

The joined query as a Grafana panel

The joined view is a single Grafana panel that combines the metric, the log, the trace, and the deploy event. The panel has:

• A metric panel filtered by version="v3.4.1" showing error rate, latency, and throughput.
• An annotation layer showing the deploy event as a vertical line with the annotation text.
• An Events panel showing the structured deploy events with the actor, the build URL, and the SHA.
• A Loki panel filtered by service_version="v3.4.1" showing the error log lines.
• A Tempo panel filtered by resource.attributes.service.version="v3.4.1" showing the failing spans.

The investigator opens one panel, sees all four signals, and walks from the metric to the log to the trace to the deploy event. The investigation compresses from minutes to seconds. A team that has only the metric panel sees the regression but not the cause; a team that has the joined panel sees the regression, the cause, the failing requests, and the deploy context in one view.

Production discipline

1. Query metrics, logs, and traces by version, not by exact timestamp. The reference is the join key; the timestamp is the wall-clock hint.
2. Propagate the version label to every surface. The label propagates to the pods, the metric labels, the SDK resource attributes, and the audit events.
3. Surface the joined view as the on-call default. The dashboard the engineer opens first must be the joined view; everything else is a drill-down.
4. Carry the version attribute on every span and log line. The OpenTelemetry SDK reads the OTEL_SERVICE_VERSION environment variable and emits it on every record.
5. Audit the propagation as a deployment-failure condition. A version that is on the workload label but missing from the SDK attribute is a deployment that cannot be queried.

Cross-course references

• This course, Part CXIII (ObservabilityIntegration) covers the timeline view that the joined query extends with the version-filtered signals.
• Observability course - Part LII (Exemplars) covers the metric-to-trace pivot that the trace-to- deploy pivot extends.
• This course, Part CXIV-03 (RuntimeIdentity) covers the version label that is the join key for the joined query.

Quiz