Service Health, User Impact, and Telemetry

Every telemetry signal exists in service of a decision. The decision is to act or not act. The signal is the bridge between an internal observation and the action. A signal that does not end in a decision is decoration.

This lesson traces the chain end to end. The chain is the same whether the decision is automated (an alert paging an engineer) or manual (an engineer looking at a dashboard to decide whether to roll back a deployment).

The chain

User
   ↓ (experiences)
Service
   ↓ (runs on)
Infrastructure
   ↓ (emits telemetry)
Telemetry
   ↓ (collected by)
Observability platform
   ↓ (read by)
Engineer
   ↓ (decides)
Action

The same diagram appears in many forms. The point is the same: telemetry exists because an operator needs to decide something. If the chain does not end in a decision, the signal serves no purpose.

What an SLI is

A Service Level Indicator (SLI) is a measured quantity. It answers a question about the service. Examples:

• ‘What fraction of /checkout requests succeed in <2 s?’
• “What fraction of /api/orders requests return 5xx?”
• “What is the 99th-percentile latency of /api/recommendations?”

The SLI is the measurement. It is what the telemetry produces. SLIs are observable quantities, not targets.

What an SLO is

A Service Level Objective (SLO) is a target over an SLI:

• “99% of /checkout requests succeed in <2 s over a 30-day window.”
• “Less than 0.1% of /api/orders requests return 5xx over a 30-day window.”

The SLO is the contract. It tells an operator when the service is acceptable. The 30-day window converts the SLI into a target that has a budget: 30 days × 24 h × 60 min × 60 s × request rate, with the success fraction leaving an allowable-error budget.

The decision chain

The chain from telemetry to decision is:

SLI measurement
   |
SLO target
   |
Budget burn rate
   |
Alert threshold
   |
Page or ticket
   |
Engineer action
   |
Mitigation
   |
SLI returns to target

The chain is the SLO-based alerting discipline — Part XXII covers it in production depth.

A telemetry signal that does not feed this chain is decoration:

• A metric with no corresponding SLO is decoration. Without an SLO, the operator does not know whether the metric being green is good.
• An alert without a runbook is decoration. Without a runbook, the paged engineer is paid to wake up and look around.
• A dashboard without an owner is decoration. Without an owner, the panel arrangement drifts; the metric stops being relevant.

The “lots of telemetry, no decision” failure

A platform team collects 10,000 metrics. The dashboards cover every host, every service, every dependency. The alerts fire hourly. The on-call engineer has 200 alert rules to attend to. After three months, the engineer stops looking at the alerts in detail; everything is muted.

This is the failure shape the SLO discipline prevents. An SLO-targeted alert fires only when an error budget is consumed faster than expected, and the alert is anchored to a user-visible metric. The number of alerts drops from “every metric above some threshold” to “the user-visible metric is in trouble.”

The SLO discipline is not perfect — it ignores internal failures that do not yet affect user-visible metrics. The course returns to this trade-off throughout Parts XX and XXII.

What every telemetry signal should answer

Three questions, in priority order:

1. Is the user-facing experience acceptable? (SLO metric)
2. What is failing inside the system? (operational metric)
3. What was the system doing at this moment? (event log / trace)

A signal that answers none of the three is decoration. A signal that answers the first is non-negotiable. A signal that answers the second is necessary for investigation. A signal that answers the third is necessary when the first two fail.

The discipline of asking these three questions before adding a new metric is the difference between signal and noise.

Production guidance

• Every alert must have a runbook link. The course returns to this in Part CXIII.
• Every panel in an operational dashboard must have an owner (a name, a team). Wallpaper dashboards correlate with investigation failures — Part XXVII.
• Every metric used for an SLO must be carefully designed (clean request-success criterion, low cardinality, accurate measurement).
• Every log shipped to Loki must have a reason (a specific incident class it supports). Shipping logs because “more is better” is a cost-driven failure.

Verification

You should be able to answer:

• What is the role of an SLI in the chain?
• What is an SLO, and how does it differ from an SLI?
• Why is “lots of telemetry, no decision” a failure shape?
• What three questions should every telemetry signal answer?

Quiz