Monitoring design - what to monitor, how to alert, and why · Linux

Monitoring is the discipline of observing a system to detect problems before users do. This lesson covers the frameworks and the practical decisions.

The four golden signals

For any service:

Latency: time to serve a request. p50, p99, p99.9.
Traffic: requests per second.
Errors: rate of failed requests (5xx, exceptions).
Saturation: how full the service is (queue depth, utilisation).

For storage:

IOPS, throughput, latency, queue depth (USE method).

For infrastructure:

CPU, memory, disk, network (USE method).

RED method (per service)

For each service:

Rate: requests per second.
Errors: errors per second.
Duration: time per request.

Apply per endpoint, per status code. The method is simple and complete.

USE method (per resource)

For each resource (CPU, memory, disk, network):

Utilisation: busy percentage.
Saturation: queue length or waiting.
Errors: error counts.

USE is fast: list resources, check three metrics each, find the bottleneck.

Choosing the right framework

Service (web server, database): RED.
Resource (CPU, memory, disk): USE.
End-to-end (request flow): golden signals.

Both are useful. RED for the service, USE for the host.

What to monitor on Linux

For every host, monitor:

CPU: utilisation, load average, per-CPU if multi-threaded.
Memory: available, swap activity, OOM events.
Disk: per-device utilisation, await, queue depth, errors.
Network: per-interface utilisation, drops, errors.
System: uptime, NTP offset, login events, sudo usage, package updates, security events.

For services, monitor:

Process running: is the service up?
Health check: does the service respond?
Latency: how long does it take?
Errors: rate of failures.

Alert design

Good alerts are:

Actionable: the responder can do something.
Specific: clear what the problem is.
Symptom-based: alert on the user-visible problem, not the internal cause.
Avoid noise: false alerts train operators to ignore.

Bad alerts:

“CPU at 80%”: not actionable if no one cares about CPU.
“Memory usage > 90%”: noisy, not actionable.
“Disk full”: too late; should have alerted at 80%.

Good alerts:

“Service is down”: clear and actionable - provided you have the metric that can produce it. See the next section; up == 0 is not it.
“API latency p99 > 1 second for 5 minutes”: user-impacting.
“Disk will fill in 24 hours at current rate”: predictive.

Where “service is down” actually comes from

“Service is down” is easy to write on a slide and easy to get wrong in Prometheus. The metric that looks like it means this does not.

# WRONG for "service is down".
up == 0

up is synthetic: Prometheus sets it to 1 when the scrape succeeded and 0 when it failed. It tells you the exporter is unreachable, not that nginx stopped. A host whose node_exporter is happily answering while nginx is dead reports up == 1, no alert fires, and the outage is reported by users.

Two mechanisms produce the real signal. Use both: they answer different questions.

Unit state, from the node_exporter systemd collector

Answers “is the unit running on this host?”. The collector is not enabled by default, and it is expensive if you let it export every unit, so pin it to the units you care about.

node_exporter --collector.systemd \
  --collector.systemd.unit-include='(sshd|nginx|postgresql)\.service'

That exports node_systemd_unit_state, one series per unit per state, with the value 1 on the state the unit is currently in.

- alert: ServiceNotActive
  expr: node_systemd_unit_state{state="active"} == 0
  for: 2m
  labels:
    severity: page
  annotations:
    summary: '{{ $labels.name }} is not active on {{ $labels.instance }}'
    runbook: 'https://runbooks.example.com/service-not-active'

User-visible reachability, from blackbox_exporter

Answers “can a user reach it?”. blackbox_exporter probes an endpoint over HTTP, TCP or ICMP from outside the host and exports probe_success and probe_duration_seconds.

- alert: EndpointDown
  expr: probe_success == 0
  for: 2m
  labels:
    severity: page
  annotations:
    summary: 'Probe to {{ $labels.instance }} is failing'

- alert: EndpointSlow
  expr: probe_duration_seconds > 1
  for: 5m
  labels:
    severity: ticket

Which one to page on

The unit-state alert is a cause; the probe is a symptom. Page on the probe, because it is what the user feels, and attach the unit-state series to the same runbook so the responder can tell in one look whether the process died or the path to it broke.

An endpoint failing with the unit active means a network, certificate or upstream problem. The unit inactive with the endpoint still passing means you are serving from somewhere else than you think.

Avoid common pitfalls

Alert fatigue: too many alerts = operators ignore them. Tune aggressively.
Symptom vs cause in the wrong lane: page on symptoms (user-impacting); route causes (CPU, memory, disk level) to a ticket. Not “never alert on causes” - a resource problem nobody is looking at becomes an outage.
A static threshold in the paging lane: not all hosts are the same, so one number cannot page correctly for all of them. Static thresholds earn their place opening tickets, where a false positive costs a glance. What pages should be a prediction or a per-host baseline.
No runbook: every alert should have a runbook.