What to monitor on Linux - the production checklist

Not every metric matters. A production monitoring plan covers the essential signals and leaves the rest for ad-hoc investigation. This lesson is the checklist.

Essential metrics

For every Linux host, monitor:

The lane column is as important as the threshold. A page interrupts a human; a ticket goes into a queue that gets read during working hours. Getting this wrong in either direction is how monitoring becomes noise or becomes decorative.

The two CPU rows are deliberate. Linux load average counts runnable tasks and tasks in uninterruptible sleep, so a host blocked on a slow NFS mount reports a load of 40 while its CPUs sit idle. An alert keyed to load > 2 * nproc pages the CPU owner for a storage incident, and it pages nobody when 4 threads saturate a 4-core box at load 4. /proc/pressure/cpu (node_pressure_cpu_waiting_seconds_total) measures the time tasks actually spent waiting for CPU, which is the thing the threshold was reaching for. Keep load average on the dashboard as context; do not route on it.

The three disk rows are three different failures. Blocks and inodes exhaust independently - a filesystem with free blocks and zero free inodes reports normal usage, refuses every new file and pages nobody (node_filesystem_files_free / node_filesystem_files). The fill prediction turns a threshold into lead time (predict_linear). And the read-only row is the remount the kernel performs after an I/O error (node_filesystem_readonly == 1): usage stops changing, so the % full alert never fires while the host quietly stops accepting writes.

Whatever query you build for these, exclude pseudo-filesystems with fstype!~"tmpfs|devtmpfs|overlay" rather than allow-listing one fstype. An allow-list such as fstype="ext4" drops XFS, the RHEL, Rocky and Alma default root filesystem, and the gap is invisible because the series simply is not there.

Service health

For every running service, monitor:

• Process running: is the service up?
• Health check: does it respond to health endpoints?
• Latency: p50, p99 of health check response.
• Errors: rate of health check failures.

Examples:

• Web server: HTTP 200 from /health endpoint, p99 < 100ms.
• Database: connection succeeds, simple query < 10ms.
• Queue: depth, throughput, error rate.

Security events

For every host, alert on:

• Failed logins (SSH, sudo, console).
• New user or group added.
• Sudoers change.
• Package install (especially security-related).
• auditd AVC denials (MAC).
• File integrity changes (AIDE, OSSEC).

These catch a class of problems that resource monitoring misses.

Performance baselines

For each host, capture a baseline:

• Idle CPU, memory, disk, network.
• Boot time.
• Process count at idle.
• Open file count at idle.

Alert on deviation from baseline (e.g. 30% higher CPU than baseline).

Long-term trends

Alert on trends, not just current state:

• Disk will fill in X days at current rate.
• Memory has been growing Y% per week.
• Certificate expires in Z days.

These predict problems before they happen.

Anti-patterns

• Alert on every metric: alert fatigue. Alert on user-impacting signals.
• No documentation: every alert needs a runbook.
• The same static threshold on every host, in the paging lane: 85% full is a sensible ticket on any host and a useless page on all of them - the log server crosses it every week and the database never does. Static thresholds are fine where they open a ticket; what pages should be a prediction (predict_linear) or a per-host baseline.
• No deduplication: 100 alerts for the same problem. Group.
• No escalation: critical alerts wake up the wrong team. Route by service.

Knowledge check