Alert design: SLOs, symptoms, and dashboards · Proxmox VE

Why this matters in production

Alerts that fire on every blip train operators to ignore them. Alerts that fire only when a user is impacted keep operators attentive. This lesson teaches the design.

SLOs

An SLO (Service Level Objective) is a target for a measurable behaviour. For Proxmox:

SLO	Definition	Target
VM availability	% of time the VM is running	99.95% for Tier 0
Backup success	% of scheduled backups that complete	99%
Cluster quorum uptime	% of time cluster has quorum	99.99%
Restore success	% of restore attempts that succeed	100%

Symptom-based alerts

Page when the user is impacted:

“VM 100 is down” — page.
“Cluster has quorum” — dashboard only.
“Backup failed for VM 100” — page for Tier 0, log for Tier 3.
“Ceph HEALTH_WARN” — page if it persists > 5 min.

An alert rule is more than a threshold

Most alert rules consist of a condition and a destination. That is enough to fire and not enough to act on, and the gap is felt entirely by the person woken at 03:00.

A complete rule carries six things:

Field	Why it exists
Condition	The measurement and the threshold
`for` duration	How long it must hold before firing. Without this, every transient blip pages
Severity	Page, ticket or dashboard — decided when the rule is written, not by the responder
A one-line statement of what is wrong	In the alert text. “CephOSDNearFull” is a rule name; “OSD.7 is at 87%, cluster writes stop at 95% on any single OSD” is a statement somebody can act on
A link to the runbook	The responder should not be searching a wiki at 03:00
An owner	Who is accountable for this rule still being correct in six months

The last two are almost always missing, and they are the two that decay. A rule with no owner is nobody’s job to delete when the thing it watches has been replaced, which is how an alert set accumulates rules that fire for conditions that no longer exist.

The missing `for` duration is why most alert sets are noisy

A threshold with no duration fires on a single sample. Every scrape is an independent chance to page.

A node’s I/O pressure crosses its threshold for eleven seconds while a backup snapshot is taken. A Ceph OSD reports slow ops during a scrub. A guest’s CPU spikes as a cron job starts. None of these is a problem and each of them is a page.

The fix is one field. for: 5m means the condition must hold continuously for five minutes, which eliminates essentially all transient noise at the cost of five minutes of detection latency — a trade that is correct for nearly everything except a hard outage.

Choose the duration from the signal, not from a default:

Signal	Reasonable `for`
Service endpoint unreachable	1–2 min
Cluster inquorate	2 min — long enough to survive a brief membership reform
Ceph writes blocked	1 min
Ceph nearfull, backup failures, certificate expiry	15 min or more; these are not urgent by the minute
Pressure or latency thresholds	5–10 min

Multi-window burn rate

For SLOs, use multi-window burn rates:

Alert if:
  (error rate over 1h > 14.4 * SLO budget) AND
  (error rate over 6h > 6 * SLO budget)

This catches both fast-burning incidents and slow-burning ones, while suppressing noise.

Where 14.4 comes from, and what the second window is actually for

The numbers look arbitrary and they are not; working them out once makes the whole technique obvious.

Take a 99.9% SLO over 30 days. The error budget is 0.1% of the period — 43 minutes and 12 seconds.

Burn rate is how fast you are consuming that budget relative to spending it evenly across the period. A burn rate of 1 means the budget lasts exactly 30 days. A burn rate of 14.4 means it is being consumed 14.4 times faster, which exhausts 30 days of budget in 30/14.4 days — a little over two days. Measured over a one-hour window, that rate means roughly 2% of requests failed in that hour, which is a real incident happening right now.

Why two windows. The short window is what makes the alert fast. It is also what makes it noisy, because one hour of bad numbers can be a deployment, a network blip, or a single unlucky minute amplified by a small denominator.

The long window is the confirmation. Requiring six hours to also be above a lower multiple (6 times budget rate) means the alert only fires when the problem has persisted long enough to be real. And because both conditions must hold, the alert resolves quickly too: as soon as the short window recovers, the alert clears, without waiting six hours for the long window to drain.

Typical pairs, and what each is for:

Short window	Long window	Multiplier	Catches
5 min	1 h	14.4	A fast burn — budget gone in ~2 days. Page
30 min	6 h	6	A moderate burn — budget gone in ~5 days. Page
6 h	3 d	1	A slow leak that will exhaust the budget by month end. Ticket

The bottom row is the one people omit and it is the one that catches the failure mode nobody notices: a service that is 99.85% available every day for a month, never bad enough to page, and comfortably in breach by the 30th.

Dashboards

Effective dashboards are layered:

Layer	Audience	Content
Overview	Leadership	SLO compliance, incidents, capacity
Operations	Ops team	Host health, VM status, storage, network
Deep-dive	Specialist	Per-component detail, logs, traces

flowchart LR
  A[Overview] --> B[Operations]
  B --> C[Deep-dive]
  C --> B
  C --> A

Pages vs tickets

Channel	Use for
Page	Production outage; user-impacting
Ticket	Operational hygiene; non-urgent; future work
Log	Diagnostics; trends

A Tier 0 VM is down → page. A Tier 3 VM has high I/O latency → ticket.

Production considerations

What happens to a dashboard designed by the person who understands the system?

It becomes unusable by everybody else, and the author is the last person to notice, because to them it is perfectly clear.

The pattern is recognisable. Thirty panels on one screen. Axis labels consisting of the raw metric name. No thresholds drawn, because the author knows what normal looks like. Panels arranged by the order they were built rather than by the order somebody would consult them. Three different time ranges across the page. Colours chosen so that everything is visible rather than so that bad things stand out.

At 03:00, woken from sleep, the person on call opens it and cannot answer the only question they have: is this thing broken, and where.

Design for that moment instead:

One screen, one question. The top-level dashboard answers is anything wrong, in a form readable from three metres away. Nothing else belongs on it.
Draw the threshold on the graph. A line at 85% turns “is 78 bad?” into a glance. Without it, every panel requires the reader to remember a number.
Say what the panel means, not what the metric is called. “Fullest OSD — cluster writes stop at 95%” instead of ceph_osd_utilization.
Green must mean good. If a panel is normally amber because a warning has been present for a year, it has stopped carrying information and belongs in a ticket instead.
Link from alert to dashboard to runbook, in that order, so the path from being woken to acting is clicks rather than searching.

The test is not whether the dashboard is comprehensive. It is whether somebody who did not build it, under pressure, reaches the right conclusion from it. Ask a colleague to try during a drill; the result is usually humbling and always cheap.

Common mistakes

Thresholds with no for duration, so every transient blip pages.
Alerts with no runbook link and no owner, so nobody deletes them when they stop being relevant.
A single burn-rate window, so either it is noisy or it is slow.
No slow-burn alert, so a service that is slightly bad every day breaches its month without ever paging.
Dashboards built for their author, unreadable by the person on call.
Panels that are permanently amber, which stop carrying information.
Paging on every metric breach.
Reviewing rules rather than reviewing the pages that fired.
Missing the alert entirely because the notification path is broken and nothing tests it.

Key takeaways

SLOs map to user experience and must be measured before they are promised.
A complete alert rule has a condition, a for duration, a severity, a plain-language statement, a runbook link and an owner.
Burn-rate multipliers are derived: 14.4 over one hour exhausts a 30-day budget in about two days.
Two windows make an alert both fast and trustworthy, and let it resolve quickly.
Add a slow-burn ticket alert for the service that is never quite bad enough to page.
Design the dashboard for the person being woken; draw the thresholds and say what the panel means.
Review the pages that fired, monthly, and delete the ones that needed no action.

Knowledge check

Why this matters

Alerting is where monitoring either becomes useful or becomes furniture, and the difference is not the quality of the metrics.

An estate with mediocre instrumentation and a small set of well-designed alerts — each with a duration, a plain-language message, a runbook and an owner — responds well to incidents. An estate with excellent instrumentation and two hundred threshold rules has an on-call engineer who has learned that the pager is usually wrong, and that lesson is applied uniformly to the pages that were right.

The two habits that matter most are unglamorous. Put a for duration on everything, because that one field removes most of the noise in most alert sets. And review the pages that actually fired rather than the rules in the abstract, because rules always sound defensible on paper and the record of what woke somebody up never does.