Defining and measuring an SLA · Proxmox VE

“We offer 99.9% availability” is a sentence that gets written into service catalogues by people who have not done the arithmetic. It commits the platform to no more than 43 minutes and 12 seconds of downtime per month, and it is worth working out immediately whether a single rolling upgrade of the cluster fits inside that.

Usually it does not. Which does not mean the target is wrong — it means the number is only half the commitment, and the other half is the definitions around it.

The arithmetic

Target	Per month	Per quarter	Per year
99%	7 h 18 m	21 h 54 m	3 d 15 h
99.5%	3 h 39 m	10 h 57 m	1 d 19 h
99.9%	43 m 12 s	2 h 10 m	8 h 46 m
99.95%	21 m 36 s	1 h 5 m	4 h 23 m
99.99%	4 m 19 s	12 m 58 s	52 m 36 s
99.999%	26 s	1 m 18 s	5 m 16 s

Two rows deserve comment.

99.99% is 4 minutes and 19 seconds per month. That is less time than it takes to reboot a guest. A single unplanned VM restart, anywhere in the month, exhausts the budget. Achieving it requires that no individual failure is user-visible, which means clustering inside the application, not underneath it.

99.999% is 26 seconds per month. No hypervisor platform delivers this, because the failure detection alone takes longer than the budget. If somebody asks for five nines, they are asking for an application architecture, and the conversation to have is about that rather than about the platform.

What Proxmox HA physically gives you

This is the ceiling, and it is worth deriving rather than asserting, because it settles the question of what you can promise.

When a node fails with HA-managed guests on it:

Stage	Elapsed
Node loses quorum; the LRM stops renewing the watchdog	0 s
Watchdog expires and the node self-fences	~60–70 s
The CRM marks the node offline (60 s since its last status update) and begins trying to acquire its lock	from ~60 s
Lock acquisition completes — worst case the lock takes a further 60 s to time out	up to ~120 s
The CRM selects a recovery node and starts the guest	+seconds
The guest boots and the application becomes ready	+30 s to several minutes

HA converts an unbounded outage into a bounded one of a few minutes. It does not make failure invisible

The realistic end-to-end recovery for an HA-managed guest is two to five minutes: roughly two minutes of detection and fencing, plus however long the guest takes to boot and the application takes to become ready.

That is an enormously valuable property. Without HA, a node failure at 02:00 is an outage that lasts until a human wakes up, connects, and decides what to do. With HA it is a bounded event that resolves itself while everyone sleeps.

But it is a restart, not a failover. In-flight requests fail, sessions held in guest memory are lost, and anything mid-transaction rolls back.

The arithmetic that follows is direct: at two to five minutes per incident, one HA recovery per month fits inside a 99.9% budget and comfortably breaks a 99.99% one. So 99.9% is the honest ceiling for a service whose resilience story is “Proxmox HA”, and anything above it requires the application to survive the loss of an instance — a clustered database, a load-balanced application tier, a queue that redelivers.

Say that explicitly when a target is being negotiated. It converts a disagreement about numbers into a conversation about architecture, which is where it belongs.

SLI, SLO, SLA — three different things

They get used interchangeably and they are not.

Term	What it is	Who it is for	Example
SLI	Service Level Indicator — the thing you actually measure	Engineering	Fraction of HTTP probes to the service endpoint returning 200 within 2 s
SLO	Service Level Objective — your internal target for that indicator	Engineering and management	The SLI is at least 99.95% over a rolling 30 days
SLA	Service Level Agreement — the commitment with a consequence attached	Customers, contracts, finance	99.9% monthly, with service credits below it

Measure from the service, not from the hypervisor

This is the single most common measurement error, and it produces reports that are technically accurate and completely useless.

Node uptime is not service availability. A node can be up for 400 days while the guest on it has been unreachable for six hours. A guest can show running in qm list while its application has been returning 502 since lunchtime. Every layer beneath the service is a necessary condition and none of them is sufficient.

Measurement point	Measures	Blind to
Node uptime	The hypervisor booted and stayed up	Everything above it
`qm status` shows running	QEMU has a process	Guest kernel panic, application failure, network isolation
Guest agent responds	The guest OS is alive	The application
Port is open	Something is listening	The application returning errors
Synthetic probe against the service endpoint, from where a user is	What the user experiences	Only genuinely user-specific problems

Read-only / Safewhat a service-level probe looks like, and where it must run from— Read-only. The important property is not the command - it is that this runs from outside the cluster, so that a failure of the cluster is a failure of the probe rather than a gap in the data.

set -euo pipefail
ENDPOINT=https://app.example.com/healthz
TIMEOUT=2

# A probe that fails is a probe that measures something. Check the status
# code AND the latency, because a service that answers correctly in nine
# seconds is not available in any sense the user recognises.
start=$(date +%s%3N)
code=$(curl -sS -o /dev/null -w '%{http_code}' --max-time "$TIMEOUT" "$ENDPOINT" || echo 000)
end=$(date +%s%3N)
latency=$(( end - start ))

if [ "$code" = "200" ] && [ "$latency" -lt 2000 ]; then
echo "ok code=$code latency_ms=$latency"
else
echo "FAIL code=$code latency_ms=$latency"
fi

# Run this from a host that does not depend on the cluster being healthy.
# A probe running inside a VM on the cluster it measures reports 100%
# availability during every outage, because it was down too.

The measurement interval decides what you can even detect

Two clusters both claiming 99.9% can be measuring completely different things.

Polling interval. A probe every 5 minutes cannot see a 90-second outage. It also cannot place an outage more precisely than ±5 minutes, so a genuine 4-minute failure is recorded as anything from 0 to 10. For a 99.9% target where the whole monthly budget is 43 minutes, a 5-minute resolution is too coarse to measure the thing you are committing to. Probe at 30 seconds or faster, and be explicit that the SLI is computed from those samples.

Aggregation window. 99.9% over a rolling 30 days and 99.9% over a calendar year are different commitments. The annual figure permits a single 8-hour-46-minute outage; the monthly figure caps any single incident at 43 minutes before the month is lost regardless of what else happens. Monthly is stricter in practice and is what customers usually assume.

Consecutive-failure rules. Most SLAs require n consecutive failed probes before an outage is counted, which suppresses noise from a single dropped packet. It also means the outage clock starts late. State the rule.

Exclusions, written so they cannot be argued about

Every SLA has exclusions. The failure mode is not having them — it is having them written vaguely enough that they become a negotiation while the incident is still running.

Exclusion	Written badly	Written so it holds
Planned maintenance	“Scheduled maintenance is excluded”	“Excluded when announced at least 5 business days in advance, within the window Sunday 02:00–06:00 local, and limited to 4 hours per calendar month”
Customer-caused	“Issues caused by the customer”	“Downtime arising from customer configuration changes, customer software, or exhaustion of a resource the customer was notified was approaching its limit”
Dependencies	“Third-party failures”	“Named upstream services: internet transit from Provider X, authoritative DNS, the corporate identity provider. Failure of a named dependency is excluded; failure of our integration with it is not”
Force majeure	“Acts of God”	Standard contractual language, defined once, referenced
Emergency security patching	Usually absent entirely	“Downtime taken to remediate a vendor advisory rated High or Critical, with notice as soon as practicable”

The maintenance exclusion is what makes the number achievable, and it must be bounded

Without a planned-maintenance exclusion, a rolling PVE major upgrade — drain each node, reboot, verify, move on — consumes the entire monthly budget of a 99.9% service in one evening, even with live migration, because guests on local storage cannot migrate and some workloads do not survive a live migration cleanly.

With an unbounded exclusion, “planned maintenance” becomes a loophole that makes the SLA meaningless, and any customer who has been burned before will say so during negotiation.

The workable form has three limbs: a notice period, a fixed window, and a cap on total excluded time per month. That last one is what makes it honest, and it is the one most often left out.

The emergency-patching exclusion is worth arguing for separately. Without it, you are contractually incentivised to delay patching a critical vulnerability until the next window, which is precisely the wrong behaviour and the exact scenario the security-advisory response lesson is about.

Error budgets: what to do with the 43 minutes you have not spent

The most useful reframing of an availability target is that it is not a target to hit but a budget to spend.

99.9% monthly grants 43 minutes. If 8 minutes have been consumed by the 14th, you are holding 35 minutes of permitted downtime, and holding it earns nothing. Spending it deliberately — on the firmware update you have been deferring, the storage migration, the kernel upgrade — is what converts an availability target from a constraint into a planning tool.

The policy that makes this real is a budget-driven change posture:

Budget consumed	Posture
Under 25%	Green — proceed with planned risky work
25–75%	Normal — standard changes proceed, risky ones need a reason
Over 75%	Freeze — only changes that reduce risk, plus emergency fixes
Exhausted	Everything stops except reliability work until the window rolls

This does two things that exhortation cannot. It gives the platform team an objective, pre-agreed basis for saying no to a change during a bad month — which is much easier to defend than a judgement call. And it removes the incentive to hoard reliability by doing nothing, because unspent budget is visibly wasted rather than quietly virtuous.

It also converts the awkward conversation. “We cannot do the upgrade this week” is an opinion. “We have used 38 of 43 minutes this month and the policy is freeze above 75%” is a fact everyone agreed to in advance.

What happens if you promise an availability target the architecture cannot deliver?

Nothing, for a while. Most months contain no incident at all, so a service that is architecturally capable of 99% will report 100% for a long stretch and everybody concludes the target is comfortable.

The bill arrives all at once. A single node failure with a slow-booting guest produces eight minutes of downtime. On a 99.9% monthly SLA that is fine. On the 99.99% somebody optimistically wrote into the service catalogue, it is nearly double the entire monthly budget, and it happened for a completely ordinary reason that no amount of operational excellence would have prevented.

What follows is worse than the outage. The team is asked what went wrong, and the honest answer — “nothing went wrong, a server failed and HA did exactly what it is designed to do” — sounds like an excuse. So the remediation actions become theatrical: more monitoring, more process, a post-incident review of an incident that had no defect in it. Meanwhile the actual fix, which is either changing the target or changing the application architecture, does not get discussed because neither is in the platform team’s gift.

The way out is upstream. When the target is being set, state the ceiling and show the derivation: HA recovery is two to five minutes, so any single node failure spends that much budget, so the target has to be one that survives a few of those a year. If the business needs better, the money goes into the application, not into the hypervisor.

Reporting without flattering yourself

A monthly availability report that is one number is not a report. Four elements make it useful:

The SLI, to two decimal places, unrounded. 99.87% is not “about 99.9%”. Rounding towards your target is the fastest way to lose the audience the one time it matters.
Budget consumed, in minutes and as a percentage. “We used 34 of 43 minutes” is understood by everyone; “99.92%” is understood by almost nobody.
Every incident, with its duration, cause and whether it counted against the budget or was excluded — and under which exclusion.
The trend. One month is noise. Six months is a signal, and it is the only thing in the report that supports a decision.

Common mistakes

Publishing a target without doing the downtime arithmetic.
SLO equal to SLA, so the first warning is the breach.
Measuring node uptime and calling it service availability.
Running the probe inside the cluster it measures.
A 5-minute polling interval against a 43-minute monthly budget.
Unbounded maintenance exclusions, or none at all.
No emergency-patching exclusion, creating an incentive to delay security fixes.
Rounding the SLI towards the target.
Treating unspent error budget as an achievement rather than as capacity that was not used.

Key takeaways

99.9% is 43 minutes a month; 99.99% is 4 minutes and 19 seconds.
Proxmox HA recovery is two to five minutes per incident, which makes 99.9% the honest ceiling for a platform-level resilience story.
SLI is measured, SLO is internal and tighter, SLA is contractual.
Measure at the service endpoint, from outside the cluster, at 30-second resolution or better.
Exclusions must be bounded and specific — notice period, fixed window, and a monthly cap.
Treat the budget as something to spend deliberately, with a pre-agreed posture at each consumption level.
Report the unrounded SLI, the budget consumed, every incident, and the trend.

Knowledge check

Why this matters

An SLA is the sentence in which a platform team makes a promise on behalf of hardware it does not control, to people who will remember the promise more clearly than the caveats.

Most of the harm done by availability targets comes from setting them casually. Somebody writes 99.9% because it looks respectable, nobody converts it to 43 minutes, no exclusions are defined, measurement is a monthly glance at node uptime, and the arrangement holds fine right up until a real incident — at which point every undefined term becomes a dispute, during the outage, with an audience.

The work in this lesson is nearly all preventative and nearly all cheap. Do the arithmetic. Derive the ceiling your architecture actually supports and say it out loud while the number is still negotiable. Write the exclusions so precisely that they are boring. Put the probe somewhere that survives the failure. Report the number you actually measured.

Do those five things and the SLA becomes what it should be: a shared understanding that makes the difficult conversations shorter, rather than a trap that springs on the worst night of the year.