On-call, escalation and handover

Most Proxmox estates are run by between two and six people. That is too few for a comfortable rota and more than enough to run the platform well, and the tension between those two facts is what this lesson is about.

The parts that go wrong are predictable: a rota that quietly requires somebody to be available every other week forever, a pager that fires for things nobody can act on at 03:00, an escalation path that has never been dialled, and a handover that says “quiet night” while leaving four temporary settings in place that nobody else knows about.

Rota arithmetic

Start with the honest numbers, because the conversation about resourcing goes much better with them than without.

What earns a page

The single largest determinant of whether a rota is survivable is the number of pages per shift. Everything else is secondary.

The test for whether something should page is not “is it important”. It is: is there an action a human can usefully take right now, and does waiting until morning make it materially worse? Two yeses is a page. Anything else is a ticket.

Escalation that has been tested

An escalation path is a list of people and a rule for when to move down it. The list is easy. The two things that make it work are harder.

Every entry has a tested contact method. Not “he’s in the directory”. A number that was dialled during a drill, by someone who was not the person who wrote the document.

Each step has a trigger, not a feeling. “Escalate if you are stuck” is not a trigger, because the people most likely to need escalation are the least likely to judge that they are stuck.

The handover, and the part everybody omits

A handover is usually written as a summary of what happened. That is the least useful part, because what happened is in the ticket system.

What is not recorded anywhere else is temporary state: the settings somebody changed to get through an incident and intends to change back. Proxmox and Ceph are generous with these, and neither will remind you.

set -euo pipefail

echo '=== Ceph flags (expect none set outside a maintenance window)'
ceph osd stat 2>/dev/null || echo 'not a Ceph cluster'
ceph health detail 2>/dev/null | head -20 || true
echo '--- muted health checks, with their expiry'
ceph health mute ls 2>/dev/null || true

echo
echo '=== HA: services not in the started state, and nodes in maintenance'
ha-manager config
ha-manager status

echo
echo '=== Quorum: expected votes should match the node count'
pvecm status | grep -E 'Expected votes|Total votes|Quorate'

echo
echo '=== Backup jobs: any disabled?'
pvesh get /cluster/backup --output-format yaml | grep -E 'id:|enabled:|comment:'

echo
echo '=== Notification matchers: any disabled?'
pvesh get /cluster/notifications/matchers --output-format yaml

echo
echo '=== Replication jobs: any disabled or failing?'
pvesr list

A handover that is worth reading

Five sections. Short is fine; complete is not optional.

1. Temporary state. The output of the sweep above, plus a line per item: what, why, who owns it, when it must be cleared. If this section is empty, say “none” explicitly — a blank section is ambiguous and an explicit “none” is a statement.

2. Open incidents. Each with its current state, what has been tried, and what the next step is. “Investigating” is not a state; “waiting for the vendor to respond to case 12345, expected tomorrow” is.

3. Changes in the last 24 hours. Anything that touched the platform, including changes made by other teams. Most incidents follow a change, and the incoming shift needs the list to correlate against.

4. Scheduled work in the coming shift. Backups, a firmware rollout, a storage migration, somebody else’s DNS change. Anything that will generate noise, so that expected noise is not investigated as an incident.

5. Fragile items. The things you would not touch tonight. “Node pve-04 has a marginal PSU, replacement Thursday.” “The Windows file server takes 12 minutes to boot, so do not assume a failed start.”

Making the rota survivable

Four practices that cost little and matter more than anything else in this lesson:

• Shadow before solo. A new engineer takes a shift alongside an experienced one before taking one alone. Two shadowed shifts is usually enough and it is the cheapest way to find out that the runbook assumes knowledge nobody wrote down.
• Compensate it. Payment, time in lieu, or both, and stated in advance. Unpaid on-call is a resignation on a delay.
• Protect the day after. Somebody paged at 03:00 does not start at 09:00. Making this a written rule rather than a favour is what stops it eroding.
• Review pages monthly, not incidents. Incidents get reviewed already. The page list — every page, whether it was actionable, whether it should have been a ticket — is what actually reduces the load, and thirty minutes a month on it is the highest-leverage meeting the team has.

Common mistakes

• Claiming 24/7 cover a three-person team cannot staff.
• Paging on warnings, which trains people to ignore the pager.
• An escalation path that has never been dialled.
• No credentials for vendor support in the on-call pack.
• Handover as a narrative of the shift rather than a register of state.
• Leaving noout, a health mute, a maintenance flag or a disabled backup job with no owner and no expiry.
• Assuming shared context makes the handover unnecessary.
• Reviewing incidents but never reviewing pages.

Key takeaways

• Below four people you have a rota, not guaranteed cover; say which one you are offering.
• A page requires an action a human can take now, where waiting makes it worse. Everything else is a ticket.
• Track pages per shift and actionable rate; deleting a useless alert is reliability work.
• Escalation steps need triggers and tested contact methods, and your Proxmox support tier has a response time that determines where it fits.
• The handover section that matters is temporary state: Ceph flags, HA maintenance and disabled states, muted checks, disabled backup jobs and matchers, lowered expected votes.
• Every temporary state gets an owner and an expiry when it is set, not when it is remembered. ceph health mute with a duration is the only one that expires by itself.

Knowledge check