Operational checklists: daily, weekly, monthly, quarterly, annual · Proxmox VE

Why this matters in production

Without regular checks, problems accumulate until they become incidents. A daily 5-minute check catches what automation misses.

A check that cannot fail is not a check

Before the cadence, the test that determines whether any of it is worth doing.

“Check the cluster is healthy” cannot fail. There is no defined outcome, so whatever the operator sees satisfies it, and the tick gets applied in four seconds by someone glancing at a dashboard that is green for reasons they have not examined.

“Run pvecm status and confirm Quorate: Yes with Total votes equal to the node count” can fail. It has one correct answer, the answer is observable, and an operator either saw it or did not.

Every line in every checklist below should be rewritable in the second form. If a line cannot be — if there is no command and no expected output — it is not a check, it is a reminder to think about something, and it belongs in a different document.

Daily

5 minutes. The first operator on shift:

Check the cluster dashboard for HEALTH_OK.
Check recent backup status.
Check the alerts inbox.
Glance at capacity warnings.

Written so each one can fail:

Read-only / Safethe daily check, with an expected answer for every line— Read-only. Run it from cron on one node and mail the output; the human reads it only when a line is not what it should be. Five minutes of attention is what this earns you, not five minutes of typing.

set -euo pipefail
FAIL=0
note() { echo "CHECK-FAIL: $*"; FAIL=1; }

# 1. Quorum. Expected: Quorate: Yes, total votes equal to expected votes.
pvecm status | grep -q 'Quorate:.*Yes' || note 'cluster is not quorate'

# 2. Every node online. Expected: no node missing from the membership.
expected_nodes=3
seen=$(pvecm nodes | awk 'NR>3' | wc -l)
[ "$seen" -eq "$expected_nodes" ] || note "only $seen of $expected_nodes nodes in membership"

# 3. Storage. Expected: every configured storage active.
pvesm status | awk 'NR>1 && $3 != "active" {print}' | grep -q . \
&& note 'a storage is not active' || true

# 4. Ceph, where present. Expected: HEALTH_OK, or a documented HEALTH_WARN.
if command -v ceph >/dev/null; then
ceph health | grep -q '^HEALTH_OK' || note "ceph: $(ceph health)"
fi

# 5. Backups. Expected: no failed task in the last 24 hours.
pvesh get /cluster/tasks --output-format json \
| grep -o '"status":"[^"]*"' | grep -v '"status":"OK"' | grep -q . \
&& note 'a cluster task ended in a non-OK state in the recent window' || true

# 6. HA. Expected: no service in an error state.
if command -v ha-manager >/dev/null; then
ha-manager status | grep -qi 'error' && note 'an HA service is in error' || true
fi

exit "$FAIL"

Weekly

30 minutes. Includes:

Review backup verification results.
Review Ceph HEALTH and capacity.
Review cluster alerts from the past week.
Confirm monitoring notifications work (fire a test).
Verify the on-call rotation.

Monthly

2 hours. Includes:

Review the workload inventory against reality.
Review capacity trends; plan additions.
Run a DR tabletop exercise.
Review change requests from the past month.
Update runbooks based on recent incidents.

Quarterly

Half day. Includes:

Capacity review; trigger procurement if utilisation > 70 %.
Review and update the hardening checklist.
Review and test the DR plan.
Review access controls (admin accounts, RBAC, TFA).
Update the production reference architecture document.

Annual

1-2 days. Includes:

Full DR exercise (live drill, not just tabletop).
Review Proxmox subscriptions and support contracts.
Hardware lifecycle review (replacement schedule).
Review the course curriculum (yes, this very course).
Compliance audit if applicable.

The cadence

flowchart LR
  A[Daily 5min] --> B[Weekly 30min]
  B --> C[Monthly 2h]
  C --> D[Quarterly 0.5day]
  D --> E[Annual 1-2d]
  E --> A

Production considerations

The checks that only a human can do

Automation covers the daily cadence well and covers the longer cadences badly, because the questions get less mechanical as the interval lengthens.

Cadence	Automatable	Needs a person
Daily	Quorum, storage, Ceph health, task failures, HA state	Nothing, if the script is good
Weekly	Verify-job results, backup freshness per group, capacity trend	Whether last week’s alerts were the right alerts
Monthly	Inventory reconciliation, tag compliance	Whether a capacity trend needs procurement, whether a runbook is still true
Quarterly	Certificate expiry, account and token age	Whether the DR plan still matches the architecture
Annual	Subscription and support expiry, hardware age	Whether the platform is still the right shape for the business

The pattern is that automation answers is it as configured and a human is needed for is it configured correctly. Both matter and only one of them scales.

Why the restore test belongs on the quarterly list and not the annual one

Of everything in the cadence, one check has a payoff so much larger than the others that it deserves separate treatment: actually restoring something.

Not verifying a backup — PBS verification confirms the chunks are readable and their checksums match, which is necessary and is not the same thing. Restoring: taking a real snapshot, restoring it to a new VMID on an isolated network, booting it, and logging in.

What that test catches, which nothing else does:

A backup that succeeds every night and produces an image whose guest will not boot, because the guest agent was never installed and every snapshot is crash-consistent against a database that does not survive it.
A restore procedure that assumes a storage which was renamed eight months ago.
An encrypted datastore whose key nobody can find.
A restore that works and takes six hours, against an RTO of two.
An operator who has never done one, discovering the interface for the first time during an outage.

The last one is the real value and it is why the cadence matters. A restore performed quarterly is a procedure the team knows. Performed annually, it is a thing somebody did once and half-remembers. The skill decays faster than the technology changes.

Quarterly, one guest, chosen at random rather than by the person doing the test, and timed. Record the elapsed time and compare it against the RTO you have published. That single number is more informative about your recovery capability than every other line in this lesson combined.

What happens when a checklist becomes theatre?

It does not stop being ticked. That is the whole problem — the artefact keeps being produced, so from the outside nothing changes.

The progression is consistent. The checklist is written well, with real checks. Over months a few lines are added that produce no signal: a check that has never once failed, a review of a dashboard nobody understands, a tick for something automation already covers. The list gets longer and its information density drops.

At some point completing it honestly takes twenty minutes instead of five, and it is being done at the start of a shift by somebody with other things to do. So it gets skimmed. Then it gets ticked in a batch at the end. Then it gets ticked in advance.

Nothing announces this. The completed checklists look identical throughout, and the team’s confidence in their operational discipline is undiminished right up to the incident that a check would have caught three weeks earlier.

Two defences, both about pruning rather than enforcement:

Delete any check that has never failed. Either it cannot fail, in which case it is noise, or the thing it watches is genuinely stable and does not need a human. Reviewing the list annually and removing lines is what keeps it short enough to be done honestly.
Make the output the evidence, not the tick. A check whose result is a command’s output pasted into the record cannot be completed without running it. A checkbox can.

The instinct when a checklist stops being followed is to add oversight. The thing that actually works is making the list short enough that following it is cheaper than pretending.

Common mistakes

Checks with no defined expected outcome, which cannot fail and therefore mean nothing.
“I’ll do it later”, after which no check happens.
A checklist that grows and is never pruned, until completing it honestly costs more than the time available.
Automating the daily check and not alerting on it failing to run, so silence is ambiguous.
Verifying backups and never restoring one.
An annual restore test, which is too infrequent for the skill to survive.
Skipping quarterly and annual reviews under pressure, which is exactly when the platform is drifting fastest.

Key takeaways

Daily 5 min, weekly 30 min, monthly 2 h, quarterly 0.5 day, annual 1–2 d.
Every line needs a command and an expected answer, or it is not a check.
Automate the daily cadence, exit non-zero on failure, and alert separately on the check not running.
Automation answers is it as configured; a person is needed for is it configured correctly.
Restore a real guest quarterly and time it against your published RTO.
Prune checks that have never failed; a short list is followed and a long one is faked.