Daily pre-shift check

A 5-minute check performed at the start of each shift.

How to use

Run each command. Note any unexpected output. Investigate anything unusual before the shift progresses.

Have the alerting console and the incident tracker open next to the terminal before you start. Four of the six items are answered by a command; the other two are answered by a person reading a queue, and hunting for the right browser tab mid-check is what turns a five-minute routine into a fifteen-minute one.

Order of work

The items are sequenced so each one narrows what the next has to explain, and the frontmatter list is in that order.

cluster-health and node-health are read first because every later reading is only as trustworthy as the quorum it was taken from. A cluster missing a node still answers questions about capacity and backups, but it answers them for the nodes that are left, and a number that looks healthy for four nodes can be hiding the fifth.

backup-status and capacity-warning are the two overnight outcomes: what ran while nobody was watching, and what it consumed while it ran. They belong together because a backup that failed and a filesystem that filled are frequently the same event seen from two directions.

alerts-inbox and ongoing-incidents close the shift handover and go last, so that anything the first four turned up can be matched against an alert that already fired or a ticket already open, rather than being raised a second time under a new number.

Notes

If anything fails, document in the incident tracker. Don’t fix things immediately unless they are user-impacting.