Weekly capacity and integrity review

A 30-minute review each week.

Before you start

Have last week’s capacity figures to hand, not just this week’s — three of the seven items are answered by a trend rather than a reading, and a single number tells you nothing about the direction it is moving in. Keep the alerting console and the on-call schedule open alongside the terminal.

Order of work

The items fall into three groups and the frontmatter list is in that order.

backup-verify, ceph-health and scrub-status are the integrity group: three independent answers to whether the data written in the past week is still readable. PBS verify covers the backup copy, Ceph covers the replicated copy, and the ZFS scrub covers the local pool. Read all three before drawing any conclusion — a failure in one and a clean result in the other two usually points at one device, while failures across two point at something shared, such as a controller or a fabric.

capacity-trend follows, because growth is only interesting once integrity is established. Two thresholds apply and they answer different questions: the weekly growth rate says how much warning is left, and the absolute utilisation says how much room is.

alert-noise, on-call-rotation and notification-test are the response group, and they check the path by which any of the above would reach a human out of hours. They go last, and they are the items most often skipped, which is precisely why a live test notification is on the list rather than an assumption that delivery still works.

Output

The review takes half an hour when nothing has moved. Anything flagged leaves the review as a ticket rather than a note; the point of a weekly cadence is that a slow trend gets caught while there is still time to order hardware.