Maintenance windows, freezes and the observation period

A maintenance window is a promise in two directions. To the business it says: expect disruption between these times and not outside them. To the operator it says: you have this long, and what you have not finished by the end gets rolled back.

Both halves are load-bearing. A window with no end time is not a window, it is a Saturday.

Sizing the window

Sum the parts, with measured numbers, then add a margin for the thing you have not thought of.

Pre-checks and capture                        15 min
Change:  12 hosts x 6 min                     72 min
Soak between cohorts: 3 gates x 20 min        60 min
Validation                                    30 min
                                            ---------
Expected                                     177 min
Rollback if needed: 12 x 9 min               108 min
Contingency (30% of expected)                 53 min
                                            ---------
Window required                              338 min  (5 h 40 m)

Three rules keep this honest.

Measure, do not estimate. The per-host durations come from the last execution or from staging. An estimate is always the best case, because that is the run the author remembers.

The rollback fits inside the window. Not after it. This is what produces the hard cut-off time from the previous lesson.

Contingency is not padding to be reclaimed. Finishing early is the correct outcome, and a team that treats early finishes as evidence the window was too long will get windows that are too short within two quarters.

Choosing when

“Overnight at the weekend” is a habit, not an analysis. Four factors decide, and they often point in different directions.

Load. Low load reduces blast radius and makes regressions easier to see against a quiet baseline. It also means low traffic, so a request-path regression may not appear until the morning - which argues for validating under load before the window closes, or for a window at the start of a low period rather than the middle of it.

Staffing. The people who can fix it must be awake and reachable. A 03:00 window staffed by one tired engineer, with the database expert asleep and unreachable, is a worse risk than a 19:00 window on a Tuesday even though more users are online.

Recovery time available. Ask what happens if the change fails and cannot be rolled back. A Friday-night window leaves a weekend of degraded service and a team recalled from time off. A Tuesday-morning window leaves a full working day, the whole team, and every vendor support line open.

Dependencies. Month-end processing, batch runs, backup schedules, someone else’s change window on a shared dependency, and the business events you do not hear about unless you ask.

Weigh those and the answer is frequently “Tuesday or Wednesday morning, early in the low period, with the full team available”, not Friday night. The instinct to hide changes in the quietest hour optimises for the case where nothing goes wrong, which is the case that needed no optimisation.

Defend the window from everything else

A window is only isolated if you make it so. The host has its own schedule and does not know about yours.

$ systemctl list-timers --all --no-pager | head -6; ls -1 /etc/cron.d/ /etc/cron.daily/ 2>/dev/null | head -8

NEXT                        LEFT     LAST                        PASSED  UNIT
Tue 2026-08-11 22:00:00 UTC 6h left  Mon 2026-08-10 22:00:00 UTC 18h ago backup.timer
Wed 2026-08-12 06:12:34 UTC 14h left Tue 2026-08-11 06:12:11 UTC 9h ago  apt-daily-upgrade.timer
Wed 2026-08-12 00:00:00 UTC 8h left  Tue 2026-08-11 00:00:00 UTC 15h ago logrotate.timer
/etc/cron.d/:
e2scrub_all
sysstat
/etc/cron.daily/:
logrotate

$ sudo systemctl mask --now apt-daily-upgrade.timer backup.timer; systemctl is-enabled apt-daily-upgrade.timer backup.timer

Created symlink /etc/systemd/system/apt-daily-upgrade.timer -> /dev/null
Created symlink /etc/systemd/system/backup.timer -> /dev/null
masked
masked

The same discipline applies to configuration management. An Ansible pull, a Puppet agent or a Salt highstate running mid-window will revert your change or fight it. Pause the agent for the window and, again, put resuming it in the procedure with a number next to it.

Change freezes

A freeze suspends non-emergency changes for a defined period: retail before a peak trading period, a financial close, a conference, a regulated release date.

Used deliberately, a freeze is a reasonable trade. It reduces the rate of self-inflicted incidents at exactly the time when an incident is most expensive, and it keeps the team available for whatever does happen.

Used carelessly, it makes things worse in three specific ways.

• Security patches accumulate, so the freeze buys availability risk with security risk - and the accumulation is invisible unless somebody is tracking it
• Changes queue up and land together the moment the freeze lifts, producing a single enormous window with entangled changes and no way to attribute a failure
• Skills and tooling decay: a team that has not performed a change in six weeks is slower and less certain when it has to perform one under pressure

A freeze that works has four properties: a defined start and end, a written exemption path for security and incident-driven changes, someone tracking what is accumulating, and a planned staged unfreeze rather than a single reopening.

The unfreeze is the dangerous moment, and it is the one nobody plans. Ordering the backlog by risk, spreading it over two weeks, and refusing to bundle unrelated changes into one window is what stops the freeze simply moving the incident to a predictable date.

The observation period

The window closes when the change is applied and validated. The change is not finished.

Validation is a point measurement: the service responds, the metrics look right, the logs are clean. Whole classes of failure cannot be seen at that point.

Set the observation period from the slowest relevant row, and state who is watching and what they are watching. Standard periods that work: one hour of active watching, twenty-four hours of heightened alerting, one week before the change is closed and the class is reconsidered.

$ systemctl show myapp.service -p MemoryCurrent -p NRestarts; date -u -d '+24 hours' +'observation ends %F %T UTC'

NRestarts=0
MemoryCurrent=412516352
observation ends 2026-08-12 15:47:11 UTC

Closing the change means saying so: the observation period completed, what was observed, and the class the change should carry next time. A change that goes well three times with a rehearsed rollback is a candidate for the standard list - which is how a change process gets faster without getting weaker.

Knowledge check