Maintenance windows — the contract between uptime and change

A maintenance window is the contract between runtime change and operational uptime. It is not a calendar slot. It is a typed agreement: what will change, who owns it, when it begins, what must be true before it is “go,” what must be true before it is “complete,” and what the rollback path is if it is not. A “window” with no rollback path is an outage in slow motion.

What a maintenance window is

A maintenance window is a change-controlled interval during which the cluster is intentionally exposed to a planned mutation. The mutation may be a node reboot, a kubelet upgrade, a CNI bump, a control-plane certificate rotation, an etcd compaction, or a workload rollout that crosses a dangerous threshold. The window exists to make the trade-off explicit: the operator accepts a bounded risk of disruption in exchange for an upgrade that is otherwise deferred or forbidden.

The window is not the change itself. The window is the scaffolding: the gate, the owner, the rollback, the notification, the freeze.

flowchart LR
    A[RFC drafted] --> B[Risk class assigned]
    B --> C[Owner assigned]
    C --> D[Rollback plan reviewed]
    D --> E[Window scheduled]
    E --> F[Pre-change snapshot]
    F --> G[Change executes]
    G --> H{Healthy?}
    H -->|Yes| I[Post-change validation]
    H -->|No| J[Rollback or fail]
    I --> K[Close window + PIR]
    J --> K

The window is the rectangle around F → G → H → I|J. Outside the rectangle, the cluster is not in a maintenance state.

What a window is not

A maintenance window is not a freeze. A freeze is a prohibition on change; a window is a controlled permission to change. Conflating the two produces the worst production failure: a “window” during which nothing was reviewed and everything was changed, with no rollback, no gate, and no owner.

A maintenance window is also not a “deploy slot.” A deploy slot is a recurring, low-risk path for ordinary rollouts. A maintenance window is reserved for changes that cannot be treated as ordinary — control-plane upgrades, kernel patches, storage-class migrations, and the like.

Window contents

A production-grade window contains:

• Change. A single, named change. If two changes need to ship, schedule two windows.
• Owner. A human whose pager lights up if the window fails.
• Risk class. Categorise the change (1 = no impact, 2 = bounded impact, 3 = cluster-wide) and require the corresponding review depth.
• Pre-change gate. What must be true before kubectl runs. Most commonly: a current etcd snapshot, a current control-plane backup, a verified drain plan, a tested rollback.
• Execution. The exact command sequence, in the order it will run, with the kill signals clearly marked.
• Validation. What must be true after the change. Specific checks, not “looks healthy.”
• Rollback. What runs if validation fails. Reversible changes get a kubectl rollout undo; irreversible changes get an etcd snapshot restore or a node rebuild.
• Notify. Who is told, when, and through what channel.
• Close. The window is closed only when validation passes and the close-out is signed.

A window with any of these missing is incomplete.

Rolling vs control-plane maintenance

The two windows that dominate a Kubernetes estate are:

1. Rolling worker maintenance. A node is drained, patched, rebooted, and rejoined. The cluster absorbs the drain because the workload has a PodDisruptionBudget and replica spread that allow it. The window is per-node; multiple windows may run in parallel across failure domains.
2. Control-plane maintenance. A control-plane component is upgraded, restarted, or reconfigured. The window is cluster-wide because the API server, the scheduler, the controller-manager, and etcd are the brains of the cluster. The window must be scheduled, executed, and validated with the same procedural discipline as a database upgrade.

The two need different windows. A worker drain that conflicts with a control-plane upgrade is a window that has not thought through the failure model.

gantt
    title "Maintenance window timeline"
    dateFormat HH:mm
    axisFormat %H:%M
    section Pre-change
    Snapshot etcd        :a1, 00:00, 15m
    Notify on-call       :a2, after a1, 5m
    section Change
    Drain node pool A    :b1, after a2, 30m
    Patch + reboot node  :b2, after b1, 20m
    Uncordon + verify    :b3, after b2, 15m
    section Post-change
    Validate PDB         :c1, after b3, 10m
    Sign off             :c2, after c1, 5m

The window’s anti-patterns

Windows that go wrong fall into a small number of patterns:

• Sliding window. A “window” that never closes because new changes were added to it.
• Bundled window. Two unrelated changes shipped together so one can blame the other when something fails.
• Owner-less window. A window scheduled by an automation, not owned by a human, with no pager.
• No-rollback window. A window that lists only the forward path.
• Sneak window. A change made outside the window because “it was small” — usually followed by an incident.

Each of these is preventable by the same discipline: write the window down, sign it, share it, and never make a non-emergency change outside it.

Production discipline

A maintenance window is the smallest unit of operational risk governance. A cluster with no windows is a cluster that waits until 03:00 for someone to do an urgent patch under pressure, with no rollback. A cluster with rolling, well-governed windows is a cluster where the cumulative risk is paid down in small pieces, during business hours, with eyes on the controls.

• Treat every window as a contract. Owner, gate, change, rollback, validation, close.
• Schedule windows, do not invent them. Last-minute “windows” are usually bypasses.
• Distinguish rolling worker maintenance from control-plane maintenance — both run in different windows, owned by different teams.
• Pair every forward path with a rollback path. The rollback is a procedure, not a hope.
• Close the window explicitly. A window that never closes is a window that has forgotten its purpose.

Quiz