Pre-change gates — what must be true before a window opens

A pre-change gate is the conditions that must be true before the window opens. The gate is the difference between a window you can run and a window you have to abort. Failing any one condition is sufficient reason to cancel; the gate is met as a whole, not in pieces.

The gate, ordered cheapest to most expensive

The gate must be ordered so the cheapest checks run first. The operator who opens the gate from the top of the list escalates each failure without paying the cost of the lower checks:

flowchart TD
    A[Gate start] --> B{Communication sent?}
    B -->|No| C[Cancel window]
    B -->|Yes| D{etcd snapshot current?}
    D -->|No| E[Take snapshot, verify restorable]
    D -->|Yes| F{Rollback tested in lab?}
    F -->|No| G[Run rollback in staging]
    F -->|Yes| H{Capacity headroom OK?}
    H -->|No| I[Add nodes or shift traffic]
    H -->|Yes| J{Drain plan rehearsed?}
    J -->|No| K[Rehearse in staging]
    J -->|Yes| L[Gate open]

The order is deliberate: communication is cheap; a rollback test in staging is expensive; a failure of any one is a stop sign.

Gate 1: communication sent

The change is communicated to all stakeholders — on-call, customer success, downstream consumers, and the security team — before the window opens. The communication includes: the change, the expected impact, the rollback path, the close-out time, and the escalation path if the change fails.

In production, the communication is the first gate because once it is sent, the operator has committed to running the change. Running the change without sending the communication is a sneak change; sending the communication and then cancelling is a normal “false-positive gate.”

Gate 2: etcd snapshot is current and restorable

The etcd snapshot is the cluster’s ultimate rollback. The snapshot must be:

• Recent. At most 24 hours old for a control-plane change; at most 1 hour for a production cluster with high write throughput.
• Verified. The snapshot must be loaded into a lab cluster and read back. A snapshot that fails to load is not a snapshot; it is a file.
• Off-cluster. The snapshot file lives on a host that is not the etcd node. A snapshot that dies with the etcd node is not a backup.

ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /var/backups/etcd-$(date -u +%Y%m%d-%H%M%S).db

The gate is not “we have a snapshot.” The gate is “we have a snapshot from the last hour, in a different failure domain, that we have verified by loading.”

Gate 3: rollback tested in the lab

The rollback procedure is run end-to-end in a staging cluster that mirrors production. The procedure is not a sentence; it is a sequence of commands. The lab test is what proves that the sequence works against a cluster with the same CNI, the same CSI, the same ingress, and the same data shape.

A rollback that has never been tested is a guess. The lab is what converts the guess into a procedure.

Gate 4: capacity headroom

The cluster has enough capacity to absorb the workloads that will be evicted by the change. For a node drain, this is the number of nodes in the pool minus the nodes that will be drained, scaled by the workload’s PDB. For a control-plane change, this is the remaining HA capacity (e.g., 2 of 3 control-plane nodes).

If the headroom is insufficient, the gate is not met. The fix is to add capacity before the window opens, not during.

Gate 5: drain plan rehearsed

The drain plan is rehearsed in staging. The rehearsal walks the exact command sequence that will run in production, including the eviction of the workload, the verification of the rescheduled replicas, and the validation of the post-drain state. A drain that has never been rehearsed is a drain that will discover the PodDisruptionBudget too late.

What happens when a gate fails

A gate failure is a stop sign. The window is cancelled and rescheduled. The failure is written down; the runbook is updated; the gate is added to the next window’s pre-flight.

The worst production failure is to “feel the gate” — to note that the snapshot is stale but proceed anyway because the change is “small.” The cluster’s discipline is to stop.

Production discipline

A pre-change gate is the contract that a window is ready to run. The gate is met as a whole: every condition is true; the cheapest checks run first; a failure cancels the window. A cluster that runs the gate well is a cluster that has the discipline to fail safely.

• Run the gate from cheapest to most expensive. The cheapest gate is communication; the most expensive is the drain rehearsal.
• Verify every snapshot by loading it into a lab cluster; a snapshot that has not been loaded is not a snapshot.
• Test the rollback in the lab before the window opens; a rollback that has never been run is a guess.
• Document gate failures. The next window’s pre-flight is built on the failures of the previous one.

Quiz