medium riskservice affecting~45 min

Runbook: Perform a Minor Version Upgrade

1 · Prerequisites

Confirm every item is in place before any state change.

The target minor version and its release notes, read rather than skimmed
A change window sized for a restart of every node in the estate, in the correct order
Package repository access on every host, and confirmation the target version is available there
A verified backup taken before the window
The estate topology: which node is primary, which are standbys, and whether any is synchronous
Knowledge of whether the application tolerates a brief write interruption or requires a switchover

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Read the release notes for every version between the current one and the target. Minor releases occasionally require an action beyond the restart — a REINDEX after an index-related fix, or an ANALYZE after a planner change. That instruction is in the notes and nowhere else.
· Confirm the current version on every node. SELECT version(); and the package version from the host. An estate where the nodes are already on different minor versions has a different problem to solve first.
· Confirm the target version is available in the repository on every host. A partial estate upgrade is worse than none, because it produces an estate nobody can reason about.
· Confirm which node is the primary. SELECT pg_is_in_recovery(); on each. Do not rely on a hostname or on a document.
· Check for synchronous standbys. SHOW synchronous_standby_names; on the primary. Restarting a synchronous standby without removing it from the set first stops every write on the primary for the duration of the restart.
· Confirm replication is healthy before starting. SELECT application_name, state, pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn) FROM pg_stat_replication; A standby that is behind will be further behind after a restart.
· Take and verify a backup. A minor upgrade rarely goes wrong; when it does, this is the only path back.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Upgrade the standbys first, one at a time. A standby running a newer minor version than its primary is supported and is the safe direction. The reverse — a primary newer than its standbys — is not guaranteed.
2If a standby is synchronous, remove it from the set before restarting it. ALTER SYSTEM SET synchronous_standby_names = '<remaining>'; SELECT pg_reload_conf(); on the primary. This is reloadable and it takes seconds; without it, every commit on the primary hangs for the duration of the standby restart.
3Update the package on the standby. The exact command is distribution-specific; the important property is that it replaces the binaries without touching the data directory.
4Restart the standby. pg_ctl -D <datadir> -m fast restart, or the service manager's restart. A minor upgrade requires a restart, not merely a reload — the binaries have changed.
5Confirm the standby came back and reconnected. SELECT version(); on the standby, then SELECT application_name, state FROM pg_stat_replication; on the primary. Wait for streaming and for the byte lag to fall before moving to the next node.
6Restore the synchronous set if you changed it, and confirm sync_state is what you expect.
7Repeat for each remaining standby, one at a time. Never restart two standbys simultaneously in an estate that depends on one being available.
8Decide how the primary is upgraded. Two options: restart it in place, accepting a brief write interruption; or perform a switchover to an already-upgraded standby, upgrade the old primary, and switch back if you want to. The second is longer and has a shorter write interruption.
9For a restart in place, announce it and time it. Update the package, pg_ctl -m fast restart, and confirm. A fast shutdown disconnects clients and rolls back open transactions; never use -m immediate, which skips the shutdown checkpoint and forces crash recovery.
10Confirm the primary came back. SELECT version(), pg_postmaster_start_time(), pg_is_in_recovery(); and confirm every standby reconnected.
11Perform any action the release notes require. A REINDEX on affected index types, or an ANALYZE, or nothing. This step exists because it is the one people skip, and its absence produces a subtle problem months later.
12Confirm the application is healthy from its own monitoring, and record the version, the duration of each restart, and any release-note action taken.

4 · Verification

Confirm the procedure actually fixed the problem.

✓SELECT version(); on every node reports the target version, checked individually rather than inferred from one.
✓The package version on every host matches, confirmed from the host rather than from the database.
✓Every standby appears in pg_stat_replication on the primary with state = 'streaming' and a small byte lag.
✓A row written on the primary appears on every standby within seconds.
✓synchronous_standby_names is back to its intended value and sync_state matches it.
✓Any action required by the release notes has been performed and recorded — or the notes have been confirmed to require none, which is also worth recording.
✓Application error rates and latency are normal, from the application's monitoring.
✓The duration of each restart is recorded, so the next window can be sized from evidence.

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶Downgrade the package and restart. A minor version is data-directory compatible in both directions within the same major version, which is what makes this the least risky upgrade PostgreSQL offers.
↶That said, downgrading is rarely the right response. If a problem appears after a minor upgrade, read the release notes for the version you moved to before assuming the upgrade caused it.
↶If a standby fails to start after the upgrade, check its log before doing anything else. A standby that cannot start is not an emergency provided the primary and another standby are healthy.
↶If the primary fails to start, this is an outage. Read the log, and if the cause is not immediately clear, promote an already-upgraded standby rather than debugging under pressure.
↶If synchronous_standby_names was changed for the window and not restored, the cluster is running without the durability guarantee it was built for. Nothing will remind you; put it in the change's completion checklist.
↶If the estate is left on mixed minor versions because one host failed to upgrade, record it explicitly and schedule the remainder. A partially upgraded estate is a real state and should not be an accidental one.

6 · Escalation

When the runbook isn't enough, contact:

· The release notes require an action you cannot perform in the window — a REINDEX on a large table, for example: escalate to the change owner and schedule it separately. Skipping it silently is the worst option.
· The target version is not available on one host: escalate to whoever owns the package repositories. Proceeding with a partial estate upgrade should be a decision, not a default.
· A standby does not reconnect after its restart: escalate before restarting anything else. Continuing through the estate while one node is down removes the redundancy the sequence depends on.
· The primary does not come back: escalate immediately as an outage and consider promoting an upgraded standby.
· The upgrade addresses a security advisory with a deadline: escalate to agree the window rather than deferring quietly. The deadline and the availability requirement are both somebody else's to reconcile.
· Performance changes materially after the upgrade: escalate to the application owner with before-and-after plans. Minor releases can change planner behaviour in narrow ways, and the release notes usually say when.

A minor upgrade is a package update and a restart. The data directory format does not change, which is what makes it the least risky upgrade PostgreSQL offers — and reversible, which no major upgrade is.

The work is in the ordering and in one step people skip.

Standbys first, one at a time

Remove a synchronous standby from the set before restarting it

-- on the primary, before restarting standby-02
ALTER SYSTEM SET synchronous_standby_names = 'ANY 1 (sync1)';
SELECT pg_reload_conf();

The step people skip

Blast radius

Action	Reversible?	What it costs if wrong
Package update	Yes, downgrade	Nothing until the restart
Standby restart	Yes	That standby’s availability, briefly
Restarting a synchronous standby without removing it first	Yes, with a reload	Every write on the primary, for the duration
Primary restart (`-m fast`)	Yes	A brief write interruption; open transactions roll back
Primary restart (`-m immediate`)	Recovery required	No shutdown checkpoint; crash recovery on start
Skipping a release-note action	Deferred	A subtle problem nobody attributes to this change

Two ways to handle the primary

Restart in place. Simplest. A brief write interruption of a few seconds plus recovery time. Right for most estates.

Switch over to an upgraded standby, then upgrade the old primary. Longer overall, shorter write interruption, and it exercises the switchover procedure — which is worth something on its own. Right when the write interruption is the binding constraint.

Decide before the window, not during it.

Confirm every node individually

SELECT version();                          -- on each node
SELECT pg_is_in_recovery();                -- confirm the topology
SELECT application_name, state,
       pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn) AS bytes_behind
FROM pg_stat_replication;                  -- on the primary

And write a row on the primary, then read it on each standby. Views agreeing with each other is not the same as the path working.