Runbook: Deploy and Hand Over a Production Cluster

1 · Prerequisites

Confirm every item is in place before any state change.

• The host, its storage layout, and confirmation of how much of it is dedicated to this cluster
• The locale and encoding the applications require, because these are fixed at initdb time and changing them means a dump and reload
• The intended topology: standalone, primary with standbys, and whether any standby is synchronous
• Storage for backups and for the WAL archive, reachable from this host, with capacity for the retention you intend
• The application inventory: which databases, which roles, and what each needs to do
• A named owner for the cluster, and agreement on the recovery time and recovery point objectives

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

• · Confirm the storage layout before initdb. A separate filesystem for pg_wal means a full WAL volume stops only the cluster rather than everything sharing the disk, and it makes WAL growth a visible metric of its own. Adding it later means moving the directory.
• · Decide encoding and locale, and write down why. UTF8 with an appropriate collation is the usual answer. These are fixed at initdb time; changing them later requires a dump and reload of the whole cluster.
• · Confirm data checksums will be on. PostgreSQL 18 enables them by default. Without them a damaged page is returned as fact with no error and no log entry; checksum_failures cannot increment on a cluster that computes no checksums.
• · **Size shared_buffers and the memory budget as an arithmetic exercise.** shared_buffers plus max_connections x work_mem x memory_nodes_per_plan plus the operating system's needs, against physical RAM. If nobody can produce that number, the memory settings have been guessed.
• · Confirm the backup destination and the archive destination exist and are writable by the postgres user on this host, tested by hand.
• · Confirm the monitoring system can reach this host and that somebody has agreed to receive its alerts.
• · Agree the recovery objectives before building anything. They determine the base backup cadence, the archive retention and whether a standby is required, and retrofitting them is more expensive than designing for them.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1. 1Run initdb with the decisions made explicitly, rather than relying on defaults you have not read: initdb -D /var/lib/postgresql/18/main --encoding=UTF8 --locale=... --data-checksums --waldir=/var/lib/postgresql/18/wal
2. 2Confirm what initdb produced. pg_controldata -D <datadir> and check the encoding, the locale, and Data page checksum version. This is the last cheap moment to correct any of them.
3. 3Set the connection and authentication configuration before the cluster is reachable. listen_addresses, port, and a pg_hba.conf with no trust rule for any address a real client can reach. Loopback trust is present in most default installations and it makes every local test pass unconditionally.
4. 4Enable TLS before any client connects. Install the certificate material, set ssl = on, and use hostssl rather than host in pg_hba.conf. Retrofitting enforcement onto a live estate means cutting off whatever is not yet configured.
5. 5Set the durability and WAL configuration. wal_level = replica at minimum, archive_mode = on with a working archive_command, and a max_wal_size sized from the expected write rate rather than left at a default nobody chose.
6. 6Set the logging configuration so the cluster can be diagnosed. log_checkpoints, log_connections, log_disconnections, log_lock_waits, log_temp_files, log_autovacuum_min_duration, and a log_line_prefix that includes the timestamp, the process id, the user and the database.
7. 7**Set log_min_duration_statement deliberately.** Too low and the log is unusable; too high and slow queries are invisible. Choose a value from the application's expected latency.
8. 8**Install pg_stat_statements** and add it to shared_preload_libraries. It requires a restart, which is free now and a change request later.
9. 9Create the databases and roles, using group roles for privileges and login roles for clients, with ALTER DEFAULT PRIVILEGES set for the owning role so future objects inherit.
10. 10Take a base backup and restore it, on a different host, before the cluster carries any production data. The restore time you measure is the recovery time objective the estate actually has.
11. 11Build the standby, if the topology calls for one, and rehearse a promotion. A failover target that has never been promoted is a hypothesis.
12. 12Configure monitoring and test each alert by causing the condition: stop the standby, pause replay, fill a volume in a test, exhaust the connections. An alert that has never fired has never been shown to work.
13. 13Hand over with the evidence, not with a document. The restore that was performed, the failover that was rehearsed, the alerts that were tested, and the timings for each.

4 · Verification

Confirm the procedure actually fixed the problem.

• ✓pg_controldata reports the intended encoding, locale and Data page checksum version: 1.
• ✓SELECT name, setting FROM pg_settings WHERE source <> 'default' ORDER BY name; matches the intended configuration exactly, with nothing set that nobody chose.
• ✓SELECT rule_number, type, database, user_name, address, auth_method FROM pg_hba_file_rules ORDER BY rule_number; contains no trust rule for any address a real client can reach.
• ✓A verify-full TLS client connects by the name in the certificate, and a client using a different name is refused with a hostname mismatch.
• ✓A base backup has been taken and restored on another host, with the restore time recorded.
• ✓pg_stat_archiver shows successful archiving and failed_count at zero, confirmed by forcing a segment with pg_switch_wal() and checking it arrived.
• ✓If a standby exists, it has been promoted once in a rehearsal and rejoined, with each step timed.
• ✓Every alert has been fired once, deliberately, and the resulting page reached a human.
• ✓A named owner is recorded, along with the agreed recovery objectives and the measured numbers against them.

5 · Rollback

If verification fails, undo the procedure in reverse order.

• ↶Before the cluster carries data, rollback is to stop it and remove the data directory. Do that rather than reconfiguring around a decision that was wrong at initdb time.
• ↶Encoding, locale and the WAL directory location are effectively fixed once data exists. Correcting them means a dump and reload, so correct them now.
• ↶Data checksums can be enabled later with pg_checksums --enable on a stopped cluster — measured at 159 ms on a 43 MB database, scaling with size — so this one is recoverable, but it is free at initdb time.
• ↶If the cluster has been handed over and a configuration decision proves wrong, treat it as an ordinary configuration change with its own change note, rather than as a correction to this procedure.
• ↶If the handover was accepted without the restore, the failover rehearsal or the alert tests, record that explicitly as an open item with an owner. An undone verification that nobody wrote down becomes an assumption.
• ↶If the cluster is being abandoned before production, follow the decommission procedure rather than deleting it — the backup destination and the archive may already hold data.

6 · Escalation

When the runbook isn't enough, contact:

• · The recovery objectives cannot be met with the storage and topology available: escalate to the service owner with the measured restore time. The objective and the budget are both theirs to reconcile.
• · The application requires an encoding or collation that conflicts with the estate standard: escalate before initdb. This is the one decision that cannot be changed without a full dump and reload.
• · No backup destination is available: escalate and do not proceed to production. A cluster without a tested restore is not a production cluster regardless of what else is configured.
• · Nobody will accept ownership of the cluster: escalate to the platform owner. An unowned cluster accumulates every problem in this course and nobody is paged for any of it.
• · The memory arithmetic does not fit the host: escalate rather than setting values that work most of the time. work_mem is per operation, and the worst case is what the out-of-memory killer acts on.
• · A required extension is not packaged for this platform: escalate to whoever owns the platform before building. An extension installed from source on a production host is a maintenance liability with no owner.

References

1. PostgreSQL 18 documentation, Server Setup and Operation
2. PostgreSQL 18 documentation, initdb
3. PostgreSQL 18 documentation, Server Configuration
4. PostgreSQL 18 documentation, Data Checksums