Packaging, and where the packaging decides things live · PostgreSQL

Three PostgreSQL installations of the same version can differ so much in layout that a procedure written for one is useless on another. The binaries are identical; where the configuration lives, what starts the service, and whether a second cluster is a supported operation are all decisions the packaging made.

This matters because runbooks are written once and used on whichever host is having the incident. A step that says “edit /etc/postgresql/18/main/postgresql.conf” is wrong on Red Hat and wrong in a container, and at 03:00 that costs more than the five minutes it would have taken to write the step properly.

The three layouts

	Debian / Ubuntu (PGDG)	Red Hat / Rocky (PGDG)	Official container image
Binaries	`/usr/lib/postgresql/18/bin/`	`/usr/pgsql-18/bin/`	`/usr/lib/postgresql/18/bin/`
Data	`/var/lib/postgresql/18/main/`	`/var/lib/pgsql/18/data/`	`/var/lib/postgresql/18/docker/`
Configuration	`/etc/postgresql/18/main/`	inside the data directory	inside the data directory
Service	`postgresql@18-main.service`	`postgresql-18.service`	PID 1, no init system
Multi-cluster	first-class, via wrappers	manual	one per container
Cluster tooling	`pg_ctlcluster`, `pg_lsclusters`	none	present but unused

The single most consequential difference is the configuration location. Debian moves postgresql.conf, pg_hba.conf and pg_ident.conf out of the data directory into /etc, which is where a system administrator expects configuration to be and which makes the files easy to manage with configuration management. Red Hat leaves them in PGDATA, which is where PostgreSQL itself puts them and which keeps a cluster self-contained.

Neither is wrong. Both are load-bearing assumptions in a procedure.

Ask the server, never assume the path

Every layout question has an answer available from a running server, and the answer is correct regardless of packaging.

SHOW config_file;      -- the postgresql.conf actually in use
SHOW hba_file;         -- the pg_hba.conf actually in use
SHOW ident_file;
SHOW data_directory;

Writing a runbook step as “read the path from SHOW config_file” makes it portable across all three layouts and across a host where somebody moved things. Writing it as a literal path makes it correct on one kind of host until somebody changes the estate.

The Debian cluster model

Debian’s packaging treats “several clusters on one host” as normal rather than exceptional, and supplies tooling for it. A cluster has a version and a name, and the pair identifies it everywhere.

Read-only / Safecreating a genuine second cluster on a host that already has one— pg_createcluster allocates the next free port and a matching data directory. Captured from a live run.

$ pg_createcluster 18 reporting

selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
Ver Cluster   Port Status Owner    Data directory                   Log file
18  reporting 5433 down   postgres /var/lib/postgresql/18/reporting /var/log/postgresql/postgresql-18-reporting.log

Note what it did without being asked: allocated port 5433 because 5432 was taken, created a data directory named after the cluster, and set up a log file. That is the value the wrappers add — the bookkeeping that otherwise has to be done by hand and remembered.

Service impact possiblestarting the second cluster and confirming it is a separate server— Two clusters, two ports, two data directories, one host.

$ pg_ctlcluster 18 reporting start && pg_lsclusters

Ver Cluster   Port Status Owner    Data directory                   Log file
18  reporting 5433 online postgres /var/lib/postgresql/18/reporting /var/log/postgresql/postgresql-18-reporting.log

-- and each answers for itself
$ psql -p 5433 -tAc 'SHOW data_directory'
/var/lib/postgresql/18/reporting

$ psql -p 5432 -tAc 'SHOW data_directory'
/var/lib/postgresql/18/docker

Two servers, sharing nothing but the kernel and the disk. This is the mechanism behind the isolation discussion in the first lesson of this course: separate clusters are how you get independent restart windows and independent restore points, and on Debian creating one is a single command.

The wrappers to know:

Command	Does
`pg_lsclusters`	List every registered cluster with version, port, status, data directory
`pg_createcluster 18 name`	Create a new cluster, allocating a free port
`pg_ctlcluster 18 name start\|stop\|restart\|reload`	Control one cluster
`pg_dropcluster 18 name`	Remove a cluster and its data
`pg_upgradecluster 17 main`	Major-version upgrade of one cluster (Part XVII)

The container image is neither layout

The official postgres image is built from the Debian packages, so the binaries live in /usr/lib/postgresql/18/bin/ and the Debian cluster wrappers are present on the filesystem. But the image does not register its cluster with them.

Read-only / Safepg_lsclusters inside the official image, with PostgreSQL running— A header and no rows. The running cluster is invisible to the tooling that would normally manage it.

$ docker exec rbpg-multi pg_lsclusters

Ver Cluster Port Status Owner Data directory Log file

PostgreSQL is running, serving on 5432, and pg_lsclusters reports nothing. The reason is that the wrappers enumerate clusters by reading /etc/postgresql/<version>/<name>/, and the image keeps its configuration inside PGDATA in the Red Hat style instead. There is no registration to find.

This is worth knowing for a practical reason: a monitoring check or an Ansible role that uses pg_lsclusters to discover clusters silently finds nothing in a container, and “no clusters” is indistinguishable from “no PostgreSQL” in most such scripts.

The image also runs PostgreSQL as PID 1 with no init system, which changes the shutdown story. Stopping the container sends a signal to PostgreSQL directly, and which signal it is decides whether you get a clean shutdown or a crash recovery on next start. Part XII covers the shutdown modes and Part XVIII covers the container-specific consequences.

Distribution packages or PGDG

Most distributions ship PostgreSQL in their own repositories, and the PostgreSQL project maintains its own APT and YUM repositories carrying every supported major version.

Distribution packages give you the version the distribution chose, supported for the distribution’s lifetime, integrated with its security updates. For Debian and Ubuntu that is usually one major version per release, and it may be several years old by the end of the release’s life.

PGDG packages give you every supported major version, packaged consistently, available on the day upstream releases. Several majors can be installed side by side, which is what makes pg_upgrade practical.

The criteria that actually decide it:

Do you need a major version the distribution does not ship? If yes, PGDG, and the question is settled.
Do you need two majors installed at once? pg_upgrade requires the binaries of both the old and the new version present simultaneously. Distribution repositories rarely allow this; PGDG is designed for it. This is the reason most production estates end up on PGDG regardless of where they started.
Does your patching process consume one repository or several? Adding PGDG means one more source to track, sign-verify and pin.
Is the host managed by a policy that forbids third-party repositories? Then the choice is made for you, and the version ceiling is a constraint to plan the upgrade calendar around.

Mixing distribution and PGDG packages for the same major version

Both repositories package the same upstream software, with different build options, different default paths in some cases, and different package names for the extension packages. Installing the server from one and an extension from the other produces a combination nobody tested.

The failure is not usually at install time. It appears when the extension’s shared library does not match the server’s build, and the symptom is a server that starts and then fails when the extension is first used, or a pg_upgrade that stops because it cannot find a matching extension for the target version.

Pin deliberately. On Debian and Ubuntu that means an APT preferences entry that keeps PostgreSQL packages coming from one origin, so that a routine apt upgrade cannot quietly move a package across the boundary.

How the server is told where its configuration lives, and what that costs you

Moving the configuration out of PGDATA creates a problem: PostgreSQL looks for postgresql.conf in its data directory by default, and a great deal of tooling assumes the same.

Debian solves it by telling the server explicitly. The service unit passes -c config_file=/etc/postgresql/18/main/postgresql.conf, so the server is instructed where to look rather than guessing. pg_ctlcluster and pg_createcluster wrap the same convention.

What it does not do is put the configuration back into PGDATA in any form. The files live only under /etc/postgresql/<version>/<cluster>/ and there is nothing in the data directory standing in for them — no copies and no links.

That is exactly why a physical backup of PGDATA on such a host contains no configuration at all. Not a broken link to it: nothing. Restore that archive onto a fresh host and you have a cluster with no postgresql.conf, no pg_hba.conf and no pg_ident.conf, which is a problem discovered at the least convenient moment. Lesson XIII-08 lists this among the failures that only a restore test finds.

The habit that protects you on any distribution is to stop assuming and ask the server:

SELECT name, setting, source FROM pg_settings
 WHERE name IN ('config_file','hba_file','ident_file','data_directory');

Those four report override as their source, because they are established before configuration is read rather than by it. Whatever the packaging did, that query tells you the truth, and it is the right first step on a host whose layout you did not build.

Identifying an unfamiliar host

# Which layout am I on? The config file location answers it.
psql -U postgres -tAc 'SHOW config_file'
psql -U postgres -tAc 'SHOW hba_file'
psql -U postgres -tAc 'SHOW data_directory'

# Which PostgreSQL versions have binaries installed?
ls -d /usr/lib/postgresql/*/bin /usr/pgsql-*/bin 2>/dev/null || true

# Debian: are there registered clusters?
command -v pg_lsclusters >/dev/null 2>&1 && pg_lsclusters || echo "no debian cluster tooling"

# Which units exist, and which are running?
systemctl list-units --type=service 'postgres*' --all 2>/dev/null || true

A config file under /etc/postgresql/ means Debian packaging. A config file inside the data directory means Red Hat packaging or a container. The presence of several directories under /usr/lib/postgresql/ means several majors are installed, which is either an upgrade in progress or an upgrade that was never finished.

Production discipline

Read paths from the server, not from memory. SHOW config_file, SHOW hba_file and SHOW data_directory are correct on every layout.
Write runbooks with the discovery step included. A literal path is a portability bug that surfaces during an incident on the one host that differs.
Choose PGDG when you need side-by-side majors. pg_upgrade needs both versions’ binaries present, and that requirement decides most estates.
Pin the repository origin for PostgreSQL packages. Mixing distribution and PGDG builds produces extension mismatches that appear at use time, not install time.
Treat pg_dropcluster as a destructive command. It removes the data directory, and the only thing distinguishing the right cluster from the wrong one is a word you type.
Remember that on Debian, PGDATA is not the whole cluster. The configuration lives in /etc and a naive archive of PGDATA contains no configuration at all.

Cross-course references

Linux for Production Sysadmins — Part XI (Packages) and Part XII (Repository security) cover repository pinning and signature verification, which is the mechanism behind the mixing warning above.
Ansible for Production Sysadmins — Part XXXIX (Platforms) covers writing roles that work across distributions, which is the automated form of the discovery step in this lesson.
Docker & Containers — Part IV (Images) covers what the official image is built from, and Part X (Production architecture) covers running a process as PID 1.