Skip to main content
RunBook Academy

PostgreSQLI · Architecture and the Process ModelArchitecture

A connection from TCP to first query

Intermediate⏱ ~25 minpsql

What you'll learn

  • Trace the six stages between connect() and the first query
  • Attribute a connection failure to a specific stage from its error text
  • Read the PostgreSQL 18 granular connection log and interpret its timings
  • Distinguish a network failure from a rejection from an authentication failure

Prerequisites

Verified against PostgreSQL 18.x · PostgreSQL (comparison targets) 17.11, 16.15 · PostgreSQL (support calendar) 18, 17, 16, 15, 14 supported · pgBackRest 2.59.1 · PgBouncer 1.25.2 · Patroni 4.1.5 · Ubuntu (host baseline) 26.04 LTS · 2026-08-27

Not yet marked complete on this device.

“The application cannot connect to the database” is one of the most common tickets an operator receives and one of the least informative. It covers a network path that never reached the host, a server that is not listening on the address tried, a rule that refused the client, a password that did not match, a database that does not exist, and a server that is running but has no connection slots left. These have nothing in common except the word connect.

The way to stop guessing is to know the sequence, because each stage fails in its own way and produces its own evidence.

The six stages

flowchart TD
    A["1. Client resolves the host\nand opens a TCP connection"] --> B["2. Postmaster accepts\non a listening socket"]
    B --> C["3. Optional TLS negotiation"]
    C --> D["4. pg_hba.conf is matched\nfirst rule wins"]
    D --> E["5. Authentication\nby the method that rule named"]
    E --> F["6. Backend is forked\nand the session initialised"]
    F --> G["ReadyForQuery"]

Stage 1 — the client gets to the host. DNS resolution, routing, firewalls. PostgreSQL is not involved and knows nothing about failures here. The evidence is a connection refused, a timeout, or a name that did not resolve, and it appears only on the client. Nothing is written to the PostgreSQL log, because nothing reached PostgreSQL.

Stage 2 — the postmaster accepts. The server must be listening on an address the client can reach. listen_addresses decides this and it is postmaster context, so a change needs a restart. The default in a package installation is frequently localhost, which is why a freshly installed server refuses remote connections while working perfectly from the host itself.

Stage 3 — TLS, if requested. The client asks whether the server supports TLS and the two negotiate before any credentials are sent. Part V covers this properly.

Stage 4 — pg_hba.conf is matched. The server walks the rules top to bottom and uses the first one whose connection type, database, user and address all match. That rule decides the authentication method. It is not a policy evaluation and later rules are not consulted, which is the single most misunderstood fact about PostgreSQL access control and has its own lesson in Part V.

Stage 5 — authentication. The method named by the matched rule is carried out. Failure here means the rule matched and the credential did not satisfy it.

Stage 6 — fork and session initialisation. The postmaster forks a backend. The backend attaches to shared memory, sets its database, and loads enough catalogue to be useful. Then it sends ReadyForQuery and the connection is usable.

Watching it happen

PostgreSQL 18 changed log_connections from a boolean into a setting that selects which stages to log, and all reports each one separately.

psql -U postgres -c "ALTER SYSTEM SET log_connections = 'all'"
psql -U postgres -c "ALTER SYSTEM SET log_disconnections = on"
psql -U postgres -c 'SELECT pg_reload_conf()'
Read-only / Safeone complete connection, PostgreSQL 18.6, log_connections = 'all'
$ docker logs rbpg-base18
2026-08-27 17:04:58.152 UTC [1] LOG:  received SIGHUP, reloading configuration files
2026-08-27 17:04:58.153 UTC [1] LOG:  parameter "log_connections" changed to "all"
2026-08-27 17:04:58.153 UTC [1] LOG:  parameter "log_disconnections" changed to "on"
2026-08-27 17:04:59.203 UTC [425] LOG:  connection received: host=[local]
2026-08-27 17:04:59.203 UTC [425] LOG:  connection authenticated: user="postgres" method=trust (/var/lib/postgresql/18/docker/pg_hba.conf:117)
2026-08-27 17:04:59.203 UTC [425] LOG:  connection authorized: user=postgres database=shop application_name=psql
2026-08-27 17:04:59.204 UTC [425] LOG:  connection ready: setup total=1.498 ms, fork=0.445 ms, authentication=0.127 ms
2026-08-27 17:04:59.204 UTC [425] LOG:  disconnection: session time: 0:00:00.001 user=postgres database=shop host=[local]

That sequence is worth reading line by line, because each line marks a stage boundary and the boundaries are exactly where connections fail.

connection received is stage 2 completing: a client reached the listening socket. If your log shows nothing at all for a failing client, the problem is upstream of PostgreSQL and no amount of pg_hba.conf editing will help.

connection authenticated is stages 4 and 5 completing, and it names the file and line number of the rule that matchedpg_hba.conf:117. This single detail resolves most access disputes outright. When somebody insists their rule should have matched, the log names the rule that actually did, and the answer is almost always that an earlier, broader rule matched first.

connection authorized is the database and user being accepted, with the application name the client supplied.

connection ready is stage 6, and PostgreSQL 18 breaks the timing into components: setup total=1.498 ms, fork=0.445 ms, authentication=0.127 ms. The fork cost from the process-model lesson is measured here rather than asserted — about half a millisecond on an idle local cluster. On a loaded host, or with an authentication method that consults an external directory, these numbers grow, and this line is how you find out which component grew.

Attributing a failure to a stage

This is the table that converts an error string into a stage, and therefore into a next action.

What the client reportsStageWhat it meansLook at
could not translate host name1DNSresolver, not the database
Connection refused1–2Nothing is listening on that address and portlisten_addresses, port, is the server running
Connection timed out1Packets are being droppedfirewall, routing, security group
no pg_hba.conf entry for host ...4Reached the server; no rule matchedthe rule set, and the address it saw
password authentication failed for user5A rule matched and the credential failedthe credential, and which rule matched
database "x" does not exist6Authenticated successfully; the target does not existpg_database
sorry, too many clients already6Server reached max_connectionspg_stat_activity
the database system is starting up6Recovery in progressthe log, and how long recovery has left

The most valuable distinction in that table is between the fourth and fifth rows, because they look similar and mean opposite things.

no pg_hba.conf entry means the connection reached the server and no rule matched it. The server tells you exactly what it was matching on — user, database, address, and whether TLS was in use — and the fix is a rule, not a credential.

password authentication failed means a rule did match and the credential did not satisfy it. Adding rules will not help. Note also that PostgreSQL deliberately returns this same message whether the role does not exist or the password was wrong, so that the error cannot be used to enumerate valid usernames.

Diagnosing from both ends

A connection problem is one of the few cases where you genuinely need evidence from the client side and the server side, because the whole question is where along the path it stopped.

# From the client host: does the TCP path exist at all?
# Replace with the real host and port for your estate.
PGHOST=db01.internal
PGPORT=5432
timeout 5 bash -c "cat < /dev/null > /dev/tcp/$PGHOST/$PGPORT" \
  && echo "tcp reachable" || echo "tcp NOT reachable"

# What does the server think it is listening on?
psql -U postgres -c 'SHOW listen_addresses' -c 'SHOW port'

# From the server host: is it actually bound where it claims?
ss -ltnp 2>/dev/null | grep -E '5432|postgres' || true

# Is the server accepting connections at all, independent of credentials?
pg_isready -h "$PGHOST" -p "$PGPORT"

pg_isready is underused. It reports whether the server is accepting connections without needing valid credentials, so it separates “the server is not there” from “the server is there and rejected me” in one command, which is precisely the stage-2-versus-stage-4 distinction.

Production discipline

  1. Establish which stage failed before proposing a fix. The error text names the stage. Fixing stage 4 when the failure is at stage 1 is the commonest wasted hour in this class of incident.
  2. Enable log_connections before you need it. It is a reload, and the rule file and line number it records settle access disputes that otherwise become arguments.
  3. Use pg_isready to separate reachability from rejection. It answers the question without credentials.
  4. Never add a broad pg_hba.conf rule as a diagnostic. First match wins, so a broad rule near the top changes authentication for every connection it covers, and it will not fail afterwards to remind you.
  5. Read no pg_hba.conf entry and password authentication failed as opposites. The first needs a rule; the second needs a credential and already has a rule.

Cross-course references

  • Linux for Production Sysadmins — Part XXII (Network troubleshooting) covers ss, packet capture and the reachability questions that stage 1 raises, and Part XXV (Firewall) covers the rules that silently drop stage-1 traffic.
  • Secrets, PKI & Certificate Management — Part VII (TLS for operators) covers the negotiation at stage 3 and what each sslmode actually verifies.
  • Observability for Production Sysadmins — Part XI (Blackbox) covers probing a listener from outside, which is the automated form of the pg_isready check.

Quiz

Knowledge check · 6 questions

  1. Q1. An application reports 'no pg_hba.conf entry for host 10.2.4.19'. What does this establish?

  2. Q2. A freshly installed PostgreSQL server works from psql on the host itself but every remote client reports 'Connection refused'. The server log contains no entry for any of the failed attempts. Which parameter should you check first?

  3. Q3. PostgreSQL evaluates all matching pg_hba.conf rules and grants access if any of them would permit the connection.

  4. Q4. PostgreSQL returns the same authentication failure message whether the role does not exist or the password was wrong.

  5. Q5. Name the command that distinguishes 'the server is not reachable' from 'the server rejected me', and explain why it works without credentials.

  6. Q6. Work the evidence, state which stage failed, and give the change you would make.

    A batch job that has run nightly for two years began failing at 02:10 with 'password authentication failed for user batch_loader'. The password in the job's configuration has not changed and is confirmed correct against a copy in the secret manager. The PostgreSQL log shows, for each failed attempt, a connection received line followed immediately by the authentication failure, with no connection authenticated line. Earlier that evening a change was deployed that added a new monitoring role and, in the same change, inserted a pg_hba.conf line above the existing entries to allow the monitoring host. Other applications are unaffected.

Passing score: 75%. Answers are checked in this browser.