Shared memory and the buffer pool · PostgreSQL

Separate processes cannot share anything by accident. Every fact that more than one PostgreSQL process needs — which pages are cached, who holds which lock, where the write-ahead log has reached, whether transaction 4,812,993 committed — has to live in memory that all of them can reach. That is the shared memory segment, and it is created once, by the postmaster, before the first backend exists.

That ordering is the whole reason a large group of PostgreSQL parameters cannot be changed on a running server. It is not caution. It is that the memory was already allocated and handed to every child.

What is in there

Read-only / Safethe largest shared memory allocations, default cluster— 150 MB total on a cluster whose shared_buffers is 128 MB. The buffer pool dominates; everything else is coordination.

$ psql -U postgres -c 'SELECT name, pg_size_pretty(allocated_size) AS size FROM pg_shmem_allocations ORDER BY allocated_size DESC LIMIT 12'

         name         |  size
----------------------+---------
Buffer Blocks        | 128 MB
<anonymous>          | 4637 kB
XLOG Ctl             | 4110 kB
AioHandleIOV         | 2784 kB
AioHandle            | 1566 kB
AioHandleData        | 1392 kB
Buffer Descriptors   | 1024 kB
transaction          | 517 kB
Checkpointer Data    | 512 kB
Checkpoint BufferIds | 320 kB
RWConflictPool       | 319 kB
(12 rows)

Read that list as five functional groups.

The buffer pool. Buffer Blocks is the cache of 8 KiB table and index pages, and it is exactly shared_buffers. Buffer Descriptors is its index: one small record per buffer holding which relation and block the buffer contains, whether it has been modified since it was read, and how recently it was used. The descriptors are what make a lookup possible without scanning the pool.

Write-ahead log control. XLOG Ctl holds the WAL insertion state and the in-memory WAL buffers. Every backend that modifies data writes a record here before the change is considered durable, so this structure is contended by definition. Part XII covers it properly.

Transaction state. The transaction allocation holds recently consulted commit status, so that deciding whether a given transaction committed usually does not require reading a file. Part VII explains why that question is asked constantly.

Asynchronous I/O handles. AioHandle, AioHandleData and AioHandleIOV track I/O requests in flight. These are new in PostgreSQL 18 and are absent on earlier releases; if you compare pg_shmem_allocations between a 17 and an 18 cluster, this is the difference you will notice first.

Coordination. The checkpointer’s work list, the lock table, the predicate-lock conflict pool, and the statistics that used to live in a separate collector process before PostgreSQL 15 moved them here.

The total, 150 MB, is worth noticing against shared_buffers of 128 MB. The overhead is real but modest, and it does not scale with connections in this segment — the per-connection memory is private and lives outside it entirely.

Why the sizes are fixed until restart

Ask the server which parameters can change and when.

Read-only / Safethe context column is the operational answer— postmaster means restart. There is no reload that will apply these.

$ psql -U postgres -c "SELECT name, setting, unit, context FROM pg_settings WHERE name IN ('shared_buffers','max_connections','max_locks_per_transaction','wal_buffers','max_wal_senders') ORDER BY name"

           name            | setting | unit |  context
---------------------------+---------+------+------------
max_connections           | 100     |      | postmaster
max_locks_per_transaction | 64      |      | postmaster
max_wal_senders           | 10      |      | postmaster
shared_buffers            | 16384   | 8kB  | postmaster
wal_buffers               | 512     | 8kB  | postmaster
(5 rows)

Every one of those parameters sizes a structure inside the shared segment, and every one is postmaster context. The connection between the two is not a coincidence: max_connections and max_locks_per_transaction together determine how large the lock table must be, so changing either means reallocating shared memory, so both require a restart.

Note also the unit column. shared_buffers reports 16384 with a unit of 8kB, which is 128 MB. Reading the raw number as megabytes is a classic and expensive misreading. Ask for it in human terms instead:

psql -U postgres -c 'SHOW shared_buffers'
psql -U postgres -c "SELECT pg_size_pretty(setting::bigint * 8192) AS shared_buffers
                       FROM pg_settings WHERE name = 'shared_buffers'"

How a page reaches a backend

The buffer pool is not a cache in front of the database. It is the only way data is read or written. No backend reads a table file directly into its own memory.

flowchart LR
    Q["Backend needs\nblock 42 of orders"] --> L["Look up in\nbuffer descriptors"]
    L -- "present" --> H["Hit: pin the buffer\nand read it"]
    L -- "absent" --> V["Choose a victim buffer\nby clock sweep"]
    V --> D{"Victim dirty?"}
    D -- "yes" --> W["Write victim out\nfirst"]
    D -- "no" --> R["Read block 42\nfrom disk"]
    W --> R
    R --> H
    H --> U["Unpin when done"]

Three properties of that path have operational consequences.

A read can cause a write. If the pool is full and the chosen victim buffer has been modified, it must be written before its slot can be reused. A workload that is nominally read-only can therefore generate write I/O, and a backend can find itself waiting on that write. This is one reason the background writer exists — to keep clean buffers available so that backends rarely have to do this themselves.

A modified page is not written when you commit. COMMIT makes the WAL record durable, not the data page. The page stays dirty in the pool and is written later by the checkpointer or the background writer. Part XII is built around this separation, and it is the single most important thing to understand about PostgreSQL durability.

Eviction is approximate. PostgreSQL uses a clock-sweep algorithm with a usage counter rather than strict least-recently-used ordering, because maintaining true LRU across many concurrent processes would require a contended global structure. The effect is that the pool retains frequently used pages well without needing a lock on every access.

Reading the hit ratio, and what it does not tell you

The buffer pool records hits and reads, so a hit ratio is easy to compute. Interpreting it is where people go wrong.

Read-only / Safebuffer pool activity by backend type— hits are pages found in shared_buffers; reads are pages that were not there.

$ psql -U postgres -c 'SELECT backend_type, object, context, reads, writes, extends, hits FROM pg_stat_io WHERE reads > 0 OR writes > 0 ORDER BY backend_type'

    backend_type     |  object  | context  | reads | writes | extends | hits
---------------------+----------+----------+-------+--------+---------+-------
autovacuum launcher | relation | normal   |     1 |      0 |         |     0
autovacuum worker   | wal      | normal   |       |     11 |         |
autovacuum worker   | relation | normal   |   137 |      0 |       9 | 11248
autovacuum worker   | relation | vacuum   |    40 |      0 |       0 |   126
checkpointer        | relation | normal   |       |    389 |         |
checkpointer        | wal      | normal   |     0 |      2 |         |
client backend      | relation | normal   |   334 |      0 |       0 | 15381
client backend      | relation | bulkread |   215 |      0 |         |   345
client backend      | wal      | normal   |     0 |     38 |         |
standalone backend  | relation | normal   |   479 |   1038 |     598 | 92643
(10 rows)

That breakdown is far more useful than a single number, and it is worth dwelling on the context column. Client backend reads split into normal and bulkread: PostgreSQL deliberately uses a small ring buffer for large sequential scans so that one big scan cannot evict the entire working set. A low hit ratio in bulkread context is the system working as designed, not a cache that is too small.

The critical limitation is this: a “read” in pg_stat_io means the page was not in shared_buffers. It does not mean the page came from a disk. PostgreSQL reads through the operating system, so a miss in the buffer pool is very often satisfied from the Linux page cache at memory speed. From inside PostgreSQL those two cases are indistinguishable.

Raising shared_buffers because the hit ratio looks low

The reasoning seems sound — misses are expensive, so cache more — and it is wrong often enough to be worth naming.

PostgreSQL sits on top of the OS page cache, so the same page can be cached twice, once in shared_buffers and once by the kernel. Memory given to shared_buffers is memory taken away from the page cache, which serves everything the buffer pool misses. Past a certain point you are moving memory from one cache to another and paying for the duplication.

A low hit ratio is a symptom with several causes, most of which are not solved by a bigger pool: a sequential scan reading a table that would never fit anyway, a bulkread ring working correctly, a missing index turning a lookup into a scan, or simply a working set larger than the host’s memory. The diagnostic order is to find the query, read its plan, and check whether it is reading more data than it needs — Part X covers exactly that — before moving memory around.

There is a genuine case for raising shared_buffers, and Part XI covers how to establish it from measurement rather than from a percentage rule. The point here is that a hit ratio alone does not establish it.

Pins, the clock sweep, and what a buffer descriptor holds

Each buffer has a descriptor holding the tag that identifies it — the relation, the fork, and the block number — plus state bits and two counters that decide its fate.

The pin count records how many backends are currently using the buffer. A backend pins a buffer before reading it and unpins when finished. A pinned buffer cannot be evicted, which is what makes it safe for a backend to hold a pointer into shared memory while it works. A buffer whose pin count never returns to zero is a bug, and it presents as the pool gradually running out of usable buffers.

The usage count is the replacement policy. It rises when a buffer is accessed, up to a small ceiling. The clock sweep walks the buffers in order looking for a victim: if it finds a buffer with a usage count above zero it decrements the count and moves on; if it finds one at zero and unpinned, that buffer is evicted. Frequently used pages get their counter refreshed before the hand comes round again, so they survive; pages touched once decay quickly. This approximates least-recently-used behaviour without any global ordering to contend on, which is the property that matters when hundreds of processes are searching at once.

The dirty bit records whether the page has been modified since it was read. It is the flag that makes the read-causes-a-write path above possible, and it is what the checkpointer scans for when it needs to write everything out.

Production discipline

Read context before planning any memory change. postmaster means a restart, and a restart means a change window for every database in the cluster.
Convert units before quoting a value. shared_buffers in pg_settings is a count of 8 KiB blocks. SHOW gives the human-readable form.
Batch restart-only changes into one window. They interact, and discovering the second one afterwards costs a second outage.
Break the hit ratio down by backend_type and context before drawing conclusions. A low ratio under bulkread is the ring buffer doing its job.
Never treat a buffer-pool miss as a disk read. The page cache sits underneath, and confusing the two leads directly to moving memory in the wrong direction.

Cross-course references

Linux for Production Sysadmins — Part XL (Memory performance) covers the page cache that sits beneath the buffer pool, and Part XXXVII (Resources) covers the shared-memory limits a large shared_buffers can meet.
Observability for Production Sysadmins — Part XV (Histograms) covers representing I/O latency distributions properly, which is the right shape for the timings pg_stat_io exposes.
Ceph & Distributed Storage — Part LXVIII (OSD latency) covers what a read actually costs when the storage is distributed rather than local.