PostgreSQL on Kubernetes: operators, StatefulSets, storage, fencing · PostgreSQL

What Kubernetes supplies

StatefulSets give stable network identity and stable per-pod storage, which is the minimum a database needs and is why Deployment is wrong for one.

PersistentVolumeClaims bind storage to a pod identity rather than to a node.

Services give a stable name in front of changing pods, which is lesson XV-06’s routing problem with an implementation.

Scheduling places pods on nodes and moves them when nodes fail.

Where it conflicts

Kubernetes was built for stateless workloads, and its instincts are wrong for a database in specific ways.

It restarts things. A pod that exits is restarted. Lesson XVII-02 measured a cluster whose pg_wal filled: it PANICs, recovery fails for the same reason, and it shuts down. Kubernetes restarts it into the same condition, in a loop, and the log line that explains it scrolls past between attempts.

It reschedules things. Moving a database pod to another node means detaching and attaching storage, which takes time and can fail — and if it partially succeeds you have the fencing problem from lesson XV-03.

It treats pods as fungible. They are not. One of them is the primary, and which one matters more than anything else about the system.

Liveness probes assume a shallow question. pg_isready returns success for a cluster in recovery, a cluster refusing writes, and a cluster serving from the corrupted page in lesson XVIII-06. A failing liveness probe restarts the pod, which for a database is rarely the right response and is sometimes the worst one.

What an operator must do

Exactly lesson XV-04’s six responsibilities, plus the Kubernetes-specific ones:

Responsibility	On Kubernetes
Detect	Deep health checks, not `pg_isready`
Decide	Consensus — often the Kubernetes API itself
Fence	Ensure a demoted pod cannot come back as primary
Promote	`pg_promote()`
Reroute	Update the Service’s endpoints
Rejoin	`pg_rewind` or rebuild
Storage	Correct PVC handling across rescheduling
Backups	Part XIII, which Kubernetes supplies nothing for

Evaluating an operator means asking the lesson XV-04 questions, plus:

What happens when the Kubernetes API is unavailable? If it is the consensus store, the operator cannot decide — and the right behaviour is to keep serving and stop deciding.
What happens when a node is partitioned rather than dead? The pod may still be running and serving.
How is the primary’s identity expressed, and does the Service follow a promotion atomically?
What does it do about the kubelet’s restart policy during a deliberate demotion?

Storage

The database’s durability rests entirely on the storage layer, and this is where lesson VI-01’s measurement matters most: an 854 µs fdatasync predicted a ~1,170 tps commit ceiling, and pgbench measured 842.

Questions worth answering before running a production database on a storage class:

Does it honour fsync? If not, nothing above it is durable, and lesson VI-01’s whole chain is broken.
What is its fsync latency? That is your commit ceiling.
Is it node-local or network-attached? Local is faster and does not move with the pod.
What happens during a node failure? Detach and reattach takes time, and that time is part of your RTO.
Can it be resized? Lesson XVII-02’s remedy for a full filesystem is “extend it”.

When Kubernetes is and is not the right answer

Reasonable when: you already run everything on Kubernetes and have the operational maturity for it; you have many similar clusters where an operator’s automation pays for itself; the databases are small enough that rescheduling is cheap; and your team can debug the operator as well as PostgreSQL.

Questionable when: this is your only database and the Kubernetes expertise is thinner than the PostgreSQL expertise; the database is large enough that moving it is a major event; you need the diagnostic access of lesson XVI-06 routinely; or the answer to “who debugs the operator at 3 a.m.” is nobody.

The honest framing is that Kubernetes adds a distributed system between you and your database. That is worth it when it is automating something you would otherwise do many times, and it is a poor trade for a single important cluster.

The two-primaries path, step by step

Worth walking through precisely, because it is the failure that matters and it is not obvious.

A network partition separates node A, running the primary, from the rest of the cluster.
The operator, running elsewhere, cannot reach node A. It cannot distinguish a dead node from an unreachable one — lesson XV-01’s fundamental problem, unchanged by Kubernetes.
The Kubernetes control plane also cannot reach node A, and after the node-monitor grace period marks the node NotReady, then eventually evicts its pods from the API’s point of view.
The pod on node A is still running. Eviction is a control-plane action; a partitioned kubelet does not receive it. PostgreSQL on node A is up, and if any clients are on A’s side of the partition, it is serving writes.
The operator promotes a replica. Now there are two primaries.
The partition heals. Both have written on their own timelines, and lesson XV-03’s arithmetic applies: one side’s committed, acknowledged writes are discarded.

The defence is the same as everywhere else and has to be built rather than assumed: the primary must fence itself. A leader lease with a TTL, held in a consensus store, where the pod demotes itself when it cannot renew — lesson XV-05’s mechanism, which works during a partition precisely because it needs nothing external.

Storage-level fencing is the stronger complement where available: ReadWriteOnce semantics enforced by the storage layer mean a volume cannot be attached to a second node while the first holds it. That is real fencing, done by a layer the partitioned node cannot argue with — and its guarantees depend entirely on the storage implementation rather than on Kubernetes, which is why “does the storage class actually enforce single-attach” is a question with a real answer worth getting.

The thing to take away is that Kubernetes does not remove the split-brain problem, does not weaken it, and adds one new way to reach it through the kubelet’s restart policy. Everything Part XV said still applies.

What to take from this

StatefulSets, PVCs and Services supply identity, storage binding and routing. That is the useful part.
Kubernetes restarts, reschedules and treats pods as fungible — all wrong for a database.
pg_isready is not a health check. A failing liveness probe restarts the pod, which is rarely right.
The kubelet is a service manager, and a restart during a deliberate demotion produces two primaries.
An operator must supply all six Part XV responsibilities plus storage handling; backups are still yours.
Storage decides durability and your commit ceiling. Ask whether it honours fsync.
Kubernetes adds a distributed system between you and the database. Worth it for many clusters, poor for one important one.

Cross-course references

Kubernetes for Production Sysadmins — Part XVII (StatefulSets), Part L (PersistentVolumes and claims) and Part LIV (Stateful workloads) cover the primitives an operator builds on; Part LIII (Storage failure modes) covers what happens when the volume does not detach; and Part XXXIV (PodDisruptionBudgets) covers keeping a drain from taking the primary.
Ceph & Distributed Storage — Part LXXXV (Kubernetes integration) and Part LXXXVIII (Kubernetes storage failure scenarios) cover the storage layer underneath.