External state and databases - the stateful backend

For most cluster architectures, the state is kept in an external system: a database, a cache, or an object store. The application is stateless. This lesson covers the patterns.

Databases as cluster state

The standard pattern: keep state in a database, keep the application stateless.

Pros:

• Database does the hard work (transactions, replication, backup).
• Application is stateless; any instance handles any request.
• Database is the single source of truth.

Cons:

• Database is a dependency (and a single point of failure if not replicated).
• Network latency for every request.

SQL vs NoSQL

For cluster state, the choice depends on the access pattern:

• Transactions and complex queries: SQL (PostgreSQL, MySQL).
• Simple key-value: NoSQL (etcd, Redis).
• Document storage: NoSQL (MongoDB).
• Graph: Neo4j.

Caching patterns

A cache reduces database load:

Application → Cache (Redis) → Database
   |              |              |
   read-through, write-through, write-behind

• Read-through: cache reads, fetches from DB on miss.
• Write-through: cache writes to DB and updates cache.
• Write-behind: cache writes asynchronously to DB.
• Cache-aside: application reads from cache, writes to DB and invalidates cache.

Cache invalidation is the hard part: ensure the cache reflects the DB state.

Object stores

For files, blobs, and large data:

• S3: de facto standard.
• MinIO: self-hosted S3-compatible.
• Ceph RGW: distributed object store.

Object stores are highly available, replicated, and durable. Use them for backups, media, large files.

Distributed state stores

For cluster coordination:

• etcd: distributed key-value, leader election, used by Kubernetes.
• Consul: service discovery, KV store, health checks.
• ZooKeeper: similar, used by older Hadoop.

These provide the state that cluster managers (like Pacemaker) need.

Choose the right state backend

Knowledge check