Block storage - the primitive every database depends on

Why this matters in production

Block storage is the primitive underneath almost every stateful production workload: database volumes, VM disks, container volumes that need filesystem semantics, and any application whose performance depends on writes hitting the disk in deterministic latency. A systems administrator who cannot describe block storage precisely cannot reason about why a Ceph cluster’s RBD service is fast for one workload and slow for another, cannot read a ceph osd perf output, and cannot design a CRUSH rule that puts replicas on devices of the right failure domain.

This lesson is the foundation. RBD, iSCSI targets, LVM-on-Linux, local NVMe, and a QEMU virtual disk are all block storage at different layers. . Distinguishing the layers matters when you design a Ceph cluster.

What block storage is

Block storage is a primitive that exposes a fixed-size address space as numbered, fixed-size blocks (usually 512 B or 4 KiB). Reads and writes address the block by its offset and length. The client is responsible for choosing where to write; the storage is responsible only for what bytes are written there.

Three properties make block storage the right primitive for specific workloads:

• Random access. A block at offset n is reachable in the same time as a block at offset m. There is no traversal.
• Predictable latency. A well-designed block device admits a predictable I/O latency profile: read at offset, write to offset, with a bounded queue depth. Filesystems and object APIs add overhead the block device does not.
• Filesystem independence. The client may put a filesystem of its choice on top: ext4, XFS, Btrfs. The block device knows nothing about the filesystem.

flowchart LR
  app[Application] --> fs[Filesystem<br/>ext4, XFS, Btrfs]
  fs --> blk[Block device]
  blk --> drv[Linux block layer<br/>/dev/sda, /dev/rbd0]
  drv --> hw[Disk or RAID or RBD image]

Why some workloads demand block storage

Block storage is the right answer when:

• Random access dominates. A database working set has no locality at the API level (the buffer pool touches any page in the cache); locality is at the index level. Object storage pays a per-object HEAD-like cost that is ruinous when the workload reads 8 KiB pages.
• Latency budget is small. Object storage commonly adds one to ten milliseconds of latency per operation. A VM’s virtio-blk write that hits RBD at 1 ms p99 will hit a cloud-grade object API at 10-50 ms p99 — order-of-magnitude worse, often unusable.
• The client owns the filesystem. A database needs recovery, snapshot, write-ahead logging, and crash-safety. The block primitive lets the client pick the filesystem that delivers those semantics. An object API gives the storage layer the filesystem role — and object storage is not designed to give crash-safe POSIX semantics.

Block storage is not the right answer for:

• Large immutable data (archives, backups, content). Object storage is cheaper to operate per TiB and avoids the filesystem responsibility.
• Many clients writing the same namespace simultaneously. A shared POSIX filesystem (CephFS, NFS) is the answer; concurrent block writes to the same volume must be serialised at the filesystem layer, which is fragile.

How Linux presents block storage

The Linux kernel exposes block devices at the block-layer abstraction (/dev/sdX, /dev/rbd0, /dev/nvme0n1, /dev/dm-0, /dev/zd0). A userspace application reads and writes through the Virtual File System (VFS) → filesystem → block device → block queue → driver → hardware path.

flowchart TD
  A[Application] --> B[read / write / fsync]
  B --> C[VFS]
  C --> D[Filesystem<br/>ext4/XFS/Btrfs]
  D --> E[Block layer]
  E --> F[Queue and scheduler<br/>none/mq-deadline/bfq]
  F --> G[Driver: virtio, nvme, rbd, sd]
  G --> H[Device or remote storage]

The block-layer scheduler is a tunable. Ceph-aware operators typically leave the default mq-deadline on ceph hosts because the workload profile is dominated by latency-sensitive random I/O and recovery scans; none may be preferable for NVMe-backed OSDs. The scheduler matters for performance but is not part of the Ceph architecture.

Ceph RBD as a block primitive

Ceph RBD is one specific implementation of the block primitive — the one most relevant to this course. An RBD image is a Ceph object striped across one or more PGs of a (typically replicated) pool. The image is exposed to the client as either:

• A kernel block device via rbd map <pool>/<image>, presenting /dev/rbdN to the host.
• A QEMU virtual disk via qemu-img or libvirt, used by KVM, Proxmox VE, and most cloud hypervisors.
• A Kubernetes persistent volume via the csi-rbd driver, exposed to the Pod as /dev/<xvda|zvd|vdX>.

In every case the interface is a block device, and the implementation is RADOS underneath. The cluster’s MON and OSD processes may be on different hosts from the client; the network and the failure domain awareness matter.

flowchart LR
  client[Client: KVM / kubelet / rbd map] --> rbd[RBD client<br/>librbd or kernel rbd]
  rbd --> rados[librados]
  rados --> pg[PG]
  pg --> osd1[OSD 1<br/>replica 0]
  pg --> osd2[OSD 2<br/>replica 1]
  pg --> osd3[OSD 3<br/>replica 2]

What RBD is not

RBD is not:

• A SAN. A SAN is a block-device-on-shared-storage topology with its own failure modes (fabric, multipath, controller, zone). RBD is block storage provided by an object store; the failure modes are different (PGs, OSDs, network, MON quorum).
• A shared filesystem. Two clients cannot mount the same RBD image read-write safely. RBD is single-writer by design. Concurrent mount requires a clustered filesystem layer (CephFS, GFS2, OCFS2) above RBD.
• A substitute for local NVMe. RBD round-trips across the network; the latency cost is real. A workload that needs sub-millisecond p99 latency should run on local NVMe with a backup-and- archive tier into Ceph.

Quiz

Production discipline

Block storage is the primitive for stateful workloads with tight latency budgets and small random I/O. RBD is the production implementation this course uses. The operational discipline is to model the latency budget end to end (client → block layer → RBD → librados → network → OSD → disk) and to size each link to keep the budget. The architectural mistake to avoid is treating RBD as if it were local NVMe; the right comparison is local-disk-equivalent-class hardware, not a single SSD.

Cross-course references

• Linux: Part XIII (Disks and Block Devices), Part XLI (Storage Performance), Part XIV (Filesystems).
• Proxmox: Part V (Storage Fundamentals), Part VIII (Ceph), Part IX (Virtual Machines and Storage Backends).
• Observability: Part LVI (Linux Observability) for monitoring block-layer latency.