Fast disks are not fast distributed storage

Why this matters in production

“We bought NVMe, so storage will be fast” is a sentence that has preceded a great many disappointing Ceph deployments. NVMe is necessary for a low-latency cluster and it is nowhere near sufficient, because the device is one of five things in the path and usually not the largest.

The honest framing: the device latency is the floor. Everything else in the distributed path is added on top, and the total is what the application experiences.

Decomposing a replicated write

flowchart TD
  A[Client issues write] --> B[librbd / librados<br/>encode, choose PG]
  B --> C[Network to primary OSD]
  C --> D[Primary OSD queue and threads]
  D --> E[BlueStore commit on primary]
  D --> F[Network to replica 1]
  D --> G[Network to replica 2]
  F --> H[BlueStore commit on replica 1]
  G --> I[BlueStore commit on replica 2]
  H --> J[Ack to primary]
  I --> J
  E --> J
  J --> K[Ack to client]

An indicative budget on a well-built NVMe cluster with a 25 GbE network:

The device itself contributes perhaps a quarter of that. A local NVMe write completing in 80 microseconds becomes a 1.5 ms distributed write, and no amount of faster NVMe closes the gap — the network and the software path are the majority of it.

The consequences that surprise people

• Faster drives have diminishing returns. Moving from SATA SSD to NVMe helps meaningfully. Moving from one NVMe generation to a faster one usually does not, because the device stopped being the bottleneck.
• The network becomes the thing to buy. Latency at 10 GbE versus 25 GbE is a real difference in the round-trip components, and there are three of them in the diagram above.
• Replication factor is a latency setting as well as a durability one. Waiting for three commits rather than two costs the difference between the second-slowest and the slowest.
• One slow device raises everyone’s latency, because the write waits for the slowest replica and any given OSD participates in many PGs.

Designing with the real number

The practical discipline is to establish the budget first and design the application to it, rather than discovering it afterwards:

• A database whose commit latency budget is 1 ms should be on local NVMe, with Ceph as the backup and archive tier.
• A database whose budget is 10 ms is comfortable on an NVMe-backed RBD pool.
• A VM fleet doing routine work is comfortable on SATA SSD or on HDD with NVMe metadata devices.
• An archive does not have a latency budget worth discussing.

Placing a workload against that list at design time is a five-minute conversation. Discovering it in production is a migration.

Quiz

Production discipline

Publish a latency budget derived from the architecture before anyone buys hardware or places a workload, and state it as a range with its components. Once NVMe is in place, spend on the network rather than on faster drives — there are two network traversals on the critical path of every write and only one device commit. And treat any requirement below roughly a millisecond as a signal that the workload belongs on local storage with Ceph in a supporting role, rather than as a tuning target.

Cross-course references

• Ceph: Part XXVIII (Ceph Networking) and Part XXIX (Network Design) for the links this budget depends on.
• Ceph: Part XII (BlueStore) for the commit path on the OSD side.
• Linux: Part XLII (Network Performance) for measuring the round trips directly.