CRUSH versus centralised lookup — what the trade actually is

Advanced⏱ ~15 minceph

What you'll learn

Compare the scaling properties of lookup and computation
Identify what CRUSH gives up in exchange
Explain how upmap recovers some lookup-like control
Judge which approach suits which storage design

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Understanding the trade explains several Ceph behaviours that otherwise look like shortcomings: why distribution is uneven, why topology changes move data, and why upmap exists.

The two approaches

Centralised lookup — a service records where every object lives. Clients query it.

Exact placement control; any object can go anywhere.
Placement can be changed without moving data’s identity.
The table grows with object count, into billions of entries.
The service is in the data path: a bottleneck and a failure point.

Computation — clients derive placement from a small shared map.

No service in the data path; scales with clients for free.
The shared state stays small regardless of object count.
Placement is pseudo-random, so distribution is uneven.
Changing the map changes placement, which moves data.

Ceph chose computation, and every consequence below follows.

What Ceph gives up

Even distribution. Pseudo-random placement over ~100 PGs per OSD leaves several percent of variance. A lookup table could be perfectly balanced.

Placement control. You cannot say “put this object on osd.47”. CRUSH decides.

Free topology change. Changing the map changes the function, which moves data. A lookup system could change placement policy for new data only.

upmap is Ceph adding a small lookup table where it helps most

upmap records explicit per-PG exceptions in the osdmap: “PG 7.3d goes to osd.91, not osd.83”.

That is a lookup table — deliberately tiny, holding exceptions rather than every mapping. It recovers the precision of lookup for the specific problem where CRUSH’s pseudo-randomness costs real capacity, without reintroducing a per-object table or a service in the data path.

ceph balancer mode upmap
ceph osd dump | grep -c upmap

Watching the entry count is the check that the exception table stays small. Tens of thousands means the balancer is compensating for a structural problem, and at that point the exceptions are no longer a tweak on top of the computation.

Why the trade favours Ceph’s design

The decisive number is object count. A petabyte of 4 MiB objects is roughly 260 million objects. A lookup table with an entry per object, replicated for durability and consulted on every operation, is a system in its own right — and it must scale with data volume while staying fast enough to sit in the data path.

Ceph’s shared state is the cluster map: a few megabytes describing topology, constant in size as data grows from terabytes to petabytes.

The distribution variance CRUSH accepts costs a few percent of capacity, recoverable with upmap. The lookup service would cost a distributed database on the critical path of every I/O.

Why this shapes what Ceph can and cannot offer

Several features Ceph does not have follow directly from computed placement.

There is no per-object placement policy — you cannot pin one object to particular hardware, because placement is a function of the name. Object-level tiering has to be expressed at the pool level instead.

There is no cheap rebalancing of policy without moving data, because policy is an input to the function rather than a stored decision. Changing a rule recomputes everything.

And there is no way to look up “which objects are on osd.12” without enumerating, because the mapping runs one way — name to location, not location to names.

These are not gaps to be filled in a later release; they are the shape of the design. Recognising them prevents both frustrated searching for features that cannot exist and designs that assume capabilities the architecture excludes.

Judging the trade

Computation suits systems where object count is enormous, clients are numerous, and placement policy is uniform across large groups of objects — which describes cluster storage precisely.

Lookup suits systems with fewer, larger entities needing individual placement decisions — which describes a SAN array or a filesystem namespace, both of which are smaller in cardinality by orders of magnitude.

Quiz

Knowledge check · 4 questions

Q1. What is the fundamental trade Ceph makes by computing placement rather than looking it up?

Lower read latency, because clients reach the primary directly, in exchange for writes that must be coordinated Stronger consistency between replicas, in exchange for availability whenever a whole failure domain is lost Smaller objects than a lookup system could address, in exchange for greater parallelism across the OSDs No metadata service in the data path, in exchange for pseudo-random distribution and data movement on topology change
Q2. upmap reintroduces a small, bounded lookup table into an otherwise computed placement system.

True False
Q3. A team asks whether they can pin a specific RBD image to particular OSDs for a latency-sensitive VM. Respond.

One VM out of 400 has a strict latency requirement. The team wants its RBD image placed on three specific NVMe OSDs known to be idle. The pool is a general SSD pool. They are asking whether upmap or some other mechanism can pin the image.
Q4. Name three capabilities that computed placement excludes, and explain why they are architectural rather than missing features.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Understand the trade so you stop looking for features the architecture excludes: no per-object placement, no policy change without data movement, no reverse lookup from OSD to object names. Use upmap as the bounded exception table it is, and watch its entry count — once it grows into the tens of thousands it has stopped being a correction and is masking something structural. And express placement requirements at pool level, because that is the granularity the design offers.

Cross-course references

Ceph: Part X (Manager Daemons) for the balancer using upmap.
Ceph: Part XVII (Pools) for pool-level placement policy.
Ceph: Part CXXI (Storage Architecture Decision-Making) for the wider comparison.