Objects — the unit of storage everything else is built from

Foundation⏱ ~15 minradosceph

What you'll learn

Describe the four components of a RADOS object
Explain how an object name determines its placement
Inspect objects directly with the rados command
Relate service-level entities to their underlying RADOS objects

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Every Ceph service is a way of arranging objects. When an RBD image misbehaves, when a bucket listing is slow, when a scrub reports an inconsistency, the thing being discussed is ultimately a RADOS object. Being able to look at one directly is a genuinely useful skill.

What an object is

name    a string, unique within its pool
data    a byte array, typically up to 4 MiB in practice
xattrs  small key/value metadata attached to the object
omap    a larger sorted key/value map, stored in RocksDB

The omap is the part that surprises people. It is a per-object key-value store held in BlueStore’s RocksDB rather than alongside the object data, and it is what makes RGW bucket indexes and CephFS directories possible. It is also why metadata-heavy workloads need fast devices for RocksDB.

Naming determines placement

An object’s name is hashed to produce a placement group:

pg_id = hash(object_name) % pg_num

then CRUSH maps that PG to an acting set. So placement is a pure function of the name, the pool’s pg_num, and the cluster map. Nothing is recorded anywhere.

rados -p rbd-vms put testobj /etc/hostname
ceph osd map rbd-vms testobj
rados -p rbd-vms get testobj -
rados -p rbd-vms ls | head
rados -p rbd-vms rm testobj

How services map onto objects

Service entity	RADOS objects
RBD image	`rbd_data.<prefix>.<seq>`, 4 MiB each, plus a header object
CephFS file	`<inode-hex>.<seq>` in the data pool; metadata in omap in the metadata pool
RGW object	one or more `<bucket-marker>_<key>` objects, plus an index entry in the bucket index object’s omap

Finding the objects behind an RBD image:

PREFIX=rbd_data.12ab34cd56ef        # from block_name_prefix below
OBJECT=rbd_data.12ab34cd56ef.0000000000000001

rbd info rbd-vms/vm-101-disk-0     # gives block_name_prefix
rados -p rbd-vms ls | grep "${PREFIX}" | head
ceph osd map rbd-vms "${OBJECT}"

That chain — image to prefix to object to PG to OSD — is the standard route from a user-visible complaint to a specific device.

Why objects are 4 MiB and what changing it costs

The default RBD object size is 4 MiB, and it is a balance.

Smaller objects spread an image across more PGs and therefore more OSDs, which increases parallelism for a single client. They also multiply the per-object metadata: more entries in RocksDB, more work during scrub and recovery, and more overhead per byte stored.

Larger objects reduce metadata overhead and suit sequential workloads, but concentrate an image’s I/O onto fewer OSDs, which can make a single image’s throughput bounded by a handful of devices.

The default suits most workloads. Changing it is a per-image decision made at creation with rbd create --object-size, and it cannot be changed afterwards without copying the image. Very large objects (32 MiB and above) are occasionally right for sequential archival workloads; very small ones are almost never right, because the metadata cost compounds across every operation the cluster performs.

Inspecting an object fully

INDEX_OBJECT=report.pdf
rados -p rbd-vms stat rbd_data.abc123.0000000000000002
rados -p rbd-vms listxattr rbd_data.abc123.0000000000000002
rados -p rbd-vms listomapkeys ${INDEX_OBJECT}

listomapkeys on an RGW bucket index object is how you see the actual index entries — useful when a bucket listing is slow and you want to know how many entries a single shard holds.

Quiz

Knowledge check · 4 questions

Q1. Where does a RADOS object omap live, and why does that matter?

In BlueStore RocksDB, so metadata-heavy workloads need fast devices for it Alongside the object data on the block device, so it adds no metadata cost at all In the monitor store, replicated across every monitor alongside the cluster map In a per-pool index object shared by every object in the pool, held on the primary OSD
Q2. Running rados -p <pool> ls on a production pool with hundreds of millions of objects is a safe, low-impact way to find objects.

True False
Q3. A deep scrub reports an inconsistent object in an RBD pool. Trace it back to the affected VM and decide what to do.

ceph health detail reports 1 inconsistent PG in pool rbd-vms, PG 7.3a. The cluster has 40 VMs across the pool. Pool size 3, min_size 2. No OSD has failed and no device shows SMART errors yet. The team wants to know which VM is affected before running a repair.
Q4. Explain the trade-offs in RBD object size and why the 4 MiB default suits most workloads.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Learn the route from a user complaint to a device: image to block_name_prefix to object to PG to acting set to host. It is four commands and it answers most questions faster than any dashboard. Avoid rados ls on production pools — it is a full scan with real cost — and reach for --pgid or the service-level index tools instead. And put the BlueStore DB on fast media wherever omap is heavily used, because that is where bucket indexes and directories actually live.

Cross-course references

Ceph: Part XII (BlueStore) for where omap is stored.
Ceph: Part XXXVI (RBD Images) for the image-to-object mapping.
Ceph: Part LXII (Inconsistent PGs) for the repair decision in depth.