Ceph architecture intro - the modern distributed storage

Ceph is the modern open-source distributed storage system. It provides block, object, and file storage with no single point of failure. This lesson covers the architecture.

RADOS

RADOS (Reliable Autonomous Distributed Object Store) is the core of Ceph. It provides:

• Object storage: data is stored as objects.
• Replication: each object is replicated across OSDs.
• Self-healing: when an OSD fails, the data is re-replicated.
• CRUSH: the algorithm that maps objects to OSDs.

RADOS is the foundation for the other Ceph services (RBD, RGW, CephFS).

Components

Block storage with RBD

RBD (RADOS Block Device) is block storage over RADOS:

• Create a RBD image.
• Map it to a node.
• Use it like a local disk (filesystem, mount).

rbd create --size 10G mypool/myimage
rbd map mypool/myimage
# /dev/rbd0 is the block device
mkfs.xfs /dev/rbd0
mount /dev/rbd0 /mnt

RBD supports snapshots, clones, and replication. Used for VM disks and database storage.

Object storage with RGW

RGW (RADOS Gateway) is an S3-compatible object API:

• Create buckets, upload objects.
• Compatible with the AWS S3 API.
• Used for application object storage.

# Configure a user
radosgw-admin user create --uid=alice --display-name="Alice"
radosgw-admin key create --uid=alice --key-type=s3 --access-key=...

# Use with s3cmd or any S3 client
s3cmd --access_key=... --secret_key=... mb s3://mybucket

File storage with CephFS

CephFS is a POSIX filesystem over RADOS:

• MDS handles metadata.
• OSDs handle data.
• Mount via kernel client or FUSE.

# Mount
mount -t ceph <mon1>:6789:/ /mnt/cephfs -o name=admin,secret=...

# Or via ceph-fuse
ceph-fuse /mnt/cephfs

When to use Ceph

• Block storage for VMs (RBD).
• Object storage for applications (RGW / S3).
• Distributed filesystem (CephFS).
• Large-scale (exabytes possible).
• Multi-datacenter.

For smaller clusters, Ceph may be overkill. Use SAN or NFS for tens of hosts. Ceph is for hundreds to thousands.

Knowledge check