Client traffic characteristics by interface

Intermediate⏱ ~16 mincephiftop

What you'll learn

Characterise the network profile of each Ceph interface
Predict bandwidth and packet-rate demands
Identify which metric is binding for a given workload
Size the public network against a mixed client mix

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Sizing a network by bandwidth alone fails for RBD, where packet rate and latency bind long before bits per second do. The three interfaces stress different dimensions, and a cluster serving all three needs the union of their requirements rather than the maximum of them.

The three profiles

RBD — block. Small random operations, latency-critical.

Dimension	Character
Request size	4–64 KiB typically
Pattern	random
Packet rate	high
Bandwidth	moderate
Latency sensitivity	very high

CephFS — file. Mixed, plus a separate metadata path.

Dimension	Character
Request size	wide range
Pattern	mixed sequential and random
Metadata ops	many small MDS round trips
Bandwidth	variable
Latency sensitivity	high on metadata

RGW — object. Large sequential transfers over HTTP.

Dimension	Character
Request size	megabytes
Pattern	sequential
Packet rate	moderate
Bandwidth	high
Latency sensitivity	low

What binds in each case

For RGW, bandwidth binds — a 10 Gb link caps you at about 1.2 GB/s regardless of anything else.

For RBD, packet rate and latency bind. 50,000 IOPS of 16 KiB operations is only 800 MB/s — trivial bandwidth — but it is 100,000+ packets per second in each direction plus acknowledgements, and each operation’s latency includes the full network round trip. A network with plenty of bandwidth headroom can still deliver poor RBD performance if its per-packet latency is high or its interrupt handling is saturated.

For CephFS, the metadata path binds first on workloads with many small files, because each metadata operation is a separate round trip to the MDS.

Measuring what you have

PUBLIC_NIC=public_nic
ceph osd pool stats                    # per-pool client op rates
iftop -i ${PUBLIC_NIC}                  # live bandwidth by peer
sar -n DEV 5                           # packets per second, not just bytes
ss -s                                  # socket counts and states

Watch packets per second alongside bandwidth. A link at 20% of its bit rate can still be saturated on packet processing.

Quiz

Knowledge check · 4 questions

Q1. A cluster serves 50,000 IOPS of 16 KiB RBD operations. What is most likely to bind first?

Packet rate, latency, and host interrupt processing Link bandwidth, since 800 MB/s saturates a 10 Gb link The monitors, which distribute an OSD map update per client op Pool capacity, because each 16 KiB write allocates a full object
Q2. For RGW workloads, link bandwidth is usually the binding network constraint.

True False
Q3. Size a network for a mixed client population.

A new cluster will serve three workloads: RBD for 400 VMs doing mostly 8–32 KiB random I/O, CephFS for a build farm with millions of small files, and RGW for a 500 TB object archive with large sequential transfers. Budget allows either 100 Gb with more hops or 25 Gb with a flatter topology.
Q4. Why can a link at 20% of its bit rate still be saturated?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Monitor packets per second alongside bandwidth on OSD and client hosts, and include mpstat softirq figures in any network investigation — the host-side ceiling is invisible in bandwidth graphs. Record the design target for each workload class so a later capacity discussion knows which dimension the network was sized against.

Cross-course references

Kubernetes: CNI overhead per packet affects small-request workloads the same way
Linux: tuning receive-side scaling and ring buffers is the standard response to this ceiling