Copying buckets with the S3 API

Intermediate⏱ ~18 minrcloneawsradosgw-admin

What you'll learn

Sync a bucket to an external target
Tune for object count and size distribution
Handle the failure modes
Verify the copy

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

An S3-level copy is portable and works to any target, and it is bounded by object count in ways that surprise people used to block backup.

Syncing

rclone sync ceph:acme-data external:acme-backup \
  --transfers 32 --checkers 64 --fast-list

aws --endpoint-url https://rgw.example.net \
  s3 sync s3://acme-data s3://acme-backup

# what is in the source
radosgw-admin bucket stats --bucket=acme-data | python3 -c '
import sys,json
d = json.load(sys.stdin)
u = d.get("usage", {}).get("rgw.main", {})
print("objects: %d  size: %.1f GiB" %
      (u.get("num_objects",0), u.get("size_actual",0)/1024**3))'

Tuning for the object distribution

The cost is per object, not per byte. A bucket of ten million small
objects takes far longer than one of a thousand large ones at the same
total size.

Distribution	Constraint	Tuning
Many small objects	request rate	raise `--transfers` and `--checkers`
Few large objects	bandwidth	multipart concurrency
Mixed	both	measure, then tune the binding one
Deeply nested prefixes	listing	`--fast-list` to reduce list calls

# listing cost, which precedes any transfer
time rclone lsf ceph:acme-data --recursive | wc -l

# split by prefix and run concurrently
for p in a b c d e f; do
  rclone sync "ceph:acme-data/$p" "external:acme-backup/$p" --transfers 16 &
done
wait

Failure modes

Failure	Behaviour
Interrupted mid-sync	resumable; rerun completes it
Source object modified during sync	the copy is of no single moment
Target rate-limited	transfers retry and slow
Object metadata differences	not all headers survive every target
Very large objects	multipart limits differ between implementations
Deleted objects	`sync` with delete propagates deletions to the backup

# do NOT use --delete on a backup target
rclone sync ceph:acme-data external:acme-backup   # no delete flag

`sync` semantics differ between tools. Anything that mirrors deletions
to the backup target defeats the backup's purpose entirely.

# copy rather than sync, so deletions are never propagated
rclone copy ceph:acme-data external:acme-backup

Verifying

rclone check ceph:acme-data external:acme-backup --one-way

# object counts, from both ends
radosgw-admin bucket stats --bucket=acme-data | python3 -c '
import sys,json
print(json.load(sys.stdin).get("usage",{}).get("rgw.main",{}).get("num_objects"))'
rclone size external:acme-backup

# spot-check content, not only counts
rclone cat external:acme-backup/known-object | sha256sum
rclone cat ceph:acme-data/known-object | sha256sum

Quiz

Knowledge check · 4 questions

Q1. What bounds the duration of an S3-level bucket copy?

Object count — each object needs HTTP round trips whose per-request overhead does not shrink with object size Total byte count, since the copy is bounded by the link between the source and target gateways The replication factor of the source pool, since each replica is read before the object is sent The RGW instance count, since each gateway serves a fixed share of the copy's concurrent requests and cannot exceed it
Q2. A backup target should accumulate objects rather than track what the source currently holds.

True False
Q3. Back up an RGW bucket to an external target.

A bucket holds 12 million objects totalling 4 TiB. The proposed command is a nightly `rclone sync --delete`.
Q4. What should be verified after a bucket copy, beyond object counts?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Use copy rather than sync semantics for backup targets — sync propagates deletions to the backup, usually before anyone notices the source deletion. Tune for object count, not size: concurrency is the only lever that helps a request-rate-bound copy.

Cross-course references

Kubernetes: a reconciler that mirrors deletions is not a backup either
Linux: per-object overhead dominates when objects are small