Building an RGW backup inventory

Why this matters in production

Object storage accumulates buckets nobody remembers creating, and backing up everything is usually neither affordable nor necessary.

Enumerating

radosgw-admin bucket list
radosgw-admin user list

radosgw-admin bucket list --format json | python3 -c '
import sys,json,subprocess
for b in json.load(sys.stdin):
    s = json.loads(subprocess.check_output(
        ["radosgw-admin","bucket","stats","--bucket",b]))
    u = s.get("usage", {}).get("rgw.main", {})
    print("%-32s %-24s %10d obj  %9.2f GiB" %
          (b, s.get("owner","?"),
           u.get("num_objects",0), u.get("size_actual",0)/1024**3))'

# buckets per user
radosgw-admin user list --format json | python3 -c '
import sys,json,subprocess
for u in json.load(sys.stdin):
    out = subprocess.check_output(["radosgw-admin","bucket","list","--uid",u])
    print("%-28s %d buckets" % (u, len(json.loads(out))))'

Characterising

radosgw-admin bucket stats --bucket=acme-data | python3 -c '
import sys,json
d = json.load(sys.stdin)
print("owner:      ", d.get("owner"))
print("created:    ", d.get("creation_time"))
print("versioning: ", d.get("versioning", d.get("flags")))
print("placement:  ", d.get("placement_rule"))'

# usage over time, to find idle buckets
radosgw-admin usage show --show-log-entries=false --start-date=$(date -d '30 days ago' +%Y-%m-%d) \
  | python3 -c '
import sys,json
d = json.load(sys.stdin)
for e in d.get("summary", []):
    t = e.get("total", {})
    print("%-28s ops %8d" % (e.get("user"), t.get("ops",0)))'

Deciding what to protect

For each bucket, ask the owner:
  can this be regenerated from another source?
  what happens if it is lost?
  what happens if it is a week out of date?

# record it
cat >> /secure/rgw-backup-inventory.txt <<'EOF'
acme-data      | owner acme | 12M obj 4.1TiB | daily copy + versioning
build-cache    | owner ci   |  2M obj 900GiB | none — regenerable
audit-logs     | owner sec  | 400k obj 80GiB | object lock, 7 year retention
EOF

Keeping it current

# diff against the recorded inventory
diff <(radosgw-admin bucket list --format json | python3 -c '
import sys,json; print("\n".join(sorted(json.load(sys.stdin))))') \
     <(awk -F'|' '{gsub(/ /,"",$1); print $1}' /secure/rgw-backup-inventory.txt | sort)

A bucket in the cluster and not in the inventory is unprotected and
unattributed. That diff is the whole value of keeping the record.

Quiz

Production discipline

Ask bucket owners whether their data is regenerable before sizing any backup — build caches and pipeline outputs are often the majority of capacity and need no protection at all. Diff the live bucket list against the inventory on a schedule to catch unattributed buckets.

Cross-course references

• Kubernetes: an unlabelled namespace has no owner and no backup decision either
• Linux: identifying reproducible data is the cheapest capacity saving available