Capability lifecycle: issuance, drift, and review

Why this matters in production

Least privilege is easy at creation and hard to keep. Capabilities drift upward because widening them fixes an immediate problem and narrowing them never does.

An issuance process

For every new entity, record before creating it:
  who or what it is for
  which pools it needs and with which access
  who requested it and who approved
  when it should be reviewed
  where its keyring will live

ceph auth get-or-create client.app-billing \
  mon 'profile rbd' \
  osd 'profile rbd pool=billing-rbd' \
  mgr 'profile rbd pool=billing-rbd'

# and record it outside Ceph
cat >> /etc/ceph/entity-register.txt <<'EOF'
client.app-billing | billing app RBD volumes | pools: billing-rbd
  requested: platform-team | approved: storage-lead
  review: quarterly | keyring: k8s secret ceph-billing in ns billing
EOF

Ceph stores the capabilities but not the reason. The register is what
makes a later review possible — without it, every entity looks equally
justified.

How capabilities drift

# entities with unrestricted capabilities
ceph auth ls --format json | python3 -c '
import sys,json
for e in json.load(sys.stdin)["auth_dump"]:
    caps = e.get("caps", {})
    if any("allow *" in v or v.strip() == "allow *" for v in caps.values()):
        print("%-32s %s" % (e["entity"], caps))'

# entities not seen recently — candidates for removal
ceph auth ls --format json | python3 -c '
import sys,json
for e in json.load(sys.stdin)["auth_dump"]:
    print(e["entity"])' | sort > /tmp/all-entities.txt
comm -23 /tmp/all-entities.txt /tmp/known-active-entities.txt

The periodic review

Quarterly, for every entity:
  does it still exist in the register?
  is the application still running?
  are its capabilities still the minimum that works?
  has its keyring been in a place it should not be?
  when was its key last rotated?

Removing entities safely

# Substitute your own values before running:
POOL=rbd-vms
IMAGE=vm-disk-01
ENTITY=client.old-app

# confirm nothing is using it first
rbd status "$POOL/$IMAGE" 2>/dev/null | grep -i watcher
ceph tell mds.0 client ls 2>/dev/null | grep -i "$ENTITY"

# narrow before removing, so breakage is recoverable
ceph auth caps client.old-app mon 'allow r' osd 'allow r pool=oldpool'
# observe for a period, then
ceph auth rm client.old-app

Narrowing first means a client that turns out to still exist fails
visibly and can be restored by widening. Removing outright means it fails
and the key must be reissued and redistributed.

ceph auth get client.old-app -o /root/client.old-app.keyring   # keep a copy
ceph auth rm client.old-app

Quiz

Production discipline

Keep an entity register outside Ceph recording each key’s purpose, owner, intended scope, and keyring location — Ceph stores capabilities but not reasons, and without reasons a review has nothing to compare against. Narrow before removing, and export the key first.

Cross-course references

• Kubernetes: RBAC bindings drift upward for the identical asymmetric-incentive reason
• Linux: privilege review needs a record of intent, not only current state