Finding every copy of a keyring

Why this matters in production

Rotation, revocation, and incident response all require knowing where a key exists. Almost no cluster does.

Where keyrings legitimately live

ceph auth ls --format json | python3 -c '
import sys,json
for e in json.load(sys.stdin)["auth_dump"]:
    print(e["entity"])'

That is the list of keys that exist. The question is where each one has
been copied to.

Sweeping for copies

# on every Ceph and client host
find / -xdev -name '*.keyring' -o -name 'ceph.conf' 2>/dev/null | \
  grep -v '^/proc' | head -40

# files containing a cephx key, wherever they are named
grep -rIl --exclude-dir=/proc --exclude-dir=/sys \
  -E 'key[[:space:]]*=[[:space:]]*AQ[A-Za-z0-9+/=]{20,}' / 2>/dev/null | head -20

Ceph keys are base64 and conventionally begin AQ, which makes them
greppable. That is convenient for an audit and equally convenient for
anyone else with filesystem access.

# git history, in a repo that may have held one
git log -p --all -S 'AQ' -- '*keyring*' 'ceph.conf' 2>/dev/null | head -40

# container images
# Substitute the image you are auditing before running:
IMAGE=registry.example.com/apps/backup-agent
TAG=v2.4.1

skopeo inspect --config "docker://$IMAGE:$TAG" 2>/dev/null | grep -i ceph

Assessing each copy

# what a found key grants
NAME=acme
ceph auth get client.${NAME}

Assess before panicking and before ignoring. A read-only monitoring key
in a git repository is a real finding with a modest consequence.

Reducing the footprint

For each copy that should not exist:
  establish whether anything uses it
  if not, delete the file and record it
  if yes, move that consumer to the intended distribution path
  then delete the file
  if the copy was in git history or a registry, the key must be rotated

# rotation is required, not optional, for a key that reached a
# location you cannot fully purge
# the exposed entity, from the assessment above
NAME=acme

ceph auth get "client.${NAME}" -o /tmp/old.keyring   # keep for rollback

Quiz

Production discipline

Rotate — do not merely delete — any key found in git history, a container registry, or anywhere else that cannot be fully purged. Configure secret-scanning for the cephx key pattern; catching a key before it spreads costs far less than sweeping for it afterwards.

Cross-course references

• Kubernetes: a Secret committed to git requires rotation for the identical reason
• Linux: credential scanning at commit time is cheaper than credential archaeology