Testing tenant isolation before launch

Why this matters in production

Every isolation configuration works when tested with the operations it is meant to permit. The tests that matter are the ones that should fail, and they are the tests nobody runs — which is how a capability with a typo in the pool name reaches production as an unbounded grant.

The principle

For each tenant credential, attempt every operation it should not be able to perform, and confirm each is denied. Record the command and the output.

RBD tests

T=/etc/ceph/ceph.client.tenant-a.keyring

# should SUCCEED — its own namespace
rbd -n client.tenant-a --keyring $T ls --namespace tenant-a rbd-shared

# should FAIL — another tenant's namespace
rbd -n client.tenant-a --keyring $T ls --namespace tenant-b rbd-shared

# should FAIL — the pool without a namespace
rbd -n client.tenant-a --keyring $T ls rbd-shared

# should FAIL — a different pool entirely
rados -n client.tenant-a --keyring $T -p rbd-gold ls

# should FAIL — administrative operations
ceph -n client.tenant-a --keyring $T osd pool delete rbd-shared rbd-shared \
    --yes-i-really-really-mean-it

CephFS tests

# Substitute one of your own monitor addresses before running:
MON=192.0.2.11:6789
SECRET=$(ceph auth print-key client.tenant-a)

mount -t ceph "$MON:/" /mnt/tenant-a -o "name=tenant-a,secret=$SECRET"

# should FAIL or show nothing — another tenant's subtree
ls /mnt/tenant-a/volumes/research/project-beta

# should FAIL — mounting the filesystem root
mount -t ceph "$MON:/" /mnt/root-test -o "name=tenant-a,secret=$SECRET"

# with root_squash, should FAIL — writing as root outside permitted ownership
sudo touch /mnt/tenant-a/root-owned-file

RGW tests

# should FAIL — reading another tenant's bucket
aws --endpoint-url $EP --profile tenant-a s3 ls s3://globex:backups

# should FAIL — exceeding quota (write past the limit and confirm refusal)
# expect a bounded overshoot before the refusal, and record how much

# should FAIL — administrative operations
aws --endpoint-url $EP --profile tenant-a s3api list-buckets   # only own buckets

Recording results

Test                                  Expected  Actual  Evidence
tenant-a reads tenant-b namespace     DENY      DENY    <command output>
tenant-a lists pool without namespace DENY      DENY    <command output>
tenant-a deletes pool                 DENY      DENY    <command output>
tenant-a mounts CephFS root           DENY      DENY    <command output>
tenant-a exceeds quota                DENY      DENY    overshoot 340 MB

This table is the evidence that the boundary exists. It is also what makes a re-test after any capability change a five-minute job rather than a fresh design exercise.

Re-test triggers

• Any capability change on a tenant credential
• Adding a new isolation mechanism or tenant type
• Ceph major version upgrades
• Before any external audit
• Periodically, as configurations drift

Quiz

Production discipline

Build the negative test suite once and script it, so re-testing after a capability change costs minutes. Record the exact error returned rather than just pass or fail — the difference between EACCES and ENOENT is a real finding on services where tenant existence is confidential.

Cross-course references

• Kubernetes: RBAC boundary testing with kubectl auth can-i --as is the direct equivalent
• Linux: verifying that a confined service account cannot reach what it should not is the same discipline