Continuous isolation verification and cross-tenant incidents

Why this matters in production

Isolation verified once at launch drifts as capabilities are widened, pools are added, and tenants are migrated.

Continuous verification

#!/bin/bash
# isolation-check.sh — run on a schedule, output diffable
set -u
ceph auth ls --format json | python3 -c '
import sys,json,re
for e in json.load(sys.stdin)["auth_dump"]:
    ent = e["entity"]
    if not ent.startswith("client.") or ent == "client.admin": continue
    caps = e.get("caps", {})
    osd = caps.get("osd", "")
    scoped = ("pool=" in osd) or ("namespace=" in osd)
    star = "allow *" in " ".join(caps.values())
    flag = "UNSCOPED" if not scoped else ("STAR" if star else "ok")
    print("%-10s %-30s %s" % (flag, ent, osd[:70]))'

bash isolation-check.sh | grep -v '^ok'

Any entity that is neither pool-scoped nor namespace-scoped is a
boundary that does not exist, whatever the design document says.

# and the negative test, per tenant
for T in acme beta gamma; do
  for OTHER in acme beta gamma; do
    [ "$T" = "$OTHER" ] && continue
    if rbd -n "client.${T}-prod" -p "${OTHER}-prod-rbd" ls >/dev/null 2>&1; then
      echo "ISOLATION FAILURE: $T can read ${OTHER}-prod-rbd"
    fi
  done
done

Detecting a weakened boundary

diff <(bash isolation-check.sh) /secure/isolation-baseline.txt

One tenant affecting others

Isolation of access does not give isolation of performance. Tenants
share OSDs, network, and the OSD op queue.

ceph osd pool stats

pool acme-prod-rbd id 12
  client io 1.4 GiB/s rd, 890 MiB/s wr, 41k op/s rd, 12k op/s wr
pool beta-prod-rbd id 13
  client io 2.1 MiB/s rd, 900 KiB/s wr, 60 op/s rd, 22 op/s wr

ceph osd perf | sort -k2 -rn | head
ceph daemon osd.0 dump_historic_slow_ops 2>/dev/null | head -30

rbd config image set acme-prod-rbd/vol01 rbd_qos_iops_limit 5000
rbd config image set acme-prod-rbd/vol01 rbd_qos_bps_limit $((200*1024*1024))

# or for every image in the pool
rbd config pool set acme-prod-rbd rbd_qos_iops_limit 5000
rbd config pool ls acme-prod-rbd

RBD QoS is enforced by librbd on the client, so it bounds a cooperating
tenant. A tenant running their own client can remove it.

Attributing impact

ceph osd pool stats --format json | python3 -c '
import sys,json
rows = []
for p in json.load(sys.stdin):
    io = p.get("client_io_rate", {})
    rows.append((p["pool_name"],
                 io.get("read_op_per_sec",0) + io.get("write_op_per_sec",0),
                 io.get("read_bytes_sec",0) + io.get("write_bytes_sec",0)))
for n, ops, bw in sorted(rows, key=lambda r: -r[1])[:8]:
    print("%-24s %8d op/s  %8.1f MiB/s" % (n, ops, bw/1048576))'

Per-pool I/O rates are what convert "the cluster is slow" into "this
tenant is doing 40k op/s", which is the difference between an
investigation and a conversation.

Quiz

Production discipline

Run the isolation check on a schedule and diff it against a baseline — capabilities widen, pools appear, and a boundary verified at launch is not a boundary that still exists. Remember that access isolation is not performance isolation; only device classes or CRUSH roots give that.

Cross-course references

• Kubernetes: NetworkPolicy isolation drifts the same way without continuous verification
• Linux: a control verified once is a control assumed, not a control confirmed