Tenant onboarding and isolation testing — the operational discipline

Tenant onboarding and isolation testing are the operational discipline of multi-tenancy. This lesson walks the onboarding checklist, the isolation tests, the audit cadence, and the operational discipline.

The onboarding checklist

flowchart LR
    A[Tenant onboarding] --> B[Create namespace]
    B --> C[Apply RBAC]
    C --> D[Apply NetworkPolicy]
    D --> E[Apply ResourceQuota]
    E --> F[Apply PSS labels]
    F --> G[Configure secrets]
    G --> H[Document in runbook]
    H --> I[Tenant is ready]

The onboarding checklist:

1. Create namespace. With metadata labels (tenant ID, environment, owner).
2. Apply RBAC. Roles and RoleBindings scoped to the namespace.
3. Apply NetworkPolicy. Default-deny + explicit allow rules.
4. Apply ResourceQuota. Bounded CPU, memory, object counts, storage.
5. Apply PSS labels. restricted enforcement.
6. Configure secrets. Vault, sealed-secrets, or External Secrets Operator.
7. Document in runbook. Tenant ID, contacts, quotas, policies.

A onboarding script

#!/bin/bash
# onboard-tenant.sh <tenant-name> <environment>

TENANT=$1
ENV=$2
NS="${TENANT}-${ENV}"

# 1. Create namespace
kubectl create namespace "$NS" \
  --labels="tenant=$TENANT,env=$ENV,pod-security.kubernetes.io/enforce=restricted"

# 2. Apply NetworkPolicy (default-deny + DNS allow)
kubectl apply -n "$NS" -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
      ports:
        - port: 53
          protocol: UDP
EOF

# 3. Apply ResourceQuota
kubectl apply -n "$NS" -f - <<EOF
apiVersion: v1
kind: ResourceQuota
metadata:
  name: tenant-quota
spec:
  hard:
    requests.cpu: "32"
    requests.memory: 64Gi
    pods: "200"
    persistentvolumeclaims: "50"
EOF

# 4. Apply LimitRange
kubectl apply -n "$NS" -f - <<EOF
apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
spec:
  limits:
    - type: Container
      default:
        cpu: 500m
        memory: 512Mi
      defaultRequest:
        cpu: 100m
        memory: 128Mi
EOF

# 5. Apply RBAC
kubectl create rolebinding "$TENANT-developers" \
  --clusterrole=edit \
  --namespace="$NS" \
  --group="$TENANT-developers"

The script applies the full stack. Every tenant goes through the same onboarding.

The isolation tests

flowchart TD
    A[Isolation tests] --> B[Cross-namespace network blocked]
    B --> C[Cross-namespace RBAC denied]
    C --> D[ResourceQuota enforced]
    D --> E[PSS blocks privileged]
    E --> F[External secrets not accessible]
    F --> G[Audit log]

The isolation tests:

1. Cross-namespace network blocked. A Pod in tenant A cannot reach a Pod in tenant B.
2. Cross-namespace RBAC denied. A user with RoleBinding in tenant A cannot list objects in tenant B.
3. ResourceQuota enforced. A namespace that exceeds its quota is rejected.
4. PSS blocks privileged. A Pod that requests privileged: true in a restricted namespace is rejected.
5. External secrets not accessible. A Pod in tenant A cannot mount a Secret from tenant B.

Each test verifies a different layer.

An isolation test script

#!/bin/bash
# isolation-test.sh

PASS=0
FAIL=0

# Test 1: cross-namespace network blocked
kubectl run --rm -it --image=busybox -n tenant-a test -- \
  wget --timeout=2 tenant-b-pod-ip 2>&1 > /tmp/test1
if grep -q "timeout" /tmp/test1; then
  echo "PASS: cross-namespace network blocked"
  PASS=$((PASS+1))
else
  echo "FAIL: cross-namespace network allowed"
  FAIL=$((FAIL+1))
fi

# Test 2: cross-namespace RBAC denied
kubectl auth can-i list pods --namespace=tenant-b \
  --as=tenant-a-user
# Should be "no"

# Test 3: ResourceQuota enforced
# Create a Pod that exceeds the quota
kubectl apply -n tenant-a -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: over-quota
spec:
  containers:
    - name: app
      image: myapp
      resources:
        requests:
          cpu: 1000  # exceeds the quota of 32 cores for the namespace
EOF
# Should be rejected by the quota admission controller

echo "Results: $PASS passed, $FAIL failed"

The script automates the isolation tests. Run nightly or in CI.

The audit cadence

flowchart LR
    A[Audit cadence] --> B["Daily: automated isolation tests"]
    B --> C["Weekly: review namespace list"]
    C --> D["Monthly: review RBAC bindings"]
    D --> E["Quarterly: review policies and quotas"]
    E --> F["Annually: external security review"]

The cadence:

• Daily. Automated isolation tests; alert on failures.
• Weekly. Review the namespace list; new tenants applied the full stack?
• Monthly. Review RBAC bindings; any over-permissioned identities?
• Quarterly. Review policies (NetworkPolicy, ResourceQuota, PSS); any drift?
• Annually. External security review; verify the multi-tenancy model is appropriate.

Quiz

The operational discipline

Tenant onboarding and isolation testing in production rest on five non-negotiable elements:

• Onboarding script. Every tenant goes through the same process.
• Runbook entry. Each tenant has documented quotas, policies, contacts.
• Automated isolation tests. Daily or in CI.
• Audit cadence. Daily, weekly, monthly, quarterly, annually.
• Drift detection. Alert when a tenant’s policies change unexpectedly.

Multi-tenancy is a process, not a one-time setup. The discipline is the process: onboarding, testing, auditing, drift detection.