Soft multi-tenancy with RBAC, NetworkPolicy, and Quotas — the production stack

The soft multi-tenancy production stack combines RBAC, NetworkPolicy, ResourceQuota, and PSS. This lesson walks each layer, the integration, the anti-patterns, and the operational discipline.

The stack

flowchart LR
    A[Production stack] --> B[RBAC]
    A --> C[NetworkPolicy]
    A --> D[ResourceQuota]
    A --> E[PSS]
    B --> B1["Identity: who can do what"]
    C --> C1["Traffic: who can talk to who"]
    D --> D1["Resources: how much can be used"]
    E --> E1["Workloads: what can run"]

The four layers:

• RBAC. Identity and authorisation. Who can create, read, update, delete objects.
• NetworkPolicy. Network traffic. Who can talk to whom.
• ResourceQuota. Compute and storage limits. How much can be used.
• PSS. Workload security. What kinds of Pods can run.

Each layer addresses a different isolation requirement. Missing any one breaks the stack.

RBAC for multi-tenancy

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: tenant-a-developer
  namespace: tenant-a-prod
rules:
  - apiGroups: ["", "apps", "batch"]
    resources: ["pods", "services", "configmaps", "deployments", "statefulsets", "jobs"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: tenant-a-developers
  namespace: tenant-a-prod
subjects:
  - kind: Group
    name: tenant-a-developers
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: tenant-a-developer
  apiGroup: rbac.authorization.k8s.io

RBAC best practices for multi-tenancy:

• Roles are namespace-scoped. Use Roles, not ClusterRoles, for tenant permissions.
• Subjects are Groups. Bind to groups, not individual users.
• No wildcards in verbs or resources. Be specific.
• ClusterRoles only for platform components. cert-manager, Argo CD, etc.

NetworkPolicy for multi-tenancy

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: tenant-a-prod
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-same-namespace
  namespace: tenant-a-prod
spec:
  podSelector: {}
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector: {}  # any pod in the same namespace
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-egress
  namespace: tenant-a-prod
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
      ports:
        - port: 53
          protocol: UDP
        - port: 53
          protocol: TCP

NetworkPolicy best practices:

• Default-deny first. Block all traffic.
• Explicit allow rules. Allow same-namespace traffic, DNS egress, ingress from specific sources.
• No default-allow. A cluster without NetworkPolicy is default-allow; add explicit policies.

ResourceQuota for multi-tenancy

apiVersion: v1
kind: ResourceQuota
metadata:
  name: tenant-a-quota
  namespace: tenant-a-prod
spec:
  hard:
    requests.cpu: "32"
    requests.memory: 64Gi
    limits.cpu: "64"
    limits.memory: 128Gi
    pods: "200"
    persistentvolumeclaims: "50"
    requests.storage: 1Ti
    services: "100"
    secrets: "100"
    configmaps: "100"
    count/deployments.apps: "50"
    count/statefulsets.apps: "20"

ResourceQuota best practices:

• Bound compute. CPU and memory requests and limits.
• Bound storage. requests.storage for PVCs.
• Bound object counts. pods, services, secrets, configmaps, etc.
• Per-namespace. Each tenant has its own quota.

PSS for multi-tenancy

apiVersion: v1
kind: Namespace
metadata:
  name: tenant-a-prod
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: latest
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/audit-version: latest
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/warn-version: latest

PSS levels:

• privileged. Unrestricted; no security controls. Avoid.
• baseline. Minimal restrictions; prevents known privilege escalations.
• restricted. Strong restrictions; hardened baseline.

Production namespaces should use restricted.

The integration

flowchart LR
    A[User authenticates] --> B[kubectl]
    B --> C{API server: RBAC check}
    C -->|Denied| D[403 Forbidden]
    C -->|Allowed| E[Object created]
    E --> F{Admission: PSS check}
    F -->|Failed| G[Rejected]
    F -->|Passed| H[Object stored]
    H --> I{Pod scheduled}
    I --> J{CNI: NetworkPolicy check}
    J -->|Denied| K[Traffic blocked]
    J -->|Allowed| L[Traffic flows]

The layers integrate:

1. RBAC checks the user’s identity and permissions.
2. Admission (including PSS) checks the object’s spec against policy.
3. The scheduler places the Pod on a node.
4. The CNI enforces NetworkPolicy.

A failure at any layer blocks the action. The discipline is to apply every layer.

Quiz

The operational discipline

The multi-tenancy stack in production rests on five non-negotiable elements:

• Apply every layer. RBAC, NetworkPolicy, ResourceQuota, PSS. None is optional.
• Default-deny NetworkPolicy. Block by default; allow explicitly.
• PSS restricted. No privileged containers in production namespaces.
• Test every layer. Quarterly: verify each layer is enforced.
• Audit policies. Periodic review of every namespace.

The stack is a system. Missing any layer breaks the system. The discipline is to apply every layer and test each one.