vCluster and Kyverno — hard multi-tenancy and per-tenant policy

vCluster and Kyverno extend the multi-tenancy options. This lesson walks vCluster for hard multi-tenancy within a cluster, Kyverno for per-tenant policy, the integration, and the operational discipline.

vCluster for hard multi-tenancy

flowchart LR
    A[Host cluster] --> B["vCluster: tenant-a"]
    A --> C["vCluster: tenant-b"]
    B --> B1["API server: tenant-a"]
    B --> B2["etcd: tenant-a"]
    B --> B3["Scheduler: tenant-a"]
    C --> C1["API server: tenant-b"]
    C --> C2["etcd: tenant-b"]
    C --> C3["Scheduler: tenant-b"]
    B -->|schedules on| F[Host nodes]
    C -->|schedules on| F

vCluster runs a virtual Kubernetes control plane inside a host cluster. Each tenant gets:

• Its own API server (running as a Pod in the host).
• Its own etcd (running as a Pod in the host).
• Its own scheduler and controller manager.

Pods from the virtual cluster schedule on the host cluster’s nodes. The tenants are isolated at the control-plane level: they cannot see each other’s objects.

A vCluster example

# Create a vCluster
vcluster create tenant-a --namespace tenant-a-vcluster

# Use the vCluster's kubeconfig
vcluster connect tenant-a --namespace tenant-a-vcluster

# Inside the vCluster, create a namespace
kubectl create namespace my-app

# Outside, in the host cluster, the vCluster Pods are visible
kubectl get namespaces -n tenant-a-vcluster

The vCluster is managed by a Helm chart; the vcluster CLI creates, connects, and deletes vClusters.

Kyverno for per-tenant policy

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-labels
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: check-for-labels
      match:
        any:
          - resources:
              kinds:
                - Pod
              namespaceSelector:
                matchLabels:
                  tenant: tenant-a
      validate:
        message: "The labels 'app' and 'owner' are required."
        pattern:
          metadata:
            labels:
              app: "?*"
              owner: "?*"

Kyverno policies:

• Validate. Block objects that don’t match the pattern.
• Mutate. Modify objects to match the pattern.
• Generate. Create additional objects based on the input.

Policies can be scoped to specific namespaces via namespaceSelector. A Kyverno policy can apply different rules to different tenants.

The integration

flowchart LR
    A[Tenant creates Pod] --> B[API server]
    B --> C["Admission webhook: Kyverno"]
    C --> D{Validates?}
    D -->|No| E[Rejected]
    D -->|Yes| F[Object stored]
    F --> G["vCluster: schedules on host"]
    G --> H["CNI: NetworkPolicy check"]
    H -->|Allowed| I[Pod runs]

The integration:

1. The tenant creates a Pod in the vCluster’s namespace.
2. The vCluster’s API server receives the request.
3. Kyverno (via admission webhook) validates the Pod.
4. The Pod is stored in the vCluster’s etcd.
5. The vCluster’s scheduler places the Pod on the host cluster’s nodes.
6. The host cluster’s CNI enforces NetworkPolicy.

Each layer enforces its policy. The combination provides defense in depth.

The operational trade-offs

flowchart LR
    A[vCluster] --> B[+ Hard isolation at control plane]
    A --> C[+ Cheaper than separate clusters]
    A --> D[- Adds operational complexity]
    A --> E[- Tenants share host nodes]
    F[Kyverno] --> G[+ Per-tenant policy]
    G --> H[+ YAML-based syntax]
    G --> I[- Requires webhook infrastructure]

The trade-offs:

vCluster:

• Pros: Hard isolation at control plane; cheaper than separate clusters.
• Cons: Operational complexity; tenants share host nodes (so node failure affects all).

Kyverno:

• Pros: Per-tenant policy; YAML-based syntax; validation, mutation, generation.
• Cons: Requires webhook infrastructure.

Quiz

The operational discipline

vCluster and Kyverno in production rest on five non-negotiable elements:

• vCluster for untrusted tenants. Untrusted tenants need control-plane isolation.
• Kyverno for policy enforcement. Per-tenant rules that the soft stack cannot express.
• NetworkPolicy still required. vCluster does not replace NetworkPolicy.
• Test isolation. Quarterly: verify tenants cannot access each other’s resources.
• Document the topology. vCluster and Kyverno are part of the architecture diagram.

vCluster and Kyverno extend multi-tenancy. The discipline is to choose the right tool for the use case and to test the isolation.