Lab 21: Enforce Pod Security Standards

Objective

By the end of this lab you will have moved one workload from no security controls at all to restricted, and you will have done it in the order that does not cause an outage.

The thing you are really learning is that Pod Security admission has a delayed fuse. Labelling a namespace enforce: restricted does not stop a single running Pod. Everything keeps working. The rejection arrives at the next Pod creation — which might be a rollout you did on purpose, or might be a node reboot at 03:00 on a Sunday, three days after the label went on and completely disconnected from it in everyone’s mind.

You will reproduce that delay deliberately, find the error message in the place it actually appears, and then learn the one command that would have prevented the whole thing.

Architecture

Pod Security admission is a validating step inside the API server. It has no webhook, no controller, and no reconcile loop — which is exactly why it cannot act on Pods that already exist.

flowchart TB
    CREATE["CREATE pod<br/>(kubectl, ReplicaSet, DaemonSet, kubelet)"] --> PSA{"PodSecurity admission<br/>reads namespace labels"}
    PSA -->|"enforce: violated"| REJ["403 rejected<br/>message lists every violated control"]
    PSA -->|"audit: violated"| LOG["entry in the API server audit log"]
    PSA -->|"warn: violated"| WARN["Warning: on the client that made the call"]
    PSA -->|"passes"| OK["Pod object created"]
    RUNNING["Pods created before the label"] -.->|"never re-evaluated"| OK

Note who the callers are on that first line. Only one of them is you. The ReplicaSet controller creates Pods when a node dies, when a Pod is evicted under memory pressure, and when an HPA scales up — and it does so with no human present. That is where the deferred failure lands.

Requirements

A disposable kubeadm cluster: one control-plane node and two workers, Kubernetes 1.34.x. The cluster built in Lab 01 is exactly right. The PodSecurity admission plugin is enabled by default at this version and needs no API server configuration.
kubectl 1.34.x and jq on your workstation.
Cluster-admin, because you will label namespaces.
No SSH to the nodes is required, and nothing in this lab reconfigures the API server. One task creates a hostPath volume at /var/cache/legacy-app on whichever worker the Pod lands on; Cleanup removes the workload but not that directory, which is itself part of the lesson.
The manifests below pin busybox:1.36. Confirm the tag still resolves before you begin.

Scenario

A security review lands with one action: “enforce Pod Security Standards restricted in production”. An engineer works through the namespace list on a Friday afternoon, applies the label everywhere, watches kubectl get pods for a few minutes, sees everything Running, and closes the ticket.

On Monday morning a worker node is rebooted for a kernel update. The Pods that were on it are recreated elsewhere — except four of them, which are not recreated at all. The Deployments show the old replica count, no Pod is Pending, no Pod is CrashLoopBackOff, and kubectl describe deployment says nothing useful. The on-call engineer spends fifty minutes looking for a scheduling problem that does not exist.

Your job is to make that sequence happen on purpose, find the message in under a minute, and then build the workflow that would have caught it on the Friday.

Tasks

Task 1: Inventory what is enforced today

You cannot report on a migration you have not measured. Start with the labels that exist:

WORKDIR="$HOME/k8s-pss-lab"
mkdir -p "$WORKDIR"
cd "$WORKDIR"

kubectl get namespaces -o json | jq -r '
  ["NAMESPACE","ENFORCE","AUDIT","WARN","VERSION"],
  (.items[] | [
    .metadata.name,
    (.metadata.labels["pod-security.kubernetes.io/enforce"] // "-"),
    (.metadata.labels["pod-security.kubernetes.io/audit"]   // "-"),
    (.metadata.labels["pod-security.kubernetes.io/warn"]    // "-"),
    (.metadata.labels["pod-security.kubernetes.io/enforce-version"] // "-")
  ])
  | @tsv' | column -t | tee psa-inventory.txt

Read-only / Safeworkstation— Four PSA labels per namespace; a dash means the control is not set

$ kubectl get namespaces -o json | jq -r '.items[] | [.metadata.name, (.metadata.labels["pod-security.kubernetes.io/enforce"] // "-")] | @tsv'

default	-
kube-node-lease	-
kube-public	-
kube-system	privileged
kube-flannel	privileged

Illustrative output

Every dash is a namespace running with no Pod-level security policy at all. That is not a bug in your cluster: privileged is the effective default and always has been, because a cluster that rejected Pods on day one could not bootstrap itself. It does mean that “we run Pod Security Standards” is a claim about labels, not about the software, and the inventory above is the only honest way to check it.

Note which namespaces carry privileged explicitly. kube-system needs it — the control plane’s static Pods use host networking and host paths — and an explicit label there is better practice than leaving it blank, because it records a decision instead of an omission.

Task 2: Deploy a workload with no controls at all

legacy-app.yaml:

apiVersion: v1
kind: Namespace
metadata:
  name: legacy-app
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: legacy-app
  namespace: legacy-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: legacy-app
  template:
    metadata:
      labels:
        app: legacy-app
    spec:
      containers:
        - name: app
          image: busybox:1.36
          command: ["/bin/sh", "-c"]
          args:
            - |
              while true; do
                date -u +%Y-%m-%dT%H:%M:%SZ >> /cache/heartbeat.log
                sleep 10
              done
          volumeMounts:
            - name: cache
              mountPath: /cache
          resources:
            requests:
              cpu: 10m
              memory: 16Mi
            limits:
              memory: 64Mi
      volumes:
        - name: cache
          hostPath:
            path: /var/cache/legacy-app
            type: DirectoryOrCreate

There is no securityContext anywhere in that manifest, which is the default state of most manifests that predate a security review. The container therefore runs as UID 0, with the runtime’s default capability set, with privilege escalation permitted, with no seccomp profile, and with a writable directory on the node’s filesystem.

cd "$HOME/k8s-pss-lab"

kubectl apply -f legacy-app.yaml
kubectl rollout status deployment/legacy-app -n legacy-app --timeout=120s

kubectl exec -n legacy-app deploy/legacy-app -- id
kubectl exec -n legacy-app deploy/legacy-app -- tail -2 /cache/heartbeat.log

id returns uid=0(root). The heartbeat file is on the node, outside the container’s lifecycle, and it will still be there after you delete this entire namespace.

Task 3: Find out what would break, before changing anything

This is the command the engineer in the Scenario did not run. A namespace label change is validated by the PodSecurity admission plugin, and when you raise the enforced level, that plugin checks the Pods that already exist and warns about every one that would not have been admitted.

--dry-run=server sends the request through admission and discards it, which means you get the full warning list and change nothing:

Read-only / Safeworkstation— Evaluates every existing Pod in the namespace against restricted, then throws the change away

$ kubectl label --dry-run=server --overwrite ns legacy-app pod-security.kubernetes.io/enforce=restricted

Warning: existing pods in namespace "legacy-app" violate the new PodSecurity enforce level "restricted:latest"
Warning: legacy-app-6c9d4f7b8-2xk9p (and 1 other pod): allowPrivilegeEscalation != false, restricted volume types, runAsNonRoot != true, seccompProfile, unrestricted capabilities
namespace/legacy-app labeled (server dry run)

Illustrative output

Five violations, named, before anything changed. Run it across every namespace on the cluster and you have a migration worklist rather than a project plan:

cd "$HOME/k8s-pss-lab"

for ns in $(kubectl get ns -o jsonpath='{.items[*].metadata.name}'); do
  echo "=== $ns ==="
  kubectl label --dry-run=server --overwrite ns "$ns" \
    pod-security.kubernetes.io/enforce=restricted 2>&1 \
    | grep -v 'server dry run'
done | tee would-break.txt

grep -c 'Warning:' would-break.txt

Task 4: Reproduce the deferred failure

Now do what the engineer did. Set enforce: restricted for real and watch nothing happen:

cd "$HOME/k8s-pss-lab"

kubectl label --overwrite ns legacy-app \
  pod-security.kubernetes.io/enforce=restricted

kubectl get pods -n legacy-app
kubectl exec -n legacy-app deploy/legacy-app -- tail -1 /cache/heartbeat.log

Both Pods are still Running. The heartbeat is still being written. The workload is in flagrant violation of the policy the namespace now enforces, and the cluster is entirely content, because admission is a gate on creation and these Pods were created before the gate existed.

There is a second half to this, and it is the part that turns a surprise into an outage. enforce applies to Pod objects only. A Deployment, a DaemonSet, a StatefulSet, a Job or a CronJob whose template violates the profile is admitted without complaint — the object is not a Pod, so the enforcing check does not run on it. Prove it to yourself before continuing:

cd "$HOME/k8s-pss-lab"

kubectl apply -f legacy-app.yaml 2>&1 | tee -a violations-log.txt

The apply succeeds. Nothing about it fails, and it will keep succeeding every time CI re-applies that manifest, for as long as nobody looks. The warn label is what makes the same apply print a Warning: line naming the violations — which is why setting enforce without warn produces a cluster that rejects Pods and never tells anybody who is creating them.

Now be the node reboot:

kubectl delete pod -n legacy-app -l app=legacy-app
sleep 10
kubectl get pods -n legacy-app
kubectl get deployment -n legacy-app

The Pods are gone and nothing replaced them. The Deployment reports 0 of 2 ready. There is no Pod in Pending, no Pod in Error, nothing to kubectl describe pod — because no Pod object was ever created. This is the state that costs fifty minutes.

The message exists. It is one level down, on the object that tried to make the Pod:

cd "$HOME/k8s-pss-lab"

kubectl get events -n legacy-app --field-selector reason=FailedCreate \
  --sort-by=.lastTimestamp | tail -3 | tee -a violations-log.txt

kubectl describe replicaset -n legacy-app | grep -A5 FailedCreate

Read-only / Safeworkstation— The rejection, on the ReplicaSet, listing every violated control

$ kubectl describe replicaset -n legacy-app

Events:
Type     Reason        Age   From                   Message
----     ------        ----  ----                   -------
Warning  FailedCreate  12s   replicaset-controller  Error creating: pods "legacy-app-6c9d4f7b8-t4m2v" is forbidden: violates PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "app" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "app" must set securityContext.capabilities.drop=["ALL"]), restricted volume types (volume "cache" uses restricted volume type "hostPath"), runAsNonRoot != true (pod or container "app" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "app" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")

Illustrative output

Three places to look, in this order

When a workload silently stops replacing its Pods, check in this order and you will almost never spend more than a minute:

kubectl get events -n NAMESPACE --field-selector reason=FailedCreate — the controller’s own complaint.
kubectl describe replicaset -n NAMESPACE (or daemonset, job, statefulset) — the same message, attached to the object that owns the Pod template.
kubectl get ns NAMESPACE --show-labels — the policy that is being applied, which tells you whether the answer is “fix the workload” or “somebody labelled this namespace and did not say so”.

kubectl describe deployment is deliberately not on that list. The Deployment delegates Pod creation to the ReplicaSet, so the failure is recorded one level below where most people look for it.

Task 5: Establish where the floor actually is

Before fixing everything at once, find out how far down the workload falls. Drop the namespace to baseline and try again:

cd "$HOME/k8s-pss-lab"

kubectl label --overwrite ns legacy-app \
  pod-security.kubernetes.io/enforce=baseline

kubectl rollout restart deployment/legacy-app -n legacy-app
sleep 10
kubectl get events -n legacy-app --field-selector reason=FailedCreate \
  --sort-by=.lastTimestamp | tail -1 | tee -a violations-log.txt

Still refused, and now for exactly one reason: the hostPath volume. baseline forbids spec.volumes[*].hostPath outright — there is no list of permitted paths, and mounting /var/cache/anything is refused just as firmly as mounting /. Running as root, keeping the default capability set, and having no seccomp profile are all acceptable at baseline; touching the node’s filesystem is not.

That single fact reorders the migration. The securityContext fields are five-line edits that a competent engineer can make to a manifest in an afternoon. hostPath is a design change: something on the node was being used as storage, and now it will not be, so the question “what was in that directory and who needed it” has to be answered by a person rather than by a YAML edit.

Do that one first, because it is the one that can fail on its own merits.

legacy-app-baseline.yaml — identical to the original except for the volume:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: legacy-app
  namespace: legacy-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: legacy-app
  template:
    metadata:
      labels:
        app: legacy-app
    spec:
      containers:
        - name: app
          image: busybox:1.36
          command: ["/bin/sh", "-c"]
          args:
            - |
              while true; do
                date -u +%Y-%m-%dT%H:%M:%SZ >> /cache/heartbeat.log
                sleep 10
              done
          volumeMounts:
            - name: cache
              mountPath: /cache
          resources:
            requests:
              cpu: 10m
              memory: 16Mi
            limits:
              memory: 64Mi
      volumes:
        - name: cache
          # Was: hostPath /var/cache/legacy-app. Two consequences of the
          # change, both deliberate:
          #   1. the heartbeat no longer survives a Pod restart
          #   2. the heartbeat no longer accumulates on the node forever
          emptyDir: {}

cd "$HOME/k8s-pss-lab"

kubectl apply -f legacy-app-baseline.yaml
kubectl rollout status deployment/legacy-app -n legacy-app --timeout=120s
kubectl exec -n legacy-app deploy/legacy-app -- tail -1 /cache/heartbeat.log

Keep legacy-app.yaml unchanged on disk. It is the regression test: at the end of the lab you will apply it again, expect it to be refused, and only then have evidence that the policy is doing something.

The workload now passes baseline, and the heartbeat file is inside the container, which means it starts empty after every restart. Say that out loud, because it is the cost — on a real workload the equivalent statement is “the cache is cold after every rollout” or “the queue spool is lost on eviction”, and somebody has to agree to it before the change ships.

Task 6: Climb to restricted, one violation at a time

Now put the namespace back to restricted and add the controls. Do it in one edit but read the list first, because the five entries in the error message map one-to-one onto the five fields you are about to add.

legacy-app-hardened.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: legacy-app
  namespace: legacy-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: legacy-app
  template:
    metadata:
      labels:
        app: legacy-app
    spec:
      securityContext:
        # restricted: "runAsNonRoot != true"
        runAsNonRoot: true
        runAsUser: 1000
        runAsGroup: 1000
        # Not demanded by restricted. Makes the emptyDir writable by the
        # non-root UID above, which the profile change created a need for.
        fsGroup: 1000
        # restricted: "seccompProfile". Pod-level, so it covers every
        # container without repetition.
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: app
          image: busybox:1.36
          command: ["/bin/sh", "-c"]
          args:
            - |
              while true; do
                date -u +%Y-%m-%dT%H:%M:%SZ >> /cache/heartbeat.log
                sleep 10
              done
          securityContext:
            # restricted: "allowPrivilegeEscalation != false". Must be
            # per-container; there is no Pod-level equivalent.
            allowPrivilegeEscalation: false
            # restricted: "unrestricted capabilities". Also per-container.
            capabilities:
              drop: ["ALL"]
            # Not part of restricted, and Pod Security admission never
            # checks it. Good hardening; enforce it with a policy engine if
            # it matters.
            readOnlyRootFilesystem: true
          volumeMounts:
            - name: cache
              mountPath: /cache
          resources:
            requests:
              cpu: 10m
              memory: 16Mi
            limits:
              memory: 64Mi
      volumes:
        # restricted: "restricted volume types". emptyDir is one of the
        # eight permitted types.
        - name: cache
          emptyDir: {}

Test it before you apply it. Because a Deployment is not a Pod, the check you are looking for is a Warning: line and not a failure — which is exactly why all three labels go on together below:

cd "$HOME/k8s-pss-lab"

kubectl label --overwrite ns legacy-app \
  pod-security.kubernetes.io/enforce=restricted \
  pod-security.kubernetes.io/audit=restricted \
  pod-security.kubernetes.io/warn=restricted

kubectl apply -f legacy-app-hardened.yaml --dry-run=server -o name

kubectl apply -f legacy-app-hardened.yaml
kubectl rollout status deployment/legacy-app -n legacy-app --timeout=120s

kubectl exec -n legacy-app deploy/legacy-app -- id
kubectl exec -n legacy-app deploy/legacy-app -- tail -1 /cache/heartbeat.log

id now returns uid=1000. The heartbeat is still being written, which proves fsGroup did its job — without it the non-root process cannot write into the volume, and you would have a Pod that passes the policy and fails its own purpose. That is the most common way a restricted migration goes wrong after the manifest is technically correct.

What could go wrong?

What happens if the container image only works as root?

The Pod is admitted — runAsNonRoot: true and runAsUser: 1000 are accepted by the profile — and then fails at startup, usually with a permission error on a path the image expects to own. Pod Security admission validates the manifest, not the image. The fix is an image change: a USER directive, world-writable paths for the data the process writes, and a port above 1024 or NET_BIND_SERVICE if it binds a privileged one.

What happens if you set enforce but leave warn and audit unset?

Enforcement still works, and you lose the two mechanisms that tell you about near-misses. A Pod that violates the level is rejected loudly; a Pod that would violate a stricter level than you enforce passes in silence. Setting all three to the same value, as this task does, is the configuration that makes the enforcement decision legible later.

What happens if audit: restricted is set but the cluster has no audit log?

Nothing is recorded, and nobody notices. Audit-mode violations are written to the API server’s audit log; if the API server was started without --audit-policy-file and --audit-log-path, the entries go nowhere. Lab 20 turns that on. A team that sets audit: restricted on a cluster with no audit logging has a label, a belief, and no data.

What happens if a workload passes restricted but mounts a Secret?

It is admitted. secret and projected are both on the permitted volume list, and the profile has nothing to say about what the Secret contains or who can read it. Pod Security Standards constrain what a Pod can do to the node; RBAC constrains what it can do to the API. They do not overlap, and neither one covers the other’s ground.

Task 7: Build the exception the node agent genuinely needs

Some workloads have to touch the host. A log collector reading /var/log/pods, a node exporter reading /proc, a CSI driver — these are not badly written applications, and no amount of manifest polishing will get them through restricted. The professional answer is not to weaken the production namespaces; it is a separate namespace with an explicit exception that stays visible.

node-agent.yaml:

apiVersion: v1
kind: Namespace
metadata:
  name: node-agents
  labels:
    # The exception. Explicit, so it appears in the inventory as a decision
    # rather than as an absence.
    pod-security.kubernetes.io/enforce: privileged
    # Enforcement is off; visibility is not. Every Pod here still produces a
    # warning and an audit entry against baseline, so the exception cannot
    # quietly grow.
    pod-security.kubernetes.io/audit: baseline
    pod-security.kubernetes.io/warn: baseline
  annotations:
    exception/reason: 'Reads container logs from /var/log/pods, which baseline forbids.'
    exception/owner: 'platform-team'
    exception/review-by: '2027-02-01'
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-agent
  namespace: node-agents
spec:
  selector:
    matchLabels:
      app: node-agent
  template:
    metadata:
      labels:
        app: node-agent
    spec:
      containers:
        - name: agent
          image: busybox:1.36
          command: ["/bin/sh", "-c"]
          args:
            - |
              while true; do
                ls /var/log/pods | wc -l
                sleep 30
              done
          securityContext:
            # The exception is the volume, and nothing else. Everything the
            # workload can meet, it meets.
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
            readOnlyRootFilesystem: true
          volumeMounts:
            - name: pod-logs
              mountPath: /var/log/pods
              readOnly: true
          resources:
            requests:
              cpu: 10m
              memory: 16Mi
            limits:
              memory: 64Mi
      volumes:
        - name: pod-logs
          hostPath:
            path: /var/log/pods
            type: Directory

cd "$HOME/k8s-pss-lab"

kubectl apply -f node-agent.yaml 2>&1 | tee -a violations-log.txt
kubectl rollout status daemonset/node-agent -n node-agents --timeout=120s
kubectl logs -n node-agents -l app=node-agent --tail=1

The apply prints a warning naming the hostPath violation against baseline, and then succeeds, because enforce is privileged. That combination is the point: the workload runs, and the exception announces itself every single time a Pod is created. An exception that is silent is an exception that becomes permanent.

Note also that the DaemonSet still drops every capability, still forbids privilege escalation, and mounts the host path read-only. “This workload needs an exception” almost never means “this workload needs everything”. The exception should be exactly one control wide, and the manifest should make it obvious which one.

Task 8: Pin the version, so a cluster upgrade cannot tighten it underneath you

The restricted profile is versioned. A future Kubernetes release can add a control, and a namespace that says enforce: restricted with no version follows the cluster’s own version — so an upgrade can start rejecting a workload that has not changed.

cd "$HOME/k8s-pss-lab"

kubectl label --overwrite ns legacy-app \
  pod-security.kubernetes.io/enforce-version=v1.34 \
  pod-security.kubernetes.io/audit-version=v1.34 \
  pod-security.kubernetes.io/warn-version=v1.34

kubectl get ns legacy-app --show-labels

This is a trade-off with two honest sides. Pinning means an upgrade cannot break the workload without a deliberate label change, which is what you want for anything that must not fail unattended. It also means the workload stops getting new protections automatically, so the pin needs a review date in the same place the exception annotations live. A pin with no owner is how a cluster ends up enforcing a four-year-old profile while its documentation claims otherwise.

Validation

Run these against the finished state. Each one proves an outcome rather than restating a step.

cd "$HOME/k8s-pss-lab"

# 1. The hardened workload runs, as a non-root user.
kubectl exec -n legacy-app deploy/legacy-app -- id | grep -q 'uid=1000' \
  && echo "PASS: running as UID 1000"

# 2. It still does its job - the policy did not break the application.
kubectl exec -n legacy-app deploy/legacy-app -- test -s /cache/heartbeat.log \
  && echo "PASS: heartbeat file is non-empty"

# 3. The namespace is enforcing, auditing and warning at the same level.
kubectl get ns legacy-app -o json \
  | jq -r '.metadata.labels | to_entries[] | select(.key | startswith("pod-security")) | "\(.key)=\(.value)"'

# 4. A regression is flagged. The apply SUCCEEDS - it is a Deployment, not a
#    Pod - and the warning is the whole signal, which is why warn is set.
kubectl apply -f legacy-app.yaml --dry-run=server 2>&1 | grep -qi 'violate PodSecurity' \
  && echo "PASS: the original manifest is warned about"

# 5. The exception namespace runs its workload AND announces the violation.
kubectl apply -f node-agent.yaml --dry-run=server 2>&1 | grep -q 'Warning' \
  && echo "PASS: the exception is still visible"

# 6. No namespace on the cluster is unlabelled by accident.
grep -c ' - ' psa-inventory.txt

Check 4 is the one that matters most and the one most likely to be skipped. A migration that ends without checking the old manifest has proved that the new manifest works, which is a different and much weaker statement. Note what check 4 does not do: it does not fail. The apply of a violating Deployment is always admitted, so the warning is the entire signal, and a pipeline that only looks at exit codes will never see it. Check 5 proves the exception did not become invisible.

Expected Outcome

k8s-pss-lab/
├── legacy-app.yaml               (the original; kept as the regression test)
├── legacy-app-baseline.yaml
├── legacy-app-hardened.yaml
├── node-agent.yaml
├── psa-inventory.txt
├── violations-log.txt
└── would-break.txt

On the cluster: a legacy-app namespace enforcing, auditing and warning at restricted, pinned to v1.34, running a workload that meets every control; and a node-agents namespace whose single documented exception is a read-only hostPath, with baseline audit and warn still switched on.

Production notes

Two changes live in this lab and they belong in different change windows.

Labelling a namespace is a low-risk change with a delayed blast radius. It cannot break anything at the moment you apply it, which is exactly what makes it dangerous: the normal validation — apply it, watch for a few minutes, see everything healthy — returns a false pass. The acceptance criterion for this change is not “the Pods are still running”. It is “a --dry-run=server sweep of the namespace returns no warnings”, which is a statement about what will happen next time rather than about now.

Changing a workload’s securityContext is a normal rollout with a normal blast radius, and it should be treated as one: staged environment first, one replica at a time, and an explicit check that the application still writes what it is supposed to write. The failure mode is not the admission decision — that is deterministic and testable — but the application discovering at runtime that it can no longer write to a path it has always owned.

The ordering that avoids an outage is: warn first, on its own, for long enough to cover the workload’s slowest cycle; fix what warns; sweep with --dry-run=server until it is clean; then enforce. Every step in that sequence is reversible by removing a label, and the sequence is the whole value — an engineer who goes straight to enforce has done the same work in an order that hides the result until a controller acts on it.

Hold when you cannot see the workloads. If the namespace contains CronJobs that have not fired, scaled-to-zero Deployments, or anything whose Pods you have not observed being created, you do not have the evidence to enforce. Record the inventory, leave warn in place, name an owner and a date, and come back when a full cycle has run. A namespace stuck at warn with a named owner is a project in progress. A namespace at enforce with an unobserved CronJob is an incident with a delay fuse.

Troubleshooting

The Pods stay Running after you set enforce: restricted. That is correct behaviour, not a broken label. Pod Security admission never re-evaluates an existing Pod. Delete one to see the policy act.

kubectl label --dry-run=server produces no warnings but the rollout is rejected. The dry run evaluated the Pods that existed then; the rollout creates a Pod from the template, which may differ — an init container, a sidecar injected by a mutating webhook, or a different image tag. Test the manifest with kubectl apply --dry-run=server as well as the label.

The Pod is admitted but crashes immediately with a permission error. The policy passed and the image did not. runAsNonRoot changed the UID; the process is now trying to write somewhere it does not own. Add fsGroup for volumes, or fix the image’s paths.

fsGroup did not make the volume writable. It applies to volume types that support ownership management. emptyDir, secret, configMap and most CSI volumes do; a few CSI drivers do not, and hostPath never does.

The DaemonSet in node-agents is rejected anyway. Check the namespace labels actually applied — kubectl get ns node-agents --show-labels. A kubectl apply of the Namespace object does not overwrite labels added by hand unless the manifest lists them, and a namespace created earlier in the lab may still be carrying an older label.

A warning appears about a Pod you did not create. Something else in the namespace is making Pods: a Job, a CronJob, or a controller from an operator. kubectl get pods -n NAMESPACE -o json | jq -r '.items[].metadata.ownerReferences[]?.kind' tells you who owns them.

kubectl get events shows nothing. Events expire, by default after an hour. Read the object instead: kubectl describe replicaset keeps the message for as long as the ReplicaSet exists.

Cleanup

Nothing here is destructive to the cluster, but one artefact outlives the cleanup by design and you should see it do so.

Step 1. Remove the workloads and their namespaces:

cd "$HOME/k8s-pss-lab"

kubectl delete -f node-agent.yaml --ignore-not-found
kubectl delete namespace legacy-app --ignore-not-found
kubectl delete namespace node-agents --ignore-not-found

kubectl get ns legacy-app node-agents 2>&1 | grep -q NotFound \
  && echo "namespaces removed"

Step 2. Confirm the labels you added are gone, and that nothing else was touched:

cd "$HOME/k8s-pss-lab"

kubectl get namespaces -o json | jq -r '
  .items[] | [.metadata.name,
    (.metadata.labels["pod-security.kubernetes.io/enforce"] // "-")] | @tsv' \
  | column -t | diff psa-inventory.txt - | head -20 \
  || echo "PSA labels match the starting inventory"

The diff will show column differences because the inventory has five columns and this check has two — read the namespace list, not the formatting. What must not appear is a namespace you did not touch carrying a label it did not have in Task 1.

Step 3. Deal with the hostPath residue. Deleting the namespace did not remove it:

This deletes a directory on a node, over SSH, as root

/var/cache/legacy-app is on whichever worker ran the Pod, and only that worker. The command below removes it. Confirm the path before you run it — a mistyped rm -rf under /var on a Kubernetes node can take out the kubelet’s own state directory, and there is no undo.

Find the node first, then look at the directory, then delete it:

kubectl get pods -n legacy-app -o wide 2>/dev/null || echo "namespace already deleted - use the node you noted in Task 2"

# Substitute the worker that ran the Pod:
NODE=192.0.2.12
ssh "$NODE" 'ls -la /var/cache/legacy-app'

Destructiveworker— Removes the hostPath directory the workload created; verify the path first

$ sudo rm -rf /var/cache/legacy-app

Keep psa-inventory.txt, would-break.txt, legacy-app-hardened.yaml and violations-log.txt — they are the deliverables. The rest of the working directory can go.

Under the hood

Pod Security admission is a built-in validating admission plugin, compiled into the API server and enabled by default since v1.25. It registers for Pod create and update, and for Namespace create and update. There is no webhook, so there is no network call, no certificate to rotate and no failure mode where an unreachable policy service takes the cluster down — which is the main reason it exists in this form rather than as the PodSecurityPolicy controller it replaced.

On a Pod request, the plugin resolves the Pod’s namespace, reads the three mode labels and their optional version labels, and evaluates the Pod spec against each level independently. The profiles themselves are a fixed table of controls in the API server’s source: a control names the fields it restricts (spec.securityContext.runAsNonRoot, spec.containers[*].securityContext.capabilities.drop, and so on) and the values it permits. Evaluation is a pure function of the Pod spec and the profile version — no cluster state, no ordering, no partial results — which is what makes --dry-run=server a trustworthy preview rather than an approximation.

The version labels select which table is used. Each Kubernetes release freezes the profile definitions for that version, so restricted pinned to v1.34 continues to mean what it meant in v1.34 even on a much later cluster. That is why pinning is safe against upgrades and also why it silently withholds new controls.

The namespace handler is the piece that produces the Task 3 behaviour. When a namespace’s labels change in a way that raises the enforced level, the plugin lists the Pods already in that namespace, evaluates each against the new level, and attaches a warning to the response for every one that would fail. It cannot do more than warn: the Pods exist, admission has no mechanism to un-admit them, and evicting workloads as a side effect of a label edit would be a far worse behaviour than the deferred failure you reproduced in Task 4. The warning is the whole notification, and it is delivered once, to whoever ran the command.

What You Learned

Pod Security admission has a delayed fuse. Labelling a namespace changes nothing about running Pods and everything about the next Pod creation, which may be triggered days later by a node reboot rather than by a person.
enforce applies to Pods and to nothing else. A Deployment whose template violates the profile is admitted every time; the rejection happens later, when a controller turns that template into a Pod. warn is what makes the bad apply visible at the moment it happens.
The rejection is on the ReplicaSet, not the Deployment and not a Pod. When a workload stops replacing Pods with no Pod to inspect, go to the controller’s events first.
kubectl label --dry-run=server is the whole prevention. It evaluates every existing Pod against the level you are considering and changes nothing, and a loop over every namespace turns a migration into a worklist.
baseline forbids hostPath outright, with no permitted-path list, so the volume is the migration’s real work and the securityContext fields are the easy part.
restricted does not check readOnlyRootFilesystem. Set it anyway; enforce it with a policy engine if it has to be enforced.
Passing the policy is not the same as working. runAsNonRoot moved the UID, and without fsGroup the application could no longer write to its own volume — admitted, healthy-looking, and broken.
An exception should be one control wide, in its own namespace, with audit and warn still on, so that it announces itself on every Pod creation instead of quietly becoming the new normal.

Why this matters

The engineer in the Scenario did nothing careless. They applied a well-understood control, checked that the workloads were healthy, and closed the ticket — and the check they ran was incapable of detecting the problem they had created, because the software they were configuring is specifically designed not to act on anything that already exists.

That is the shape of most Kubernetes security incidents that are not breaches: a control applied correctly, validated with the wrong question, and detonating later under someone else’s pager. The defence is a habit rather than a tool. Before any admission-time policy goes on, ask what will happen the next time a controller creates a Pod, and then find a way to answer that question today — which for Pod Security Standards is a single command with --dry-run=server in it.

Objectives

Prerequisites

Objective

Architecture

Requirements

Scenario

Tasks

Task 1: Inventory what is enforced today

Task 2: Deploy a workload with no controls at all

Task 3: Find out what would break, before changing anything

Task 4: Reproduce the deferred failure

Task 5: Establish where the floor actually is

Task 6: Climb to restricted, one violation at a time

Task 7: Build the exception the node agent genuinely needs

Task 8: Pin the version, so a cluster upgrade cannot tighten it underneath you

Validation

Expected Outcome

Production notes

Troubleshooting

Cleanup

What You Learned

Deliverables

Verification status