PSS migration — moving workloads to restricted

Migrating workloads to Pod Security Standards restricted is a multi-quarter exercise. The discipline is per-namespace, multi-step, and backed by audit tools. This lesson covers the systematic approach, the tools, the common fixes, and the production failure modes.

The migration phases

Five phases, in order:

1. Audit. Identify the violations across every namespace and workload.
2. Categorise. Group the violations by type and effort to fix.
3. Fix. Address each violation in the workload’s manifest.
4. Test. Run the fixed workload in a staging namespace with enforce: restricted.
5. Enforce. Switch the namespace’s label to enforce: restricted.

flowchart LR
    A[Audit] --> B[Categorise]
    B --> C[Fix]
    C --> D[Test]
    D --> E[Enforce]
    E --> F{Next namespace}
    F -->|yes| A

Each phase is per-namespace; the team tracks progress in a spreadsheet.

The audit tools

Three tools support the audit:

1. kube-linter. Lints Kubernetes manifests against a set of rules including PSS. kube-linter lint manifest.yaml returns violations.

kube-linter lint manifest.yaml
# error: host-network-share (deployment "api" uses host network)
# error: privilege-escalation (container "api" allows privilege escalation)
# error: run-as-non-root (container "api" does not have runAsNonRoot set)

2. Kyverno. Has a built-in policy for each PSS profile. kubectl apply -f pss-restricted.yaml runs the policy in Audit mode.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: pss-restricted
spec:
  validationFailureAction: Audit  # warn but allow
  rules:
  - name: restricted
    match:
      resources:
        kinds: ["Pod"]
    validate:
      message: "violates PodSecurity 'restricted:latest'"
      pattern:
        spec:
          containers:
          - securityContext:
              allowPrivilegeEscalation: false
              capabilities:
                drop: ["ALL"]
              readOnlyRootFilesystem: true
              runAsNonRoot: true

3. polaris. Dashboard for workload configuration; flags PSS violations alongside other best practices.

polaris audit --audit-format=pretty
# api: 8/12 checks passing
#   - runAsNonRoot: fail
#   - readOnlyRootFilesystem: fail
#   - allowPrivilegeEscalation: pass

The common fixes

Five patterns recur:

1. Run as non-root. Add securityContext.runAsNonRoot: true and set runAsUser to a non-zero UID. Update the container image’s USER directive if necessary.

securityContext:
  runAsNonRoot: true
  runAsUser: 1000
  runAsGroup: 1000
  fsGroup: 1000

2. Replace hostPath with CSI or emptyDir. A workload that uses hostPath for cache mounts can use a CSI cache volume; for ephemeral data, use emptyDir.

volumes:
- name: cache
  csi:
    driver: csi.example.com
    volumeAttributes:
      type: cache

3. Drop ALL capabilities. Add capabilities.drop: ["ALL"] and only add what is needed (typically NET_BIND_SERVICE for non-root binding).

securityContext:
  capabilities:
    drop: ["ALL"]
    add: ["NET_BIND_SERVICE"]  # only if needed

4. Add seccompProfile. Set seccompProfile.type to RuntimeDefault (the runtime’s default profile) or Localhost (a custom profile).

securityContext:
  seccompProfile:
    type: RuntimeDefault

5. Disallow privilege escalation. Set allowPrivilegeEscalation: false on every container, which restricted requires and which no other fix implies.

securityContext:
  allowPrivilegeEscalation: false

The migration runbook

A production migration runbook:

1. Audit. kube-linter lint every manifest in the namespace. Categorise violations by type.
2. Prioritise. Fix the small violations first (root, seccomp). Save the medium violations for the next sprint.
3. Test. Run the fixed manifest in a staging namespace with enforce: restricted.
4. Promote. Apply the fixed manifest to prod.
5. Verify. Run the audit again; verify no violations remain.
6. Enforce. Set the enforce label and pin pod-security.kubernetes.io/enforce-version to the current release, so a later cluster upgrade cannot tighten the profile underneath a running workload.

#!/bin/bash
# migrate-pss.sh — migrate one namespace

NS="$1"
PROFILE="restricted"

# 1. Audit
echo "=== Audit for $NS ==="
kubectl get pods -n "$NS" -o yaml | kube-linter lint /dev/stdin

# 2. Set warn
kubectl label ns "$NS" "pod-security.kubernetes.io/warn=$PROFILE" --overwrite

# 3. Set audit
kubectl label ns "$NS" "pod-security.kubernetes.io/audit=$PROFILE" --overwrite

# 4. Wait for fixes (manual)

# 5. Set enforce
kubectl label ns "$NS" "pod-security.kubernetes.io/enforce=$PROFILE" --overwrite

echo "=== Migration complete for $NS ==="

Production failure modes

1. Migration deadline is missed. A namespace stays on warn: restricted indefinitely; the violations are not fixed. The fix is a per-quarter audit with a deadline.
2. A workload cannot be fixed. The migration encounters a workload that genuinely cannot meet restricted (e.g., a legacy binary that requires root). The fix is to document the exception, pin the enforce-version, and migrate the binary.
3. Over-restriction breaks a workload. The workload is migrated to restricted but fails to start. The fix is to roll back the manifest change and investigate.
4. The CI audit is not enforced. The CI pipeline does not run kube-linter; new manifests with violations are deployed. The fix is to add the audit step to CI.

The operational discipline

1. Audit in CI/CD. Every manifest applied to the cluster passes the audit.
2. Per-namespace migration. The team tracks each namespace’s progress.
3. Quarterly review. Every namespace’s PSA labels are reviewed.
4. Documented exceptions. A list of namespaces that cannot meet restricted is maintained.
5. Auditor reports. The SIEM alerts on violations; the security team reviews the alerts weekly.

Cross-course references

• The Observability course covers the audit log and SIEM integration.
• The Linux course covers the kernel features that PSS enforces.

Quiz

Production discipline

The PSS migration is a multi-quarter exercise. The discipline is per-namespace, multi-step, and audit-driven. The CI pipeline catches new violations before they reach the cluster; the namespace label enforces the profile at admission; the audit log records every violation. A cluster whose namespaces are mostly restricted and whose migration is documented has a PSS programme that is auditable; a cluster whose namespaces are mostly privileged or baseline and whose migration is stalled has a programme that is not.