Lab 20: Build least-privilege RBAC

Objective

By the end of this lab you will have taken one ServiceAccount from cluster-admin down to a Role with three rules — and, more importantly, you will have got those three rules from the API server’s audit log rather than from a guess.

That distinction is the whole lab. Writing a minimal Role is easy. Knowing what belongs in it is the hard part, and it is the part every “we tightened RBAC” project actually fails at: somebody reads the manifest, writes what looks right, ships it, and three weeks later a code path nobody tested returns 403 at 02:00.

You will also act as the workload, using the token the kubelet projected into its container. Not an impersonation flag — the real credential, copied out of the Pod exactly as an attacker with code execution in that container would get it.

Architecture

Every request to the API server passes three gates in order. RBAC is only the second one, and knowing which gate refused you is half of every RBAC diagnosis.

flowchart TB
    R["Request + bearer token"] --> AUTH["Authentication<br/>who are you?"]
    AUTH -->|"system:serviceaccount:payments:reporter"| AZ["Authorization<br/>RBAC: union of all bound rules"]
    AZ -->|allowed| ADM["Admission<br/>PodSecurity, webhooks, quota"]
    AZ -->|no matching rule| D403["403 Forbidden<br/>names the group, resource and verb"]
    ADM --> ETCD["Persisted"]

The credential itself reaches the workload by a path that has nothing to do with RBAC, and that is worth holding in your head separately:

flowchart LR
    SA["ServiceAccount<br/>payments/reporter"] --> KUBELET["kubelet requests a bound token<br/>via the TokenRequest API"]
    KUBELET --> VOL["projected volume<br/>/var/run/secrets/kubernetes.io/serviceaccount/"]
    VOL --> C["every process in the container"]
    C --> API["API server"]

Two independent decisions live in that second diagram. Whether the container gets a credential at all is automountServiceAccountToken. What the credential can do is RBAC. Teams routinely fix the second and never look at the first, which is why a Pod that never calls the API still ships with a working cluster credential inside it.

Requirements

• A disposable kubeadm cluster: one control-plane node and two workers, Kubernetes 1.34.x. The cluster built in Lab 01 is exactly right. Do not use a cluster you care about — Task 3 deliberately creates a cluster-admin binding, and Task 4 edits the API server’s static Pod manifest.
• kubectl 1.34.x and jq on your workstation.
• Cluster-admin on the cluster, because --as impersonation requires the impersonate verb.
• SSH with sudo to the control-plane node (k8s-cp-1, 192.0.2.11 in Lab 01’s topology). Task 4 writes an audit policy file there and adds four flags to /etc/kubernetes/manifests/kube-apiserver.yaml.
• Roughly 200 MiB of disk on the control-plane node for the audit log. The policy below records one ServiceAccount and nothing else, so it stays small, but check before you start.
• The manifests below pin nginx:1.27-alpine. Confirm the tag still resolves before you begin. This lab is not about image pulls.

Scenario

Eighteen months ago the platform team installed an internal Helm chart for a finance reporting job. The chart shipped a ClusterRoleBinding to cluster-admin for its ServiceAccount, with a comment in the values file saying # TODO: narrow this down. The install worked. Nobody came back.

Last week a dependency-confusion attack put attacker-controlled code into that container’s base image. The application it compromised does three things: it reads one ConfigMap, lists the Pods in its own namespace, and writes a status ConfigMap. The blast radius was every Secret in the cluster.

Your job is not to write a smaller Role — anyone can do that. It is to establish, from evidence, which three rules the workload needs, so that the smaller Role you ship does not break it.

Tasks

Task 1: Record the starting state, and find out who you are

Cleanup deletes cluster-scoped objects. It must delete only what this lab created, and the only way to know that is to write down what was there first.

WORKDIR="$HOME/k8s-rbac-lab"
mkdir -p "$WORKDIR"
cd "$WORKDIR"

kubectl get clusterrolebindings -o name | sort > crb-before.txt
kubectl get clusterroles -o name | sort > cr-before.txt
kubectl get namespaces -o name | sort > ns-before.txt

grep -c . crb-before.txt cr-before.txt

Now establish your own identity, because every --as result in this lab depends on it:

$ kubectl auth whoami

ATTRIBUTE   VALUE
Username    kubernetes-admin
Groups      [kubeadm:cluster-admins system:authenticated]

On a kubeadm cluster at 1.29 or later, admin.conf authenticates as kubernetes-admin in the group kubeadm:cluster-admins, which is bound to cluster-admin by a ClusterRoleBinding you can read. The older system:masters identity now lives in a separate super-admin.conf that kubeadm leaves on the control-plane node. The difference matters here: an identity that gets its power from a binding is an identity whose power you can see with kubectl get clusterrolebinding, while system:masters is hard-coded in the authorizer and appears in no RBAC object at all.

Check which one you have, and record your own effective permissions so you can tell your access apart from the workload’s later:

cd "$HOME/k8s-rbac-lab"

kubectl get clusterrolebinding kubeadm:cluster-admins -o yaml \
  > my-binding.yaml 2>/dev/null \
  || echo "no kubeadm:cluster-admins binding - you are probably using super-admin.conf"

kubectl auth can-i --list > my-permissions.txt
head -5 my-permissions.txt

Task 2: Build the workload and its over-permissioned identity

payments-reporter.yaml:

apiVersion: v1
kind: Namespace
metadata:
  name: payments
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: reporter
  namespace: payments
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: report-config
  namespace: payments
data:
  schedule: "0 2 * * *"
  currency: "GBP"
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: report-status
  namespace: payments
data:
  lastRun: "never"
---
apiVersion: v1
kind: Secret
metadata:
  name: ledger-credentials
  namespace: payments
type: Opaque
stringData:
  # A canary. Nothing reads it. Its only job is to be somewhere the
  # ServiceAccount should not be able to reach, so "should not" is testable.
  password: "CANARY-LEDGER-b41f7a"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: reporter
  namespace: payments
spec:
  replicas: 1
  selector:
    matchLabels:
      app: reporter
  template:
    metadata:
      labels:
        app: reporter
    spec:
      serviceAccountName: reporter
      containers:
        - name: app
          image: nginx:1.27-alpine
          ports:
            - containerPort: 80

The container is nginx and makes no API calls of its own. That is deliberate: it keeps the exercise reproducible, and it sets up Task 7, where you will find that a workload which never touches the API server is nonetheless carrying a live cluster credential.

Apply it, then create the binding the chart shipped:

cd "$HOME/k8s-rbac-lab"

kubectl apply -f payments-reporter.yaml
kubectl rollout status deployment/reporter -n payments --timeout=120s

$ kubectl create clusterrolebinding reporter-admin --clusterrole=cluster-admin --serviceaccount=payments:reporter

That single command is the whole defect, and it is one line in a chart’s rbac.yaml. Note how ordinary it looks.

Task 3: Become the workload, and measure the blast radius

Impersonation with --as would answer the question faster, but it would answer a slightly different question — it asks what the authorizer would decide for that name. Instead, take the actual credential, the way somebody with code execution in the container would.

cd "$HOME/k8s-rbac-lab"

kubectl exec -n payments deploy/reporter -- \
  cat /var/run/secrets/kubernetes.io/serviceaccount/token > sa.token
kubectl exec -n payments deploy/reporter -- \
  cat /var/run/secrets/kubernetes.io/serviceaccount/ca.crt > sa-ca.crt

chmod 0600 sa.token
wc -c sa.token sa-ca.crt

Those two files, plus the address of the API server, are a complete set of cluster credentials. Build a kubeconfig from them:

cd "$HOME/k8s-rbac-lab"

APISERVER=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')
echo "$APISERVER"

kubectl config --kubeconfig=reporter.kubeconfig set-cluster lab \
  --server="$APISERVER" --certificate-authority=sa-ca.crt --embed-certs=true
kubectl config --kubeconfig=reporter.kubeconfig set-credentials reporter \
  --token="$(cat sa.token)"
kubectl config --kubeconfig=reporter.kubeconfig set-context reporter \
  --cluster=lab --user=reporter --namespace=payments
kubectl config --kubeconfig=reporter.kubeconfig use-context reporter

chmod 0600 reporter.kubeconfig

Confirm the identity, then measure what it can reach:

$ KUBECONFIG=reporter.kubeconfig kubectl auth whoami

ATTRIBUTE   VALUE
Username    system:serviceaccount:payments:reporter
Groups      [system:serviceaccounts system:serviceaccounts:payments system:authenticated]

The three groups are automatic and are worth memorising, because bindings made to any of them apply to this workload without naming it. That is anti-pattern five from the RBAC anti-patterns lesson, and this is what it looks like from the inside.

cd "$HOME/k8s-rbac-lab"

export KUBECONFIG="$PWD/reporter.kubeconfig"

kubectl auth can-i --list > can-i-before.txt
wc -l can-i-before.txt

kubectl get nodes
kubectl get secrets --all-namespaces | wc -l
kubectl get secret ledger-credentials -n payments -o jsonpath='{.data.password}' | base64 -d; echo

unset KUBECONFIG

The canary comes back in plain text, and so does every other Secret in the cluster — including the service account tokens of every other workload, and whatever the cluster stores for its CNI, its ingress controller and its CI system. Write down the Secret count. It is the number you will quote in the change record when somebody asks why this work was scheduled.

Task 4: Turn on auditing for exactly one identity

You now know what the ServiceAccount can do. Nothing so far tells you what it needs. The API server audit log is the only source of that answer that does not involve reading the application’s source and hoping you found every code path.

Write the policy on the control-plane node:

# Substitute your own control-plane address before running:
CP=192.0.2.11

ssh "$CP" 'sudo tee /etc/kubernetes/audit-policy.yaml > /dev/null' <<'POLICY'
apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
  - RequestReceived
rules:
  # Record metadata for one identity: verb, group, resource, name. That is
  # precisely the shape of an RBAC rule, and nothing more is needed here.
  - level: Metadata
    users: ["system:serviceaccount:payments:reporter"]
  # Everything else is dropped. On a busy cluster a catch-all rule produces
  # gigabytes per hour, and this lab needs one identity.
  - level: None
POLICY

ssh "$CP" 'sudo mkdir -p /var/log/kubernetes && sudo cp -a /etc/kubernetes/manifests/kube-apiserver.yaml /root/kube-apiserver.yaml.bak && ls -l /root/kube-apiserver.yaml.bak'

That backup is the rollback. Confirm it exists before you edit anything.

Now edit /etc/kubernetes/manifests/kube-apiserver.yaml on the control-plane node. Add four flags to the container’s command list:

    - --audit-policy-file=/etc/kubernetes/audit-policy.yaml
    - --audit-log-path=/var/log/kubernetes/audit.log
    - --audit-log-maxage=1
    - --audit-log-maxbackup=1

Add two volumeMounts to the kube-apiserver container:

    - name: audit-policy
      mountPath: /etc/kubernetes/audit-policy.yaml
      readOnly: true
    - name: audit-log
      mountPath: /var/log/kubernetes

And two matching volumes at the Pod level:

  - name: audit-policy
    hostPath:
      path: /etc/kubernetes/audit-policy.yaml
      type: File
  - name: audit-log
    hostPath:
      path: /var/log/kubernetes
      type: DirectoryOrCreate

Both mounts are required. The static Pod already mounts /etc/kubernetes/pki, but not the rest of /etc/kubernetes, so the policy file is not visible to the container unless you mount it — and an API server that cannot read its audit policy file refuses to start.

The kubelet notices the file change within a few seconds and restarts the Pod. Watch it come back:

# Substitute your own control-plane address before running:
CP=192.0.2.11

ssh "$CP" 'sudo crictl ps --name kube-apiserver'

until kubectl get --raw='/readyz' 2>/dev/null; do
  echo "waiting for the API server"
  sleep 5
done
echo

If that loop does not terminate within about two minutes, go to Troubleshooting. Do not keep editing.

Task 5: Exercise the workload and read the surface off the log

Run the three operations the reporting job performs, as the ServiceAccount:

cd "$HOME/k8s-rbac-lab"
export KUBECONFIG="$PWD/reporter.kubeconfig"

kubectl get configmap report-config -n payments -o yaml
kubectl get pods -n payments
kubectl patch configmap report-status -n payments --type merge \
  -p '{"data":{"lastRun":"2026-08-19T02:00:00Z"}}'

unset KUBECONFIG

These are driven by hand rather than by the container, because nginx does not make them. The API server cannot tell the difference: the credential is the same one the Pod holds, so the audit entries are attributed to the ServiceAccount exactly as they would be in production.

Now read the log:

# Substitute your own control-plane address before running:
CP=192.0.2.11

# The audit log is one JSON object per line. Read it over ssh and parse it
# locally, so jq only has to exist on your workstation.
ssh "$CP" 'sudo cat /var/log/kubernetes/audit.log' \
  | jq -r 'select(.user.username == "system:serviceaccount:payments:reporter")
           | [.verb, (.objectRef.apiGroup // "core"), .objectRef.resource, (.objectRef.name // "-")]
           | @tsv' \
  | sort -u | tee surface-from-audit.txt

$ ssh "$CP" 'sudo cat /var/log/kubernetes/audit.log' | jq -r '[.verb, .objectRef.resource, (.objectRef.name // "-")] | @tsv' | sort -u

get	configmaps	report-config
list	pods	-
patch	configmaps	report-status
create	selfsubjectrulesreviews	-

Four lines, and the fourth is the point of the whole task. create on selfsubjectrulesreviews is the API behind kubectl auth can-i --list — you ran it in Task 3 with this credential, and it is now in the workload’s observed surface even though the application never calls it. Nothing in the manifest would have told you that. Nothing in your reading of the code would have told you either.

Cross out the line you know is diagnostic. Three rules remain.

Task 6: Write the Role, and swap the binding

reporter-role.yaml:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: reporter
  namespace: payments
rules:
  # Read one named ConfigMap. Not "configmaps" - this one.
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["get"]
    resourceNames: ["report-config", "report-status"]
  # Write only the status object.
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["patch"]
    resourceNames: ["report-status"]
  # A list request asks for a collection, so it cannot be narrowed by name.
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: reporter
  namespace: payments
subjects:
  - kind: ServiceAccount
    name: reporter
    namespace: payments
roleRef:
  kind: Role
  name: reporter
  apiGroup: rbac.authorization.k8s.io

Note what is absent. There is no list on ConfigMaps, and that omission is doing real work: resourceNames restricts get, but a list returns a collection, so a rule granting list on configmaps returns every ConfigMap in the namespace regardless of any names in the rule. Granting list alongside get would quietly undo the scoping you just wrote.

Apply the Role, then remove the old binding:

cd "$HOME/k8s-rbac-lab"

kubectl apply -f reporter-role.yaml
kubectl delete clusterrolebinding reporter-admin

Order matters. Apply first, delete second: between the two commands the workload holds the union of both, which is harmless. Reverse the order and there is a window — however short — in which the workload holds nothing, and a job that runs in that window fails for a reason you will not be able to reproduce afterwards.

Now measure the new surface:

cd "$HOME/k8s-rbac-lab"
export KUBECONFIG="$PWD/reporter.kubeconfig"

kubectl auth can-i --list -n payments | tee can-i-after.txt

kubectl get configmap report-config -n payments        # must succeed
kubectl get configmaps -n payments                     # must be refused
kubectl get pods -n payments                           # must succeed
kubectl get secrets -n payments                        # must be refused
kubectl get nodes                                      # must be refused

unset KUBECONFIG

$ KUBECONFIG=reporter.kubeconfig kubectl get configmaps -n payments

Error from server (Forbidden): configmaps is forbidden: User "system:serviceaccount:payments:reporter" cannot list resource "configmaps" in API group "" in the namespace "payments"

Read that message carefully once, because you will read it a thousand times in production. It names the identity, the verb (list), the resource (configmaps), the API group (empty string, meaning core) and the namespace. Those five fields are the exact shape of the rule you would have to add to make it work — which means the error message is telling you what to write, and also telling you what you would be giving away.

kubectl get configmap report-config succeeding while kubectl get configmaps fails is the observable proof that resourceNames does something. Get that pair in your notes.

Task 7: Try to escalate, and watch a second mechanism refuse

The compromised container’s next move is to grant itself more. Try it:

cd "$HOME/k8s-rbac-lab"
export KUBECONFIG="$PWD/reporter.kubeconfig"

kubectl create rolebinding self-admin -n payments \
  --clusterrole=admin --serviceaccount=payments:reporter

unset KUBECONFIG

That is refused, and for the boring reason: the Role does not grant create on rolebindings. Now make it interesting. Grant exactly that one permission and try again — this is the case people assume works, and it is the one that teaches the most:

cd "$HOME/k8s-rbac-lab"

kubectl create role rbac-writer -n payments \
  --verb=create --resource=rolebindings
kubectl create rolebinding rbac-writer -n payments \
  --role=rbac-writer --serviceaccount=payments:reporter

export KUBECONFIG="$PWD/reporter.kubeconfig"
kubectl create rolebinding self-admin -n payments \
  --clusterrole=admin --serviceaccount=payments:reporter \
  2>&1 | tee escalation-attempt.txt
unset KUBECONFIG

$ KUBECONFIG=reporter.kubeconfig kubectl create rolebinding self-admin -n payments --clusterrole=admin --serviceaccount=payments:reporter

Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io "self-admin" is forbidden: user "system:serviceaccount:payments:reporter" (groups=["system:serviceaccounts" "system:serviceaccounts:payments" "system:authenticated"]) is attempting to grant RBAC permissions not currently held:
{APIGroups:[""], Resources:["pods"], Verbs:["create" "delete" ...]}
{APIGroups:[""], Resources:["secrets"], Verbs:["get" "list" ...]}

This is privilege-escalation prevention, and it is a different mechanism from the authorization decision. The SA passed authorization — it does have create on rolebindings. The RBAC admission logic then compared the permissions the new binding would grant against the permissions the creator already holds, found a delta, and refused, printing the delta.

The consequence is the useful part: an identity that can create RoleBindings can grant away everything it already has, and nothing more. It is not a free path to cluster-admin. It is a lateral-movement primitive — it can hand the SA’s own permissions to any other subject, including one the attacker controls — which is why create on rolebindings still belongs on the critical findings list in an audit.

Prove the other half, so the rule is precise rather than mythological:

cd "$HOME/k8s-rbac-lab"
export KUBECONFIG="$PWD/reporter.kubeconfig"

# Granting a Role it DOES hold is allowed: no delta, no refusal.
kubectl create rolebinding share-reporter -n payments \
  --role=reporter --serviceaccount=payments:default

unset KUBECONFIG

Two documented ways around this check exist, and both are named in the upstream RBAC documentation: the escalate verb on roles/clusterroles, which switches the delta check off, and the bind verb on a specific Role, which permits binding that Role only. Neither should ever appear on a workload ServiceAccount. Search for them in Task 8.

Task 8: Close the automount, then audit the whole cluster

The reporter Pod is nginx. It has never called the API server. Look at what it is carrying anyway:

kubectl exec -n payments deploy/reporter -- \
  ls -l /var/run/secrets/kubernetes.io/serviceaccount/

Three entries: ca.crt, namespace, and token. Turn the mount off at the ServiceAccount, which covers every Pod that uses it:

cd "$HOME/k8s-rbac-lab"

kubectl patch serviceaccount reporter -n payments \
  -p '{"automountServiceAccountToken": false}'

kubectl rollout restart deployment/reporter -n payments
kubectl rollout status deployment/reporter -n payments --timeout=120s

kubectl exec -n payments deploy/reporter -- \
  ls /var/run/secrets/kubernetes.io/serviceaccount/ \
  || echo "no projected token - the directory does not exist"

The command fails, and the failure is the pass condition. A Pod spec can also set automountServiceAccountToken: false, and the Pod-level setting wins over the ServiceAccount-level one — set it in both places for a workload that must never hold a credential, so that a later change to either object does not silently restore it.

Now sweep the cluster for the patterns you have just built and dismantled:

cd "$HOME/k8s-rbac-lab"

echo "=== ClusterRoles containing a wildcard ==="
kubectl get clusterroles -o json | jq -r '
  .items[] | . as $cr | .rules[]?
  | select(((.verbs // []) | index("*")) or ((.resources // []) | index("*")) or ((.apiGroups // []) | index("*")))
  | $cr.metadata.name' | sort -u

echo "=== Anything bound to cluster-admin ==="
kubectl get clusterrolebindings -o json | jq -r '
  .items[] | select(.roleRef.name == "cluster-admin")
  | .metadata.name as $n | .subjects[]? | $n + "  <-  " + .kind + "/" + .name' | sort -u

echo "=== Bindings whose subject is a built-in group ==="
kubectl get rolebindings,clusterrolebindings -A -o json | jq -r '
  .items[] | .metadata.name as $n | .subjects[]?
  | select(.name | startswith("system:"))
  | $n + "  <-  " + .kind + "/" + .name' | sort -u

echo "=== Roles granting the escalation verbs ==="
kubectl get roles,clusterroles -A -o json | jq -r '
  .items[] | . as $r | .rules[]?
  | select(((.verbs // []) | index("escalate")) or ((.verbs // []) | index("bind")) or ((.verbs // []) | index("impersonate")))
  | (($r.metadata.namespace // "cluster") + "/" + $r.metadata.name)' | sort -u

Every one of those will return rows on a stock kubeadm cluster, and that is correct rather than alarming: cluster-admin itself is a wildcard ClusterRole, system:kube-controller-manager holds broad grants it genuinely needs, and the built-in system:basic-user is bound to system:authenticated on purpose. The finding is never “a wildcard exists”. It is “a wildcard exists on an identity that a workload can reach”, and telling those apart is what makes an audit report worth reading. Note the rows that are not prefixed system: — those are yours.

Validation

Run these against the finished state. Each proves an outcome rather than restating a step.

cd "$HOME/k8s-rbac-lab"

# 1. The over-permissioned binding is gone.
kubectl get clusterrolebinding reporter-admin 2>&1 | grep -q NotFound \
  && echo "PASS: cluster-admin binding removed"

# 2. The SA can still do its job.
export KUBECONFIG="$PWD/reporter.kubeconfig"
kubectl get configmap report-config -n payments -o name
kubectl get pods -n payments -o name | head -1
kubectl patch configmap report-status -n payments --type merge \
  -p '{"data":{"lastRun":"validation"}}' -o name

# 3. The SA cannot reach the canary.
kubectl get secret ledger-credentials -n payments 2>&1 | grep -q Forbidden \
  && echo "PASS: secrets are refused"

# 4. Enumeration of ConfigMaps is refused even though one is readable.
kubectl get configmaps -n payments 2>&1 | grep -q 'cannot list' \
  && echo "PASS: resourceNames scoping holds"
unset KUBECONFIG

# 5. The effective surface shrank, and the diff shows by how much.
diff can-i-before.txt can-i-after.txt | head -20

# 6. The evidence trail exists.
wc -l surface-from-audit.txt escalation-attempt.txt

Check 2 must produce three successful outputs — a Role that breaks the workload is not a smaller Role, it is an outage. Check 4 is the one people skip; without it you have not shown that resourceNames did anything. Check 5 must show can-i-after.txt as a handful of lines against a page and a half before it.

Expected Outcome

k8s-rbac-lab/
├── can-i-after.txt
├── can-i-before.txt
├── crb-before.txt
├── cr-before.txt
├── escalation-attempt.txt
├── my-binding.yaml
├── my-permissions.txt
├── ns-before.txt
├── payments-reporter.yaml
├── reporter-role.yaml
├── reporter.kubeconfig     (mode 0600, deleted in Cleanup)
├── sa-ca.crt
├── sa.token                (mode 0600, deleted in Cleanup)
└── surface-from-audit.txt

On the cluster: a payments namespace whose ServiceAccount holds one namespaced Role with three rules, no ClusterRoleBinding, and no projected token in the Pod that does not need one. On the control-plane node: an audit policy recording one identity, and a backup of the original API server manifest.

Production notes

An RBAC tightening is a change with an unusual property: it does not fail at apply time. It fails later, on the first code path that needs the permission you removed — which may be a month away, on a schedule nobody thought about, at an hour nobody chose.

That shapes how it belongs in a change window.

Evidence before the change, not after. The audit rule for the identity goes in first and stays in for at least one full business cycle. The change record cites the observed surface, not the manifest. If you cannot produce that evidence, the change is not ready.

Add the new binding before removing the old one, and keep them overlapping for a defined period. The union is harmless; the gap is not. For a high-value workload, leave both in place for a week and read the audit log for calls that the new Role would have refused. There is no dry-run mode for RBAC, and this overlap is the closest thing to one.

Name the rollback and time-box it. The rollback is one command — recreate the removed binding — and it should be written out in full in the change record so the person holding the pager at 02:00 does not have to reconstruct it. Set an explicit point, in hours, after which an unexplained 403 from that workload means roll back first and investigate afterwards.

Holding is a legitimate outcome, and it needs an owner. If the observation window did not cover the workload’s monthly job, stop. Record what you have in surface-from-audit.txt, name the person who owns the next attempt, and name the date. A half-tightened Role shipped on a guess is worse than the cluster-admin binding you were trying to remove, because it converts a known, documented risk into an unknown outage.

The credential extraction in Task 3 is the exercise to repeat. Once a quarter, take a token out of a running production Pod and run kubectl auth can-i --list with it. It takes two minutes and it is the only question that matters: not what the chart intended to grant, but what a compromised container in that Pod would actually hold.

Troubleshooting

The API server never comes back after Task 4. Restore the backup over SSH and wait: sudo cp /root/kube-apiserver.yaml.bak /etc/kubernetes/manifests/kube-apiserver.yaml. The kubelet restarts the static Pod from the restored file. If kubectl is still dead after two minutes, read the container’s own output on the node with sudo crictl ps -a --name kube-apiserver followed by sudo crictl logs on the container id it prints. The commonest causes are YAML indentation in the volumes block and a hostPath type: File pointing at a file that does not exist.

kubectl exec fails with “container not found”. The Deployment has not finished rolling out, or the previous rollout restart is still in progress. kubectl get pods -n payments -w until one Pod is Running and 1/1 ready.

KUBECONFIG=reporter.kubeconfig kubectl returns “Unauthorized”. The token has expired. Tokens minted for projected volumes are time-bound and the kubelet rotates the file in the container, but your extracted copy does not rotate. Re-run the two kubectl exec ... cat commands from Task 3 and rebuild the credentials entry.

kubectl auth whoami returns “unknown command”. The subcommand needs kubectl 1.26 or later. Check with kubectl version --client; the rest of the lab works without it, and kubectl auth can-i --list will still show you the identity’s surface.

The audit log file does not exist. Confirm the --audit-log-path flag made it into the running Pod: kubectl -n kube-system get pod -l component=kube-apiserver -o yaml | grep audit. If the flags are absent, the kubelet is running an older copy of the manifest — check for a syntax error that made it refuse the new one.

The audit log is enormous. The level: None catch-all rule is missing or is above the Metadata rule. Rules are evaluated in order and the first match wins, so a catch-all must be last.

The escalation attempt succeeds. You are still holding the cluster-admin binding, or you are running the command with your admin kubeconfig rather than reporter.kubeconfig. kubectl auth whoami before each attempt; the single commonest mistake in this lab is forgetting to set KUBECONFIG.

Cleanup

Two things here are more sensitive than usual: a live token on your workstation, and an edited API server manifest on a control-plane node. Handle both explicitly.

Step 1. Destroy the credential first, before anything else can distract you:

cd "$HOME/k8s-rbac-lab"

shred -u sa.token reporter.kubeconfig 2>/dev/null \
  || rm -f sa.token reporter.kubeconfig

ls sa.token reporter.kubeconfig 2>&1 | grep -q 'No such file' \
  && echo "credential files removed"

Step 2. Remove the cluster objects this lab created:

cd "$HOME/k8s-rbac-lab"

kubectl delete clusterrolebinding reporter-admin --ignore-not-found
kubectl delete namespace payments --ignore-not-found

kubectl get clusterrolebindings -o name | sort > crb-after.txt
diff crb-before.txt crb-after.txt && echo "ClusterRoleBindings restored to the starting set"

Deleting the namespace removes the Role, the RoleBindings, the ServiceAccount, the ConfigMaps and the Secret in one operation, because all of them are namespaced. The ClusterRoleBinding is not, which is exactly why it needs its own line — and why an audit that only looks inside namespaces misses the object that mattered.

Step 3. Revert the API server:

$ sudo cp /root/kube-apiserver.yaml.bak /etc/kubernetes/manifests/kube-apiserver.yaml

# Substitute your own control-plane address before running:
CP=192.0.2.11

until kubectl get --raw='/readyz' 2>/dev/null; do sleep 5; done; echo

ssh "$CP" 'sudo rm -f /etc/kubernetes/audit-policy.yaml && sudo rm -rf /var/log/kubernetes'
kubectl -n kube-system get pod -l component=kube-apiserver -o yaml | grep -c audit \
  || echo "no audit flags on the running API server - reverted"

Keep surface-from-audit.txt, can-i-before.txt, can-i-after.txt and escalation-attempt.txt; they are the deliverables. Everything else in the working directory can go.

What You Learned

• A Pod’s credential is two files and a URL. You built a working kubeconfig out of a projected volume in three commands. That is the entire distance between “code execution in a container” and “an authenticated cluster identity”.
• The audit log is where a least-privilege Role comes from. Verb, API group, resource, object name — the four fields an audit event records are the four fields an RBAC rule needs. Guessing from the manifest missed a call you had made yourself.
• An audit-derived surface is both too wide and too narrow, and knowing which parts are which is the skill. Diagnostic calls contaminate it; code paths that did not run are missing from it.
• resourceNames scopes get and does not scope list. Granting both on the same resource quietly undoes the scoping, and the only way to know your Role is doing what you think is to run the refused command and see it refused.
• Privilege-escalation prevention is a second, separate mechanism. An identity that can create RoleBindings can hand out what it already holds and nothing more — a lateral-movement primitive, not a path to cluster-admin.
• RBAC has no deny. Permissions are a union across every binding, so tightening means removing bindings, and an audit that finds one binding and stops has found nothing.
• A Pod that never calls the API still ships with a credential until somebody sets automountServiceAccountToken: false. The default is on.