RBAC blocks operation

Reported symptoms

Nine days ago the platform team finished a piece of work everyone had wanted for a year: the release pipeline stopped using a shared cluster-admin kubeconfig and started using a ServiceAccount per environment, bound to a Role written by hand for the job.

Since then, four things have been going wrong, and they have been going wrong in four different queues.

The release engineer’s complaint. Production releases are reported as failed. They are not failed - the Deployments are updated and the new version is serving - but the pipeline’s last step, which tails the new Pod’s logs for thirty seconds to confirm a clean start, dies with a Forbidden. The team has started ignoring red releases, which is its own problem.

The capacity engineer’s complaint. The maintenance runbook scales the worker Deployment down before the batch window and back up afterwards. The scale step is Forbidden. The same pipeline, minutes earlier, edited that same Deployment.

A developer’s complaint. She used the pipeline credential to exec into a Pod to run a one-off data fix, because that is what the runbook says to do. Forbidden. She can list the Pods perfectly well with the same credential.

Nobody’s complaint, which is the interesting one. The nightly node-maintenance job cordons a node and drains it. It has been exiting non-zero every night for nine nights. Nobody looked, because it is a cron job that retries. The node is cordoned before it fails, so the cluster has been running one node short since the migration.

Two facts made the shift dismiss RBAC early. The on-call engineers can do all four operations with their own credentials without trouble. And the identical pipeline has been green in staging for a fortnight.

Evidence collected

$ kubectl auth can-i get pods -n prod --as=system:serviceaccount:ci:deploy

yes

$ kubectl auth can-i get pods/log -n prod --as=system:serviceaccount:ci:deploy

no

$ kubectl auth can-i create pods/exec -n prod --as=system:serviceaccount:ci:deploy

no

$ kubectl auth can-i update deployments/scale -n prod --as=system:serviceaccount:ci:deploy

no

$ kubectl auth can-i --list -n prod --as=system:serviceaccount:ci:deploy

Resources                Non-Resource URLs   Resource Names   Verbs
configmaps               []                  []               [get list watch create update patch]
services                 []                  []               [get list watch create update patch]
pods                     []                  []               [get list watch delete]
deployments.apps         []                  []               [get list watch create update patch]

$ kubectl auth can-i create pods/eviction -n prod --as=system:serviceaccount:platform:maintenance

no

$ kubectl auth can-i patch nodes --as=system:serviceaccount:platform:maintenance

yes

$ grep -n 'resources:' roles/prod-deploy-role.yaml

7:  resources: ["pods", "services", "configmaps"]
11:  resources: ["deployments"]

$ kubectl -n staging get rolebinding deploy -o jsonpath='{.roleRef.kind}/{.roleRef.name}'

ClusterRole/edit

Work the evidence before reading on

Four tools, four error messages, four teams. Resist the urge to fix them one at a time.

1. Line up the four failing operations and write down, for each one, the API path kubectl actually calls. Not the kind - the path.
2. Compare that list against the four rows in auth can-i --list. What shape of thing is present in one list and absent from the other?
3. The on-call engineers can do all four. What role are they bound through, and did anybody write it?
4. Staging is green. Read its roleRef again. Is staging testing the same thing production runs?
5. The drain job cordons and then fails. Which of those two actions needed a permission the job has, and which needed one it does not?

Before continuing: the release itself succeeded every time. Does that make this a reporting bug, or is it telling you something about which part of the API surface was granted?

Root cause

1. A subresource is a separate resource

RBAC does not authorize kinds. It authorizes requests, and a request is identified by its API group, its resource path and its verb. Several operations that look like operations on an object are served at their own path underneath it:

A rule granting every verb on pods grants nothing on pods/log. The two are different resources to the authorizer, and there is no inheritance between them - which is deliberate, because reading a Pod object and reading what that Pod has printed to stdout are very different privileges.

2. The Roles were written from a list of kinds

The migration replaced a shared cluster-admin credential with two hand-written roles: a Role in each application namespace for the deploy identity, and a ClusterRole for the maintenance identity. Both were composed the same way - by listing the kinds the work touches. Pods, Deployments, Services, ConfigMaps. Nodes and Pods.

That is a reasonable thing to do and it produces a Role that is correct about everything it names. The gap is not a missing kind; every kind the procedures touch is present. The gap is that four steps of those procedures do not address a kind at all.

Reading nothing but the resources: lines of a Role manifest is the compact form of this check, and it takes seconds. Almost every realistic operational procedure needs logs at minimum. A hand-written Role whose resource lists contain no path at all has usually been written from a list of nouns.

3. Everything else was already using a role that contained them

The two facts that made the shift dismiss RBAC turn out to be the same fact.

The on-call engineers are bound, through their identity-provider group, to one of the built-in ClusterRoles that ship with the cluster. Staging binds the deploy ServiceAccount to a built-in ClusterRole too - the roleRef says so - because staging was configured in an afternoon and production was the one done properly.

The built-in roles are not magic; they are ordinary ClusterRole objects you can read with kubectl get clusterrole edit -o yaml, and they were written by people who had this exact list in front of them. So every path into the cluster except the new one already carried the subresources, which is precisely why nothing caught the omission until the new path was the only one in use.

The careful work is what broke. That is worth sitting with for a moment before writing the fix.

Resolution

1. Fix the cordoned node first. It is unrelated to the release and it is costing capacity every night: uncordon it, and disable the nightly job until its permissions are corrected so it stops cordoning a tenth one.
2. Write the mapping before writing the YAML. For each step of the release procedure and each step of the maintenance procedure, record the verb and the full resource path it calls. That table is the change request; the manifests are a transcription of it.
3. Add the four rules the table produces and no others: get on pods/log, create on pods/exec, update on deployments/scale in the apps group, and create on pods/eviction for the maintenance identity.
4. Review the diff against the table, not against the four symptoms. A rule that fixes a symptom but is not on the table is a rule nobody asked for, and it will still be there in a year.
5. Apply to one namespace first and confirm with kubectl auth can-i for each new path, using the exact ServiceAccount and namespace rather than your own credentials. Your credentials will say yes to everything and prove nothing.
6. Run the real operations, not just the permission checks: tail a log, scale the Deployment by one and back, exec true in a Pod, and let one full maintenance run evict and complete.
7. Roll the change to the remaining namespaces once one has been proven, and point the staging RoleBinding at the same Role production uses so the environments stop diverging.
8. Add a pre-flight check to the pipeline that loops over the required verb and resource pairs and fails in seconds with the missing one named, before any work starts.

Verification

1. Each of the four paths answers yes for the right identity. Run kubectl auth can-i for pods/log, pods/exec, deployments/scale and pods/eviction with the ServiceAccount and namespace spelled out.
2. The operations themselves work, which is a stronger claim than the permission check. A log tailed, a Deployment scaled and restored, a command executed, a node fully drained and returned to service.
3. The checks can fail. Remove one subresource rule in a scratch namespace and confirm the corresponding check goes red. A verification that passes with the rule missing is measuring nothing at all.
4. The surface grew by exactly what was intended. Compare kubectl auth can-i --list from before and after: the difference should be the rows on the table and nothing else.
5. No broad binding was left behind. Search the cluster for bindings naming either ServiceAccount and confirm each roleRef is the intended Role.
6. The previously cordoned node is schedulable and carrying Pods, and the maintenance job has completed one full run end to end rather than exiting after the cordon.
7. Staging and production bind the same Role. Read both roleRefs and require them to name the same object; an environment that tests a different role tests a different system.
8. The pipeline pre-flight check works. Break a permission deliberately and confirm the pipeline fails at the check with the missing pair named, not eight minutes later inside a log tail.

Prevention

• Write Roles from operations, not from kinds. The unit the authorizer uses is a verb and a resource path; anything else is a translation step performed by a human under time pressure.
• Keep the step-to-permission table next to the procedure. It is the only artefact that lets the next person tell a rule that is needed from a rule that was added during an incident.
• Treat a hand-written Role with no path in its resources as suspect. Almost every real procedure needs logs at minimum, so a Role with no slash in it is usually a Role written from a list of nouns.
• Check permissions before doing the work. A pre-flight loop over the required pairs fails in seconds and names the missing one; discovering the same fact through a tool-specific error message costs a release.
• Bind the same Role in staging that production uses. Staging is only worth running if it is the same, and a roleRef is a cheap thing to compare.
• Treat a partly failed automated job as an incident, not a retry. The drain job left the cluster worse off than either outcome would have - a node cordoned and not drained - and it did it quietly, nine times, because the exit code went somewhere nobody reads.
• Remember that permissions only ever widen. Nothing you add later can narrow a surface a previous binding opened, so the moment to be careful is when the binding is created.