Overprivileged ServiceAccount · Break/Fix · Kubernetes

Reported symptoms

At 09:20 the quarterly compliance scan lands in the platform team’s queue with two Critical findings against one identity:

CRITICAL  monitoring/prometheus  ServiceAccount is bound to cluster-admin
CRITICAL  monitoring/prometheus  ServiceAccount can create clusterrolebindings

The team has seen the first one before. Four months ago the same scanner raised it, the team traced it to the monitoring chart’s default RBAC, disabled that default with a values override, wrote a minimum-surface ClusterRole covering exactly what Prometheus needs, put both in the platform repository, and closed the finding. git log shows no change to any RBAC manifest since.

Three facts make this look like a scanner fault rather than a cluster fault:

The team’s own audit disagrees. A nightly job walks every ServiceAccount in the cluster and reports over-permissioned ones. It ran at 06:00 and reported zero findings, as it has every night for four months.
Nothing is broken. Prometheus is scraping, every target is up, no alert fired, and no user raised anything. Whatever changed, it did not remove a capability from anything.
The reviewed fix is still there. helm get values metrics-stack -n monitoring prints the override the team wrote, unchanged, exactly as the repository has it.

The on-call engineer’s first instinct is that the scanner has been upgraded and is now reporting false positives. That instinct is reasonable, it is the cheapest hypothesis to test, and it is wrong.

Evidence provided

Read-only / Safethe effective permissions of the monitoring identity

$ kubectl auth can-i --list --as=system:serviceaccount:monitoring:prometheus

Resources   Non-Resource URLs   Resource Names   Verbs
*.*         []                  []               [*]
[*]                 []               [*]

Illustrative output

Read-only / Safethe escalation primitive, asked directly

$ kubectl auth can-i create clusterrolebindings --as=system:serviceaccount:monitoring:prometheus

yes

Illustrative output

Read-only / Safetwo bindings name the same ServiceAccount

$ kubectl get clusterrolebinding -o custom-columns=NAME:.metadata.name,ROLE:.roleRef.name,SUBJECT:.subjects[*].name | grep prometheus

metrics-stack-prometheus     cluster-admin                prometheus
platform-prometheus-scrape   platform-prometheus-scrape   prometheus

Illustrative output

Read-only / Safewho owns the binding the team did not write

$ kubectl get clusterrolebinding metrics-stack-prometheus -o yaml | head -12

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
annotations:
meta.helm.sh/release-name: metrics-stack
meta.helm.sh/release-namespace: monitoring
labels:
app.kubernetes.io/managed-by: Helm
name: metrics-stack-prometheus
roleRef:
kind: ClusterRole
name: cluster-admin

Illustrative output

Read-only / Safean upgrade nobody requested, on the night before the scan

$ helm history metrics-stack -n monitoring

REVISION  UPDATED                   STATUS      CHART                 APP VERSION  DESCRIPTION
17        Mon Aug 10 02:14:07 2026  superseded  metrics-stack-4.9.2   2.54.1       Upgrade complete
18        Tue Aug 11 02:14:11 2026  deployed    metrics-stack-5.0.0   2.55.0       Upgrade complete

Illustrative output

Read-only / Safethe pipeline command, unchanged for a year

$ grep -n 'helm upgrade' deploy/monitoring/deploy.sh

42:helm upgrade --install metrics-stack platform/metrics-stack \
43:  --namespace monitoring -f values/monitoring.yaml --wait

Illustrative output

Read-only / Safethe rendered release contains the binding

$ helm get manifest metrics-stack -n monitoring | grep -B4 'name: cluster-admin'

kind: ClusterRoleBinding
metadata:
name: metrics-stack-prometheus
roleRef:
kind: ClusterRole
name: cluster-admin

Illustrative output

Work the evidence before reading on

Two scanners disagree about the same cluster on the same morning. Exactly one of them is right, and the interesting question is not which finding is true but why the other tool could not see it.

There are two ClusterRoleBindings naming this ServiceAccount, and both are valid objects. What is the effective permission set of an identity that has two bindings? Is there any mechanism in RBAC by which the narrower one constrains the wider one?
The team’s override is still in helm get values, and the chart version in helm history is not the version the change record names. What does Helm do with a values key that the chart it is rendering does not read?
Read the nightly audit’s check one more time. It runs kubectl auth can-i --list and greps the output for cluster-admin. Look at the output of that command in the evidence above. Which column would the string cluster-admin appear in?

Before continuing: the ServiceAccount has cluster-admin and the audit that looks for cluster-admin reports clean. Both statements are true at the same time. How?

Root cause

1. RBAC is additive, so the widest binding wins

The team’s platform-prometheus-scrape ClusterRole is correct. It grants get, list and watch on the objects Prometheus discovers targets from, and nothing else. It is still bound, still valid, and still exactly what a reviewer would approve.

It is also irrelevant. Kubernetes RBAC is purely additive: there are no deny rules, and no binding can subtract from what another binding grants. An identity’s effective permissions are the union of every rule reachable from every binding that names it. Add cluster-admin to a subject that already has a narrow Role and the result is cluster-admin, with the narrow Role contributing a subset of what the union already contains.

This is why kubectl describe clusterrolebinding platform-prometheus-scrape looks perfect and proves nothing. Inspecting one binding answers “is this object what I intended”, which is a different question from “what can this identity do”.

2. The binding came back because the values key stopped existing

The pipeline runs helm upgrade --install with no --version. Without a pin, Helm installs whatever version the repository holds at that moment, so the deployed software is a function of the calendar rather than of the repository. On the night of 11 August the mirror had metrics-stack-5.0.0 where the last staging validation had exercised 4.9.2, and the pipeline took it.

The new major version renamed the values key that suppressed the chart’s default RBAC. The team’s values/monitoring.yaml still sets the old key. Helm does not object: unless a chart ships a values.schema.json, values keys the chart never reads are accepted silently and simply have no effect. The upgrade succeeded, reported Upgrade complete, and rendered the chart’s default ClusterRoleBinding to cluster-admin because, from the chart’s point of view, nobody had asked it not to.

That is the entire mechanism. A successful upgrade, a green pipeline, an unchanged repository, and a silently ignored override.

3. The audit grepped for a string the command never prints

The nightly job’s check is, in essence:

NS=monitoring
SA=prometheus
if kubectl auth can-i --list -n "$NS" \
     --as="system:serviceaccount:$NS:$SA" | grep -q cluster-admin; then
  echo "FINDING: $NS:$SA is cluster-admin"
fi

It has never produced a true positive and never can. kubectl auth can-i --list returns the effective rules for an identity: a table of resources, non-resource URLs, resource names and verbs. It does not return, and has no field in which to return, the names of the Roles or ClusterRoles those rules came from. The string cluster-admin cannot appear in that output regardless of what the identity is bound to.

The green result was therefore not evidence of a healthy cluster. It was the absence of a signal the check was structurally incapable of producing - which is the failure mode that makes a broken check worse than no check, because the team had been treating four months of green as confirmation.

Under the hood

kubectl auth can-i --list is not a client-side traversal of Roles and bindings. It posts a SelfSubjectRulesReview to the API server, and the API server answers with the rules that apply to the requesting identity - or, when --as is used, to the impersonated identity, which is why the flag requires impersonation permission on the caller.

The response type carries resourceRules and nonResourceRules. Each rule has API groups, resources, resource names and verbs. There is no field for the provenance of a rule, and this is deliberate: the authorizer chain may include webhook or node authorizers that have no Role objects at all, so a rule’s origin is not a concept the API can promise to report.

Two operational consequences follow. First, any audit that wants to know why an identity has a permission has to read ClusterRoleBinding and RoleBinding objects directly; can-i answers whether, not why. Second, can-i is the better tool for the whether question precisely because it is evaluated by the same authorizer that will evaluate the real request. An audit built on object graphs can be defeated by an aggregated ClusterRole, a binding to a Group the ServiceAccount happens to be in, or an authorizer the graph does not model.

The single-question form, kubectl auth can-i create clusterrolebindings --as=..., also sets its exit status: 0 for yes, non-zero for no. That makes it usable directly in a pipeline gate without parsing anything.

Deleting the binding is not the fix, and it destroys the evidence

The obvious first move is kubectl delete clusterrolebinding metrics-stack-prometheus. It is fast, it closes the finding, and it is wrong in three ways.

The object is owned by a Helm release. The next pipeline run renders the same chart with the same values and re-creates it, so the exposure returns on a schedule while the finding shows as closed. The deletion also removes the annotations that identify which release created it, which is the evidence that leads to the actual defect. And it leaves the real cause - an unpinned upgrade and an override that silently stopped applying - in place to do the same thing to the next chart.

Delete the object only as a deliberate, recorded stop-gap when the release cannot be corrected immediately, and only after capturing the object to a file. Then treat the re-creation as certain, not possible.

Resolution

Capture the evidence before changing anything: kubectl get clusterrolebinding metrics-stack-prometheus -o yaml > /tmp/cluster-admin-binding.yaml, plus the outputs of helm history, helm get values and helm get manifest. The incident record needs the object as it was.
Establish the exposure window from helm history: the binding exists from the timestamp of the revision that created it until now. That window, not the age of the finding, is the period the audit log must be reviewed for.
Decide explicitly between fixing now and holding. Holding is legitimate when the change window is closed or the monitoring stack is mid-upgrade; it is only legitimate when it is recorded with an owner, a stated exposure, and an end time. An undecided delay is not a hold.
Find the key the new chart actually reads. helm show values platform/metrics-stack --version 5.0.0 prints the new default values; compare it against the override in values/monitoring.yaml and identify which key the team was setting and what replaced it.
Pin the pipeline to the chart version that was last validated in staging by adding --version to the helm upgrade line. Pinning first means the next run is reproducible, which every later step depends on.
Correct the override in the repository, in a reviewed pull request, alongside the pin. Both changes belong to the same review because either one alone leaves a release that can still drift.
Render before you apply: helm template metrics-stack platform/metrics-stack --version <pinned> -f values/monitoring.yaml | grep -A3 cluster-admin must return nothing. This is the cheapest check in the whole procedure and it answers the question directly.
Run the pipeline. Confirm from helm get manifest metrics-stack -n monitoring that the rendered release no longer contains the binding, and that Helm has removed the object rather than orphaning it.
Review the audit log for the exposure window, filtering on the ServiceAccount as the requesting user. Read it for what the identity did, not for what it could have done: an over-permissioned identity that only ever performed its normal scrape discovery is a different incident from one that touched Secrets or RBAC.
Repair the nightly audit check in the same change. A check that cannot produce a true positive is a defect of the same severity as the finding it missed.

Verification

The effective permissions changed. kubectl auth can-i --list --as=system:serviceaccount:monitoring:prometheus no longer prints a *.* row, and the rules that remain match the minimum-surface ClusterRole.
The escalation primitive is gone. kubectl auth can-i create clusterrolebindings --as=system:serviceaccount:monitoring:prometheus answers no and exits non-zero.
The check can fail. Run the same command against an identity that genuinely is cluster-admin and confirm it answers yes. A control that reports success against every input is not reporting anything.
Only one binding remains. Re-run the kubectl get clusterrolebinding -o json filter and confirm it returns the team binding alone.
The workload still works. Every Prometheus target is up, and the Prometheus container log contains no forbidden or cannot list resource lines. Removing authority is only a success if the system that lost it still functions.
The fix survives a converge. Re-run the deploy pipeline and repeat the first two checks. This is where a hand-deleted object is exposed, and it is the step teams skip.
The repaired audit reports the truth. Point the fixed nightly check at a deliberately over-permissioned test ServiceAccount in a scratch namespace and confirm it raises a finding, then delete the test identity.
The two scanners now agree. Re-run the compliance scan and confirm both findings close, with the internal audit still clean for the right reason rather than the previous one.

Prevention

Pin the chart version, and change the pin in a pull request. An unpinned helm upgrade makes the deployed software a function of when the pipeline ran. Every incident that begins “nothing changed” and is caused by an unpinned dependency was preventable by one flag.
Assert the output, not the input. A values override is a request; a rendered manifest is a fact. Gate the pipeline on helm template output - specifically, on the absence of any binding to cluster-admin - so an override that silently stops applying fails the build instead of shipping.
Prefer charts with a values.schema.json, and ask for one where it is missing. A schema turns a renamed key from a silent no-op into an error at render time, which is the only place it is cheap to fix.
Audit effective permissions, never role names. The escalation questions worth asking are concrete and name-independent: can this identity create clusterrolebindings, can it get Secrets outside its namespace, does its rule list contain a *.* row. All three are one kubectl auth can-i call and none depend on knowing what anything is called.
Test the audit against a known-bad identity on a schedule. A detection control that has never fired is indistinguishable from a broken one. Create a deliberately over-permissioned ServiceAccount in a scratch namespace, confirm the audit finds it, and delete it - monthly, in the audit job itself.
Treat scanner disagreement as an incident. When two tools report opposite results about the same object, at least one is wrong, and which one is wrong matters more than the original finding. The morning the internal audit contradicted the external scanner was the morning the team learned their audit had never worked.
Alert on the creation of any binding to cluster-admin. It is a rare, high-signal event with almost no legitimate cause outside cluster bootstrap, and it is trivially detectable in the audit log.

What if the same upgrade had bound the namespace's default ServiceAccount instead?

Worse in reach, and considerably harder to unwind.

Every Pod that does not set spec.serviceAccountName uses the default ServiceAccount of its namespace, and by default that identity’s token is projected into the Pod at /var/run/secrets/kubernetes.io/serviceaccount/token. Binding default to cluster-admin therefore does not grant cluster-admin to one workload; it grants it to every current and future workload in that namespace that never made a decision about identity - including third-party sidecars, one-off debug Pods, and anything a developer applies next week.

The blast radius also grows quietly over time rather than appearing at once, which is what makes it hard to reason about after the fact. Any Pod created during the exposure window had a cluster-admin token on its filesystem, so the audit-log review has to cover every identity in the namespace rather than one.

Two controls make the default ServiceAccount inert and are worth having before you need them: bind it to nothing, and set automountServiceAccountToken: false on it so that Pods which never asked for an identity do not carry one. A workload that genuinely needs the API server then has to name its own ServiceAccount, which is the point.

Why this matters

The exposure in this incident lasted seven hours and, as far as the audit log shows, was never used. It is still the most instructive kind of finding, because every control the team had built behaved exactly as designed and the outcome was still wrong.

The chart upgrade was a supply-chain event that arrived through a pipeline nobody ran and a repository nobody changed. The override that stopped working did so without an error, because Helm cannot know that an unread key was meant to matter. The audit reported green because it was asking a question the API does not answer. None of these are bugs in the ordinary sense; they are gaps between what a tool promises and what a team assumed it promised.

The transferable habit is to distinguish, every time, between checking that an object is what you intended and checking what an identity can actually do. kubectl describe on the binding you wrote answers the first question and will answer it correctly for years while the second answer drifts. RBAC is additive, so the only honest question is the union - and the only honest way to ask it is to ask the authorizer.

Reported symptoms

Evidence

Root cause

Remediation

Verification

Prevention

Reported symptoms

Evidence provided

Work the evidence before reading on

Root cause

1. RBAC is additive, so the widest binding wins

2. The binding came back because the values key stopped existing

3. The audit grepped for a string the command never prints

Resolution

Verification

Prevention