Troubleshooting authentication — the diagnostic workflow

Authentication failures in Kubernetes are frustrating because the error messages do not always point to the cause. A 401 Unauthorized could mean a missing token, an expired token, a wrong CA, a wrong context, or an IdP issue. This lesson walks the systematic diagnostic workflow, from kubectl auth whoami to the API server logs, and the remediation for each common failure mode.

The diagnostic workflow

Five steps, in order:

1. kubectl auth whoami — verify the current identity. If this fails, the issue is at the authentication layer.
2. kubectl config view --minify — inspect the kubeconfig. The context, the user, the cluster, and the namespace are all in the kubeconfig.
3. Inspect the credentials — for OIDC, the ~/.kube/cache/ directory; for client cert, the cert file; for projected tokens, the /var/run/secrets/kubernetes.io/serviceaccount/ mount.
4. API server logs — the kube-apiserver logs every authentication decision. The authentication flag includes the request’s headers and the matched authenticator.
5. Audit log — the audit log records every authentication decision with the UserInfo. The user.username is the resulting identity.

flowchart TD
    A[kubectl auth whoami] -->|fail| B[kubectl config view]
    B --> C[Inspect credentials]
    C --> D[API server logs]
    D --> E[Audit log]
    E --> F[Remediation]

Failure mode 1: expired token

Symptom: kubectl auth whoami returns Unauthorized: invalid bearer token. The token’s expiry has passed.

# Check the token's expiry
kubectl auth whoami -v=8 | grep -i exp
# Or:
TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
echo "$TOKEN" | cut -d. -f2 | base64 -d 2>/dev/null | jq .exp
# 1755356400 → 2026-08-16 14:00:00 UTC
date -u
# Current time

Remediation: refresh the token.

• For OIDC: clear the cache and re-authenticate. rm -rf ~/.kube/cache/oidc-* && kubectl get pods (opens the browser).
• For projected tokens inside a Pod: the kubelet rotates automatically; if the token is expired, the workload is reading a cached value. The fix is to re-read the file before each request.
• For CI/CD: re-issue the token per run.

Failure mode 2: wrong context

Symptom: kubectl auth whoami succeeds but returns the wrong identity or works against the wrong cluster.

# Check the current context
kubectl config current-context
# alice-staging  (wrong)

# List contexts
kubectl config get-contexts
# alice-prod    prod.example.com    alice
# alice-staging staging.example.com alice

# Switch
kubectl config use-context alice-prod

# Verify
kubectl auth whoami
# Username: alice@example.com (prod)

Remediation: switch the context and verify.

Failure mode 3: missing CA bundle

Symptom: kubectl auth whoami returns x509: certificate signed by unknown authority. The client’s CA bundle does not match the API server’s serving cert.

# View the cluster's CA
kubectl config view --minify --flatten -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' | base64 -d
# Should match the cluster's CA cert

# Or use insecure-skip-tls-verify (only for debugging)
kubectl --insecure-skip-tls-verify auth whoami
# Then fix the CA bundle

Remediation: update the kubeconfig’s certificate-authority-data or certificate-authority field.

Failure mode 4: anonymous denied

Symptom: kubectl auth whoami returns Unauthorized. The request is anonymous; the API server is configured with --anonymous-auth=false.

# Check the API server flag
kubectl -n kube-system get pod -l component=kube-apiserver \
  -o jsonpath='{.items[*].spec.containers[*].args}' | tr ' ' '\n' | grep anonymous-auth
# --anonymous-auth=false

Remediation: present credentials. The flag is correct; the request must include a token, cert, or impersonation.

Failure mode 5: RBAC denial masquerading as authn

Symptom: kubectl auth whoami succeeds (you have an identity), but kubectl get pods returns Forbidden. RBAC has denied the verb on the resource.

# Confirm RBAC is the issue
kubectl auth can-i get pods -n prod --as=alice
# no

# Check the user's bindings
kubectl get rolebindings,clusterrolebindings -A -o json | \
  jq '.items[] | select(.subjects[]?.name == "alice")'

# Verify the user's groups
kubectl auth whoami
# Groups: [idp:dev]
# Check the IdP mapping and the RBAC binding's group

Remediation: add a binding for the user or group, or ask the cluster admin to do so.

Failure mode 6: projected token not mounted

Symptom: inside a Pod, kubectl auth whoami (or any API call) returns Unauthorized. The Pod’s SA token is not mounted.

# Substitute your own value before running:
POD=web-5f9c7d8b6c-2xk9p

# Inside the Pod
ls -la /var/run/secrets/kubernetes.io/serviceaccount/
# token file is missing or empty

# Check the Pod spec
kubectl get pod "$POD" -o yaml | grep -A1 automountServiceAccountToken
# automountServiceAccountToken: false

Remediation: set automountServiceAccountToken: true on the Pod or on the SA. If the SA does not exist, create it.

flowchart LR
    A[Symptom: Unauthorized] --> B{Inside Pod?}
    B -->|yes| C[Check token file]
    C -->|missing| D[automountServiceAccountToken=false]
    C -->|empty| E[SA not found]
    C -->|expired| F[Client caching]
    B -->|no| G[kubectl config view]
    G -->|wrong context| H[Switch context]
    G -->|wrong creds| I[Update credentials]
    G -->|wrong CA| J[Update CA bundle]

Production failure modes

1. Diagnosis stops at the kubeconfig. The operator fixes the context but does not check the token’s expiry. The next failure (in an hour) is a repeat. The fix is the full workflow.
2. API server logs are not searched. The diagnostic stops at whoami and config view. The API server’s authentication log has the authoritative answer. The fix is to ship logs to the SIEM with alerting on auth failures.
3. Audit log is not consulted. The audit log has the UserInfo and the matched authenticator. The fix is to enable audit logging at RequestResponse level for authentication events.

Cross-course references

• The Observability course covers the SIEM rules for authentication failures.
• The Linux course covers the file permissions for the kubeconfig and the credentials cache.

Quiz

Production discipline

A defensible authentication troubleshooting workflow follows: whoami → kubeconfig → credentials → API server logs → audit log. The first command is always kubectl auth whoami; the first diagnostic question is always “what identity does the cluster see?” The fix for each failure mode is well known: expired token → refresh; wrong context → switch; bad CA → update; anonymous denied → add credentials; RBAC denial → add binding; projected token missing → mount. The discipline is to follow the workflow every time, never to skip steps, and to document the remediation in the runbook so the next operator does not re-discover it.