The API server pipeline — authentication, authorisation, admission, validation

Every request that reaches the API server — from kubectl, from a controller, from kubelet, from a webhook — runs through the same pipeline. This lesson walks the stages, identifies what each one enforces, and shows how to diagnose the rejection that comes out of any of them.

The pipeline, with reasons for rejection

flowchart LR
    R[Request] --> TLS[TLS termination]
    TLS --> A[Authentication<br/>401 if unknown]
    A --> AZ[Authorisation<br/>403 if denied]
    AZ --> MA[Mutating Admission<br/>may modify object]
    MA --> VA[Validating Admission<br/>may reject object]
    VA --> SV[Schema validation<br/>OpenAPI]
    SV --> P[Persistence<br/>write to etcd]

Each stage can reject. The HTTP status code tells you which stage:

A 403 is not the same as a 422. The first is “you can’t do this”; the second is “the object is wrong”. Production diagnosis starts with reading the status code.

Stage 1: Authentication

The API server tries every configured authenticator in turn. The first that returns a known identity wins; if no authenticator returns an identity, the request is rejected with 401.

flowchart LR
    R[Request with credentials] --> X[X.509 client cert]
    R --> B[Bearer token]
    R --> S[ServiceAccount token]
    R --> O[OIDC token]
    X --> K[kube-apiserver<br/>returns UserInfo]
    B --> K
    S --> K
    O --> K
    K -->|unknown| N[401 Unauthorized]

X.509 client certificates

The TLS handshake itself authenticates the client by certificate. The Common Name (CN) becomes the username; the Organisation (O) becomes the groups. Production clusters use this for ServiceAccounts (legacy, deprecated in 1.24+) and for human admin certificates.

# /etc/kubernetes/pki/admin.conf
client-certificate-data: BASE64
client-key-data: BASE64

Bearer tokens

A bearer token is presented as Authorization: Bearer <token>. The API server validates the token’s signature against a configured key set. Tokens are:

• ServiceAccount projected tokens — JWT signed by the API server; used by Pods.
• OIDC tokens — JWT signed by the configured OIDC provider.
• Legacy bearer tokens — long-lived; deprecated in 1.24+ and removed from many deployments.

OIDC integration

The API server is configured with an OIDC issuer URL, client ID, and certificate authority. Tokens issued by the OIDC IdP (e.g., Dex, Okta, Azure AD) are accepted; the username and groups come from configurable claims.

kube-apiserver \
  --oidc-issuer-url=https://dex.example.com \
  --oidc-client-id=kubernetes \
  --oidc-username-claim=email \
  --oidc-groups-claim=groups

Production clusters increasingly use OIDC for human access (short-lived tokens, MFA, central audit) and projected ServiceAccount tokens for workloads.

Anonymous requests

If --anonymous-auth=true (default), the API server treats unauthenticated requests as the system:anonymous user and system:unauthenticated group. RBAC can be configured to allow or deny this user. Most production clusters disable anonymous auth (--anonymous-auth=false).

Stage 2: Authorisation

Once authenticated, the API server asks: is this identity allowed to perform this verb on this resource, in this namespace? Authorisers are tried in order; the first to explicitly allow or deny wins.

flowchart LR
    R[Authenticated request] --> N[Node]
    N -->|not kubelet| RB[RBAC]
    RB -->|no role| W[Webhook]
    W -->|no response| AB[ABAC]
    AB -->|no allow| AA[AlwaysAllow / AlwaysDeny]

Node authoriser

Special-case authoriser for kubelet. Allows kubelets to write Pods and Node status for their own node only. Production clusters always enable this.

kind: ClusterRole
metadata:
  name: system:node
rules:
- apiGroups: [""]
  resources: ["pods", "pods/status", "nodes", "nodes/status"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

RBAC

The default production authoriser. Roles define allowed verbs; bindings grant roles to identities.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: team-a-prod
  name: developer
rules:
- apiGroups: [""]
  resources: ["pods", "pods/log"]
  verbs: ["get", "list", "watch"]

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: team-a-prod
  name: developers
subjects:
- kind: Group
  name: developers
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: developer
  apiGroup: rbac.authorization.k8s.io

A request is allowed if at least one binding in the chain permits the action. RBAC decisions are pure functions of identity, resource, verb, and (sometimes) subresource.

Webhook authoriser

Delegates the decision to an external service. Useful for centralised policy (OPA, Cerbos, custom). The webhook returns Allow or Deny; an unreachable webhook is treated as Deny by default (--authorization-webhook-failure-policy=...).

ABAC

Attribute-based; deprecated. Not used in production.

Default deny

If no authoriser allows, the request is denied. The default in production is --authorization-mode=Node,RBAC.

kubectl auth can-i delete pods -n team-a-prod

This tests what the current identity can do. Production operators run this before any action that might fail with 403.

Stage 3: Mutating Admission

Mutating webhooks run before schema validation. They can modify the object. The built-in mutating admission controllers:

• Namespace defaulting — fills in metadata.namespace when not specified.
• ServiceAccount injection — sets spec.serviceAccountName to default when not specified.
• PodSecurityContext defaulting — applies cluster defaults.
• Image pull secret defaulting — adds imagePullSecrets from the ServiceAccount.
• LimitRanger — applies default request/limit from LimitRange.
• PodSecurity (mutating side) — applies Pod Security Standards (privileged → baseline → restricted).

Then configured mutating webhooks in MutatingWebhookConfiguration:

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: sidecar-injector
webhooks:
- name: sidecar.istio.io
  sideEffects: None
  admissionReviewVersions: ["v1"]
  clientConfig:
    service: {name: istiod, namespace: istio-system, path: /inject}
  rules:
  - operations: ["CREATE"]
    apiGroups: [""]
    resources: ["pods"]
  failurePolicy: Fail
  namespaceSelector:
    matchLabels:
      istio-injection: enabled

A failing mutating webhook (timeout, error) rejects the request unless failurePolicy: Ignore (risky in production).

Stage 4: Validating Admission

Validating webhooks run after schema validation. They can only accept or reject; they cannot modify. Built-in:

• PodSecurity (validating side) — rejects Pods that violate the namespace’s Pod Security Standards.
• LimitRanger (validating side) — rejects Pods that exceed LimitRange.
• ResourceQuota — rejects Pods that would exceed the namespace’s quota.
• EventRateLimit (deprecated) — rate-limit excessive requests.

Custom validating webhooks handle policy that doesn’t fit the built-ins (image registry allowlists, label enforcement, image digest pinning).

Stage 5: Schema validation

The OpenAPI schema for the object’s apiVersion/kind is checked. A field that does not match the schema rejects with 400 Bad Request. Common failures:

• Typo in field name (replcas instead of replicas)
• Wrong type (string vs number)
• Missing required field
• Unknown field (typo, deprecated field, or the field is in a newer API version)

kubectl explain <resource>.<field> shows the schema:

kubectl explain deployment.spec.strategy

RESOURCE: strategy <Object>
DESCRIPTION:
  ...
FIELDS:
  rollingUpdate    <Object>
    maxSurge       <string>     (or int)
    maxUnavailable <string>     (or int)
  type             <string> -required-
    Enum: [Recreate RollingUpdate]

Stage 6: Persistence

The validated object is encoded and written to etcd. The write includes the resourceVersion the client expected; mismatch returns 409 Conflict. etcd commit failure returns 500 Internal Server Error.

The API server then returns 200 OK (or 201 Created for a new object) with the persisted object’s full representation.

How to diagnose request rejections

401 Unauthorized

No identity. Check credentials:

kubectl config view --minify
kubectl auth whoami    # requires auth-whoami subresource

403 Forbidden

Identity known but lacks permission. Check RBAC:

# Substitute the verb, resource and namespace you are checking:
VERB=create
RESOURCE=deployments
NS=prod-app

kubectl auth can-i "$VERB" "$RESOURCE" -n "$NS"

For deeper diagnosis, enable audit logging and read the audit log:

journalctl -u kube-apiserver | grep -i "forbidden\|denied" | tail

422 Unprocessable Entity

Admission rejection. The API server response body names the webhook that rejected:

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "admission webhook \"deny-privileged\" denied the request: ...",
  "code": 422
}

400 Bad Request

Schema validation. Check the manifest against the API:

# Substitute the resource and the field the error named:
RESOURCE=deployment
FIELD=spec.template.spec.containers

kubectl explain "$RESOURCE.$FIELD"
kubectl apply --dry-run=server -f manifest.yaml

500 Internal Server Error

etcd failure. Check the etcd cluster:

etcdctl endpoint status
etcdctl endpoint health

Audit logging

The API server logs every request that triggers a state change (or matches an audit policy) to an audit log. Audit logging is configured with --audit-policy-file and --audit-log-* flags.

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: RequestResponse
  namespaces: ["prod-*"]
  verbs: ["create", "update", "patch", "delete"]
- level: Metadata
  resources:
  - group: "" 
    resources: ["secrets"]

Audit logs are the source of truth for “who did what, when, and what changed”. Production clusters must ship audit logs to a central store with retention and tamper protection.

Cross-course references

• The Linux course part XXVI-Linux-SSH covers authentication primitives that map onto the API server’s authentication model.
• The Observability course part IX-Observability-Exporters covers the metrics the API server exposes about its own pipeline.
• The Linux course part XXVII-Linux-Auth covers central identity (LDAP, Kerberos) which is a common backend for OIDC integration.
• The Docker course part XXXVIII-Docker-Secrets covers secret management patterns that interact with the API server’s admission pipeline.

Quiz

Production discipline

• Treat the API server pipeline as multi-layered defence. Authentication proves identity; authorisation proves the identity is permitted; admission proves the object is safe. No one layer replaces another.
• Disable anonymous auth in production (--anonymous-auth= false).
• Configure every admission webhook with HA and explicit failurePolicy. A flaky webhook can block the cluster.
• Enable audit logging for every state-changing request; ship to a central, tamper-resistant store.
• Test RBAC with kubectl auth can-i before any privileged operation. RBAC confusion is the most common reason operators use cluster-admin for everything.