Admission pipeline — the gatekeeper between authn and persistence

The admission pipeline is the gatekeeper between authentication and etcd persistence. Every write request — create, update, delete, patch — flows through the chain. The chain runs after RBAC (so unauthorised requests are rejected first) and before etcd (so the API server can mutate, validate, or reject before the change is persisted). This lesson covers the chain order, the failure modes, and the operational patterns.

The chain order

For a write request, the chain runs in this order:

1. Authentication — the API server authenticates the request and resolves the UserInfo.
2. Authorisation — RBAC (or other authorizers) decide whether the user can perform the action.
3. Mutating admission — webhooks and built-in controllers can mutate the object before persistence.
4. Object schema validation — the API server validates the object against the OpenAPI schema.
5. Validating admission — webhooks and built-in controllers validate the object; rejections here are final.
6. etcd persistence — the object is persisted.

flowchart LR
    R[Request] --> A[Authentication]
    A --> Z[Authorisation]
    Z --> M[Mutating admission]
    M --> S[Schema validation]
    S --> V[Validating admission]
    V --> E[etcd persistence]
    V -.reject.-> R[403 Forbidden]

The mutating phase runs first because some webhooks need to add fields (e.g., sidecar injection, default labels) before the validating phase can check them. The validating phase runs second so rejections are based on the final state of the object.

Built-in admission controllers

The API server ships with built-in admission controllers that always run (when enabled):

These are enabled with --enable-admission-plugins= on the API server. The recommended set for production:

--enable-admission-plugins=NodeRestriction,PodSecurity,LimitRanger,ServiceAccount,DefaultStorageClass,Priority,Event,MutatingAdmissionWebhook,ValidatingAdmissionWebhook

A controller that is not enabled is not enforced. The CIS Benchmark requires PodSecurity and NodeRestriction.

Dynamic admission (webhooks)

Webhooks extend the chain with custom logic:

• MutatingAdmissionWebhook — can modify the object before persistence (e.g., inject sidecars, set defaults).
• ValidatingAdmissionWebhook — can reject the object (e.g., enforce custom policies).

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionWebhookConfiguration
metadata:
  name: policy-validator
webhooks:
- name: validate.example.com
  clientConfig:
    service:
      name: policy-service
      namespace: policy
      path: /validate
    caBundle: <base64 CA>
  rules:
  - operations: ["CREATE", "UPDATE"]
    apiGroups: ["", "apps"]
    apiVersions: ["v1"]
    resources: ["pods", "deployments"]
  admissionReviewVersions: ["v1"]
  sideEffects: None
  failurePolicy: Fail  # reject if webhook is down
  namespaceSelector:
    matchExpressions:
    - key: policy.example.com/enforce
      operator: Exists

This webhook is invoked for every Pod and Deployment in namespaces with the policy.example.com/enforce label. If the webhook is down and failurePolicy: Fail, the request is rejected.

ValidatingAdmissionPolicy (CEL)

The 1.30+ declarative alternative to webhooks is the ValidatingAdmissionPolicy:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: no-privileged
spec:
  failurePolicy: Fail
  rules:
  - operations: ["CREATE", "UPDATE"]
    apiGroups: [""]
    apiVersions: ["v1"]
    resources: ["pods"]
  validations:
  - expression: "object.spec.containers.all(c, !c.securityContext.privileged == true)"
    message: "privileged containers are not allowed"

The policy is expressed in CEL (Common Expression Language); the API server evaluates it without calling out to a webhook. CEL-based policies are faster, more reliable, and easier to audit.

flowchart LR
    A[ValidatingAdmissionPolicy] --> B[CEL evaluation]
    B -->|pass| C[Allow]
    B -->|fail| D[Reject with message]
    B -->|error| F[Apply failurePolicy]

Common failure modes

1. Webhook timeout. The webhook takes longer than the configured timeout (default 10 seconds); the request is rejected (with Fail) or allowed (with Ignore). The fix is to make the webhook faster, or increase the timeout for non-critical webhooks.
2. Webhook down. The webhook service is unreachable; the request is rejected (with Fail) or allowed (with Ignore). The fix is HA replicas and observability.
3. Webhook misconfiguration. The webhook returns the wrong shape; the API server rejects the response. The fix is to test the webhook before deployment.
4. Built-in controller disabled. A controller like PodSecurity is not enabled; the policy is not enforced. The fix is to enable the controller and verify with a test workload.
5. Order conflict. Two mutating webhooks modify the same field; the result is non-deterministic. The fix is to design webhooks for orthogonal mutations.

Production failure modes

1. failurePolicy: Ignore on a security webhook. The webhook is a bypass. The fix is Fail.
2. Single webhook replica. The webhook is a single point of failure. The fix is 3+ replicas.
3. Webhook timeout too short. The webhook is legitimate but slow; the request is rejected. The fix is to tune the timeout (10s default; longer for complex policies).
4. No webhook observability. The webhook is silent; failures are invisible. The fix is to log every admission decision and export to the SIEM.

Cross-course references

• The Observability course covers the audit log entries for admission decisions.
• The Linux course covers the TLS and CA management that the webhook’s client config requires.

Quiz

Production discipline

The admission chain is the gatekeeper for every write. A defensible admission programme enables the required built-in controllers (PodSecurity, NodeRestriction), uses ValidatingAdmissionPolicy for declarative policies, uses webhooks with failurePolicy: Fail for external logic, and runs the webhooks with HA replicas and observability. The audit log is the proof: every admission decision is recorded. A cluster whose admission chain is enabled and tested has a gatekeeper that is auditable; a cluster whose admission chain is bypassed (Ignore) or disabled has a gatekeeper that is not.