ValidatingAdmissionWebhook — external policy at admission

A ValidatingAdmissionWebhook is an admission controller that calls an external service for a policy decision. The service can accept or reject; it cannot mutate. This lesson covers the configuration, the failurePolicy choice, the common use cases, and the operational patterns for running webhooks reliably.

How validating webhooks work

The validating phase runs after the mutating phase. For each registered webhook, the API server sends an AdmissionReview request and waits for a decision. The decision is binary: allow or deny.

sequenceDiagram
    participant AS as API server
    participant WH as Webhook
    AS->>WH: AdmissionReview (object, request)
    WH->>WH: Evaluate policy
    WH->>AS: AdmissionResponse (allow or deny with reason)
    AS->>AS: Apply decision

The webhook’s response includes a status field that is returned to the user if the request is denied:

{
  "apiVersion": "admission.k8s.io/v1",
  "kind": "AdmissionReview",
  "response": {
    "uid": "...",
    "allowed": false,
    "status": {
      "message": "image registry.example.com/myapp:v1.0 is not signed"
    }
  }
}

The user sees the message in the kubectl output.

Configuration

A validating webhook is configured via ValidatingWebhookConfiguration:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionWebhookConfiguration
metadata:
  name: image-signature
webhooks:
- name: verify.example.com
  clientConfig:
    service:
      name: cosign-verifier
      namespace: policy
      path: /verify
    caBundle: <base64 CA>
  rules:
  - operations: ["CREATE", "UPDATE"]
    apiGroups: [""]
    apiVersions: ["v1"]
    resources: ["pods"]
  admissionReviewVersions: ["v1"]
  sideEffects: None
  failurePolicy: Fail  # reject if webhook is down
  namespaceSelector:
    matchExpressions:
    - key: image-signature
      operator: In
      values: ["required"]

The webhook applies to Pods in namespaces with the image-signature=required label. If the webhook is down and failurePolicy: Fail, the request is rejected.

Common use cases

Four use cases recur:

1. OPA / Gatekeeper. A Rego policy bundle is evaluated against the object. Use cases: complex multi-resource policies, policy as code.

2. Kyverno. A Kyverno policy is evaluated against the object. Use cases: validation, mutation, image verification.

3. Cosign signature verification. A Cosign verifier checks that every container image is signed by a trusted key. Use cases: supply chain security.

4. Third-party policy engines. Conftest, jsPolicy, and others. Use cases: integration with existing policy infrastructure.

Choosing the failurePolicy

failurePolicy is the most consequential configuration:

• Fail (the secure default): reject the request if the webhook is down. Use for security-relevant policies (image signature, PSS enforcement, privilege restriction).
• Ignore (the permissive default): allow the request if the webhook is down. Use for advisory policies (linting, naming conventions).

The CIS Benchmark requires Fail for security webhooks. A webhook with Ignore is a bypass that an attacker can exploit.

flowchart LR
    A[Webhook] -->|reachable| B[Decision]
    A -->|unreachable| F{failurePolicy}
    F -->|Fail| G[Reject]
    F -->|Ignore| H[Allow]

Operational patterns

A defensible validating webhook programme:

1. HA replicas. Run 3+ webhook Pods behind a Service. A single replica is a single point of failure.
2. Latency SLO. The webhook must respond in under 500ms; the API server’s timeout is 10 seconds. Slow webhooks degrade every API call.
3. Observability. Every decision is logged with the policy name, the resource, the decision, and the latency. SIEM rules alert on outages and on unexpected rejections.
4. Readiness probe. The webhook Pod has a readiness probe that reports the webhook’s health. The Service routes traffic away from unhealthy Pods.
5. Cache. A webhook that depends on external data (e.g., a Cosign registry) caches the data for the appropriate TTL.

Production failure modes

1. Single webhook replica. The webhook is a single point of failure. The fix is HA replicas.
2. failurePolicy: Ignore on a security webhook. The webhook is a bypass. The fix is Fail.
3. Slow webhook. The webhook takes >10s; the request times out. The fix is to make the webhook faster or to increase the timeout.
4. No observability. Webhook outages and rejections are invisible. The fix is to log every decision and alert on anomalies.
5. Webhook has side effects. A webhook that calls an external service with side effects (e.g., sending an email) must set sideEffects: None with documentation. The fix is to make the webhook idempotent and read-only.

Cross-course references

• The Observability course covers the audit log entries for webhook decisions.
• The Linux course covers the TLS and CA management that the webhook’s client config requires.

Quiz

Production discipline

Validating webhooks are the right primitive for external policy. A defensible webhook programme uses HA replicas with an SLO on latency, chooses Fail for security and Ignore for advisory, observes every decision in the SIEM, and documents the disable procedure in the runbook. A cluster whose webhooks are HA, fast, and observed is a cluster whose admission chain is reliable; a cluster whose webhooks are single-replica and silent is a cluster whose admission chain is a bottleneck.