Admission best practices — the operational discipline

A defensible admission programme layers controls: built-in controllers for the well-known policies (Pod Security, NodeRestriction), declarative policies (ValidatingAdmissionPolicy, Kyverno) for the workload-specific rules, and webhooks for the external lookups (signature verification, OPA Rego bundles, third-party integrations). This lesson covers the layered approach, the failurePolicy choices, the observability, and the production patterns.

The layered approach

Three layers, each with a different role:

flowchart LR
    L1[Layer 1: Built-in controllers] --> L2[Layer 2: Declarative policies]
    L2 --> L3[Layer 3: Webhooks for external lookups]
    L1 --> A[Admission chain]
    L2 --> A
    L3 --> A

Layer 1 — Built-in controllers. The API server’s built-in controllers enforce the well-known policies without external dependencies:

• PodSecurity for PSS enforcement.
• NodeRestriction for kubelet permissions.
• LimitRanger for default limits.
• ResourceQuota for namespace quotas.
• ServiceAccount for SA defaulting.

These run in the API server itself; they are fast, reliable, and don’t need a separate deployment. The CIS Benchmark requires PodSecurity and NodeRestriction.

Layer 2 — Declarative policies. For workload-specific rules (image registry, label requirements, resource limits), use ValidatingAdmissionPolicy (CEL) or Kyverno. These are evaluated in the API server or in a small controller; they don’t depend on external services.

Layer 3 — Webhooks for external lookups. For policies that require external data (signature verification against Cosign, OPA bundles from an external server, third-party integrations), use a webhook. The webhook is a separate service with its own HA, observability, and SLO.

The failurePolicy choice

For each webhook, choose Fail or Ignore:

• Fail for security webhooks: image signature, privilege restriction, PSS enforcement. The webhook outage means the policy cannot be enforced; the request is rejected.
• Ignore for advisory webhooks: linting, naming conventions, optional best practices. The webhook outage means the policy is skipped; the request is allowed.

The CIS Benchmark requires Fail for security webhooks. A webhook with Ignore is a bypass that an attacker can exploit.

flowchart LR
    A[Webhook] -->|reachable| B[Decision]
    A -->|unreachable| F{failurePolicy}
    F -->|Fail| G[Reject]
    F -->|Ignore| H[Allow]

Observability

Every admission decision should be observable:

1. Audit log. Enable audit logging at RequestResponse level for the admission phase. The audit log records every decision with the UserInfo, the policy name, the resource, and the decision.
2. Webhook metrics. Every webhook exposes Prometheus metrics: request rate, latency, success rate, error rate. SIEM rules alert on anomalies.
3. Admission events. Rejected requests produce Kubernetes events. kubectl get events shows the policy violations.
4. Policy audit logs. ValidatingAdmissionPolicy decisions are logged with the policy name and the reason.

A cluster without admission observability has a gatekeeper that is silent; a cluster with observability has a gatekeeper that is auditable.

Migration from webhooks to CEL

For policies that can be expressed in CEL, the migration from webhooks to CEL improves reliability:

1. Identify the policy. Find a webhook whose handler logic is purely declarative (no external lookups).
2. Express in CEL. Translate the logic into CEL expressions.
3. Deploy the CEL policy. Create the ValidatingAdmissionPolicy and binding.
4. Run in Warn mode. Set validationActions: [Warn].
5. Verify. Check that the warning events match the expected rejections.
6. Switch to Deny. Set validationActions: [Deny].
7. Disable the webhook. Delete the ValidatingAdmissionWebhookConfiguration.

A migration reduces the cluster’s dependence on external services for policy enforcement.

Production failure modes

1. All webhooks, no CEL. The cluster’s admission chain depends on webhooks for everything; an outage blocks the cluster. The fix is to migrate declarative policies to CEL.
2. Ignore on a security webhook. The webhook is a bypass. The fix is Fail.
3. Single webhook replica. The webhook is a single point of failure. The fix is HA replicas.
4. No observability. Admission decisions are invisible. The fix is audit logging + Prometheus metrics + SIEM alerts.
5. Built-in controllers disabled. The PodSecurity controller is off; PSS is not enforced. The fix is to enable the controller.

Cross-course references

• The Observability course covers the audit log and SIEM integration.
• The Linux course covers the TLS and CA management for webhook client config.

Quiz

Production discipline

A defensible admission programme layers controls: built-in for the well-known policies, declarative for the workload-specific rules, webhooks for the external lookups. Every webhook has the right failurePolicy (Fail for security, Ignore for advisory), HA replicas, observability, and a documented disable procedure. The audit log is the proof: every decision is recorded. A cluster whose admission chain is layered, observed, and documented is a cluster whose gatekeeper is auditable; a cluster whose admission chain is webhooks-only and silent is not.