MutatingAdmissionWebhook — modifying objects at admission

A MutatingAdmissionWebhook is an admission controller that can modify an object before persistence. The mutating phase runs after RBAC and before validation; the webhook can add fields, change values, or remove fields. This lesson covers the common use cases, the configuration, the idempotency requirement, and the operational risks.

How mutating webhooks work

When a request reaches the mutating phase, the API server sends an AdmissionReview to each registered mutating webhook. The webhook returns a patch that modifies the object (a JSON Patch, by default). The API server applies the patches in order, then proceeds to validation.

sequenceDiagram
    participant AS as API server
    participant WH as Webhook
    AS->>WH: AdmissionReview (object, request)
    WH->>WH: Inspect and decide
    WH->>AS: AdmissionResponse (patch or allow)
    AS->>AS: Apply patch
    AS->>WH: Re-invoke if re-mutated
    AS->>AS: Validation phase

The API server may call the webhook multiple times for the same object (re-invocation) if the patch triggers another mutation. The re-invocation limit is 16 by default; exceeding it is treated as an error.

Common use cases

Three use cases recur:

1. Sidecar injection. A service mesh (Istio, Linkerd) injects an envoy proxy into every Pod:

[
  {
    "op": "add",
    "path": "/spec/containers/-",
    "value": {
      "name": "istio-proxy",
      "image": "istio/proxyv2:1.20.0",
      "args": ["proxy", "sidecar"]
    }
  }
]

The webhook checks for the istio-proxy container’s existence; if not present, it adds it. Idempotent.

2. Default labels. A platform team wants every workload to have app.kubernetes.io/name. The webhook adds the label if it is missing.

[
  {
    "op": "add",
    "path": "/metadata/labels/app.kubernetes.io~1name",
    "value": "default-name"
  }
]

3. Image rewriting. A registry mirror rewrites myapp:v1.0 to mirror.example.com/myapp:v1.0:

[
  {
    "op": "replace",
    "path": "/spec/containers/0/image",
    "value": "mirror.example.com/myapp:v1.0"
  }
]

Configuration

A mutating webhook is configured via a MutatingWebhookConfiguration:

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: sidecar-injector
webhooks:
- name: sidecar.injector.example.com
  clientConfig:
    service:
      name: injector
      namespace: policy
      path: /inject
    caBundle: <base64 CA>
  rules:
  - operations: ["CREATE"]
    apiGroups: [""]
    apiVersions: ["v1"]
    resources: ["pods"]
  admissionReviewVersions: ["v1"]
  sideEffects: None
  failurePolicy: Ignore  # mutations are best-effort
  namespaceSelector:
    matchExpressions:
    - key: sidecar-injection
      operator: In
      values: ["enabled"]

The webhook applies to Pods in namespaces with the sidecar-injection=enabled label. The failurePolicy: Ignore is the conventional setting for mutating webhooks: an outage does not block the request, but the mutation is not applied.

The idempotency requirement

A mutating webhook must be idempotent. The API server re-invokes the webhook if the previous call returned a patch that may have triggered another mutation.

# Idempotent sidecar injection (Python pseudocode)
def inject(request):
    pod = request.object
    if any(c.name == "istio-proxy" for c in pod.spec.containers):
        return AdmissionResponse(allowed=True)  # already injected
    patch = jsonpatch.make_patch(pod, add_sidecar(pod))
    return AdmissionResponse(allowed=True, patch=patch)

A non-idempotent webhook returns a patch every time; the API server keeps invoking; eventually the re-invocation limit is exceeded and the request is rejected.

Operational risks

1. Re-mutation across webhooks. Two webhooks modify the same field; the result is non-deterministic. The fix is to design webhooks for orthogonal mutations.
2. Idempotency failure. A non-idempotent webhook triggers the re-invocation limit. The fix is to check before mutating.
3. Outage with Ignore. A webhook outage means mutations are skipped. A security-relevant mutation (e.g., adding an audit-logging sidecar) is silently bypassed. The fix is to choose Ignore or Fail based on the criticality of the mutation.
4. Latency. Each mutating webhook adds latency. Multiple webhooks compound. The fix is to keep webhooks fast (under 100ms) and limit the chain.

Common failure modes

1. Webhook timeout. The webhook takes longer than the timeout (default 10 seconds); the request is rejected or the mutation is skipped. The fix is to optimise the webhook or increase the timeout.
2. Webhook down. The webhook service is unreachable; the mutation is skipped (Ignore). The fix is HA replicas.
3. Wrong patch format. The webhook returns a malformed patch; the API server rejects. The fix is to test the patch format.
4. Side effects. A webhook with sideEffects: Unknown is rejected by Kubernetes 1.27+; the required value is None (or Some with sideEffectClass documented).

Cross-course references

• The Observability course covers the audit log entries for mutating webhook invocations.
• The Linux course covers the TLS and CA management that the webhook’s client config requires.

Quiz

Production discipline

Mutating webhooks are the right primitive for sidecar injection, default labelling, image rewriting, and similar best-effort mutations. A defensible mutating-webhook programme keeps the webhook idempotent, chooses Ignore or Fail based on the mutation’s criticality, runs HA replicas with observability, and tests the webhook’s idempotency in CI/CD. The re-invocation limit is a hard ceiling — a non-idempotent webhook is a Critical finding. A cluster whose mutating webhooks are idempotent and fast is a cluster whose admission chain is reliable.