Request lifecycle — authentication, authorisation, admission

A kubectl apply that writes a single object to the cluster passes through ~10 distinct pipeline stages on its way to etcd. Each stage enforces a different correctness invariant. A request that succeeds is the result of every stage passing. A failure at any stage produces a specific error class. This lesson walks the stages and what each one enforces.

The stages

flowchart LR
    A[Client] -->|TLS| B[TLS termination]
    B -->|authn| C[Authentication]
    C -->|authz| D[Authorisation RBAC]
    D -->|mutate| E[Mutating admission]
    E -->|validate| F[Schema validation]
    F -->|concurrency| G[Optimistic concurrency]
    G -->|validate| H[Validating admission]
    H -->|encrypt| I[Encryption at rest]
    I -->|commit| J[etcd write]
    J -->|log| K[Audit log]
    K -->|fanout| L[Watch events]
    L --> M[Response 200 OK]

The eleven stages, with their typical costs:

A healthy cluster has sub-100ms p99 API request latency. A failing cluster has a specific stage at > 1 second; identifying the slow stage is the diagnosis.

Stage 1 — TLS termination

The API server terminates TLS on the secure port (default 6443). The handshake establishes the client’s identity via the certificate.

# Sample TLS handshake
Client -> Server: ClientHello (TLS 1.3)
Server -> Client: ServerHello, Certificate, CertificateVerify
Client -> Server: Finished
Server -> Client: Finished

The server’s certificate chain is verified against the cluster’s CA bundle. If the chain is broken or the certificate is expired, the handshake fails.

Stage 2 — Authentication

TLS establishes transport; authn establishes the user’s identity. Six authn methods:

• X.509 client certs.
• Bearer tokens (ServiceAccount tokens, legacy static tokens).
• OIDC (configured via --oidc-* flags).
• Webhook token authentication.
• Anonymous (rarely enabled in production).
• ServiceAccount tokens projected into Pods.

The authn stage resolves to a User object with username, groups, and extra. For a client cert:

User:
  username: client
  groups:
  - system:authenticated
  - system:masters
extra:
  cn: kubernetes-admin

For a ServiceAccount token:

User:
  username: system:serviceaccount:prod:web
  groups:
  - system:serviceaccounts
  - system:serviceaccounts:prod
  - system:authenticated

The authn stage is cached; an identity verified recently is not re-verified for the duration of the cache.

$ kubectl auth whoami

system:anonymous

For an unauthenticated request (anonymous auth is enabled), the user has no verified identity.

Stage 3 — Authorisation (RBAC)

The authorisation plugin decides whether the user can perform the verb on the resource.

SubjectAccessReview {
  spec: {
    user: "system:serviceaccount:prod:web",
    groups: ["system:serviceaccounts"],
    resource: "pods",
    verb: "get",
    namespace: "prod",
  }
  status: {
    allowed: true,
    reason: "RBAC: allowed by RoleBinding ..."
  }
}

The RBAC evaluation is:

1. Look up RoleBindings/ClusterRoleBindings for the user.
2. For each binding, evaluate if the verb is allowed on the resource in the scope (cluster-wide or namespace).
3. If any binding allows, return allowed=true; otherwise 403 Forbidden.

Stage 4 — Mutating admission

Mutating admission modifies the request before persistence. The chain of mutators runs in order:

sequenceDiagram
    autonumber
    participant Mut1 as MutatingWebhookConfiguration A
    participant Mut2 as MutatingWebhookConfiguration B
    participant SD as Defaulting mutator
    Mut1->>Mut1: receive AdmissionReview
    Mut1-->>Mut2: patched object
    Mut2->>Mut2: receive AdmissionReview
    Mut2-->>SD: patched object
    SD->>SD: apply built-in defaults
    SD-->>SD: defaulted object

A common mutating webhook is a service mesh sidecar injector that adds an envoy container to every Pod.

The mutating stage’s latency is dominated by the webhook’s response time. A webhook that takes 5 seconds to respond slows every request through it.

Stage 5 — Schema validation

The mutator’s output is validated against the OpenAPI schema. Schema validation is fast (microseconds) but strict:

• Required fields present.
• Field types correct.
• Enumerations respected.
• Numeric ranges respected.

A schema failure produces a 400 Bad Request with a clear error pointing at the offending field.

Stage 6 — Optimistic concurrency

The API server compares the request’s resourceVersion to etcd’s ModRevision for the key. Mismatch returns 409 Conflict.

# 409 Conflict
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "Operation cannot be fulfilled on ...",
  "reason": "Conflict",
  "details": {
    "kind": "configmap",
    "name": "..."
  },
  "code": 409
}

The conflict response is normal in concurrent updates; the client should re-fetch and retry.

Stage 7 — Validating admission

Validating admission rejects requests that mutate undesired configurations. The chain of validators:

• PodSecurity admission (privileged/baseline/restricted).
• ValidatingAdmissionPolicies (CEL-based).
• ValidatingWebhookConfiguration.
• ReferenceGrant (Gateway API).
• Built-in validators.

A failure here returns 403 with a reason pointing at the admission policy.

Stage 8 — Encryption at rest

If EncryptionConfiguration is enabled, the API server encrypts the Secret values (or other configured resources) before the etcd write.

flowchart LR
    A[Secret payload] --> B{Encryption provider}
    B -->|kms| C[KMS plugin: request DEK]
    C --> D[DEK in memory]
    B -->|aescbc| D
    D --> E[AES encrypt]
    E --> F[ciphertext]

The encryption stage is sub-ms for AES providers; tens of milliseconds for KMS providers that involve a network round-trip.

Stage 9 — etcd write

The encrypted payload is written to etcd through the gRPC client. The write is a Txn (CAS operation) for “only if resourceVersion matches”:

Txn(
  Compare([ModRevision == expected_revision]),
  Success([Put(key, ciphertext)]),
  Failure([Range(key)])
)

If the comparison fails, the API server retries with the new ModRevision.

The etcd write’s latency is dominated by WAL fsync. A slow disk on the etcd host amplifies API server p99.

Stage 10 — Audit log

The audit log records the request and the authn identity:

{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "RequestResponse",
  "timestamp": "2026-08-16T12:00:00Z",
  "user": {"username": "...", "groups": [...]},
  "objectRef": {"resource": "pods", "namespace": "prod", "name": "..."},
  "verb": "create",
  "responseStatus": {"code": 201}
}

Audit logging does not block the request; if the audit log write fails, the request may still succeed (configurable policy).

Stage 11 — Watch fan-out, response

The API server responds to the client with 200 OK (or 201 Created). Concurrently, the watch event is published to all matching watchers.

The response is sent as soon as etcd commits; watchers receive the event slightly later.

Latency budget at p99

A healthy cluster’s p99 should be ≤ 200 ms; sustained departures suggest a specific stage is slow.

Failure diagnostics

Quiz

Production discipline

• Every stage is a failure point. Production incident response should know each stage’s likely failure modes.
• Webhook latency is API server latency. Every webhook in the chain extends the request’s slowest path.
• Cache authn and authz. The first authn might be slow; subsequent ones use the cache. Set the cache TTL to a few minutes.
• Tune each stage’s timeout. The webhook timeout (default 30 seconds) bounds the worst-case latency for the webhook stage.
• Monitor per stage. Per-stage metrics distinguish a slow admission from a slow etcd.

The request lifecycle is the cluster’s contract for every state change. Operating it well is operating the cluster’s correctness.