Reserve, permit, bind — the cycle's latter phases

Reserve, Permit, and Bind are the three closing phases of the scheduling cycle. Together they take a chosen node (the score output) and commit the Pod to it through the API server. Each phase has its own failure modes. This lesson walks the three phases and the production discipline of each.

The reserve phase

After scoring selects the chosen node, the scheduler reserves resources on that node:

sequenceDiagram
    autonumber
    participant SC as Scheduler
    participant NC as Node cache
    SC->>NC: reserve resources for Pod on node X
    NC->>NC: increment reserved_cpu
    NC->>NC: increment reserved_memory
    Note over NC: state is reserved
    SC->>SC: continue to next phase

The reservation is in the scheduler’s Node cache, not in the API server. It prevents two parallel scheduler threads from racing on the same node’s resources.

The permit phase

The Permit plugins may delay or reject the binding:

flowchart LR
    PR[Pod + chosen node] -->|Permit| Webhook[Permit webhook]
    Webhook -->|Approve| B[Bind]
    Webhook -->|Reject| R[Rejected]
    Webhook -->|Wait| WQ[Waiting queue]
    WQ -->|retry on signal| Webhook

Three outcomes:

• Approve. The binding proceeds.
• Reject. The Pod is marked unschedulable.
• Wait. The Permit plugin holds the binding until a signal or timeout.

A common Wait case is the VolumeBinding plugin’s WaitForFirstConsumer mode: the binding waits for the chosen node’s kubelet to communicate the CSI topology, then the binding writes the bind.

A 30-second timeout on Permit prevents endless waits.

The bind phase

The Bind plugin writes spec.nodeName to the API server:

POST /api/v1/namespaces/production/pods/checkout-api-7d9f6c8b45-r4nq2/binding
{
  "apiVersion": "v1",
  "kind": "Binding",
  "metadata": {"name": "checkout-api-7d9f6c8b45-r4nq2"},
  "target": {"apiVersion": "v1", "kind": "Node", "name": "cp-3"}
}

The API server validates the binding, persists it to etcd, and the kubelet on the chosen node observes spec.nodeName and starts the Pod.

sequenceDiagram
    autonumber
    participant SC as Scheduler
    participant AS as API server
    participant K as kubelet on node
    SC->>AS: POST binding
    AS->>AS: validate (target node exists, RBAC ok)
    AS->>AS: persist to etcd
    AS-->>SC: 200 OK
    AS->>K: watch event: spec.nodeName updated
    K->>K: start Pod

The Bind plugin returns success to the scheduler; the Pod is now considered scheduled.

The race condition

Two scheduler threads might both decide on the same node for different Pods:

sequenceDiagram
    autonumber
    participant T1 as Thread1
    participant T2 as Thread2
    participant NC as NodeCache
    T1->>NC: reserve Pod A on node X
    T2->>NC: reserve Pod B on node X
    NC-->>T1: ok
    NC-->>T2: ok
    T1->>T1: bind Pod A
    T2->>T2: bind Pod B
    Note over NC: Both reserve both bind

The reservation is supposed to prevent this, but the cache may have a brief inconsistency. The Reserve plugin handles the failure by retrying with a different node.

Permit-timeout (WaitForFirstConsumer)

The most common Permit behaviour is the WaitForFirstConsumer mode of volume binding:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data
spec:
  storageClassName: standard
  accessModes: [ReadWriteOnce]
  volumeMode: Filesystem

When the binding reaches Permit, the VolumeBinding plugin signals the storage provisioner to provision the volume on the chosen node. The Permit holds the binding until the volume is provisioned.

1. Pod scheduled
2. Filter+Score chooses node X
3. Permit holds; volume is being provisioned on node X
4. Volume provisioning completes (~10-60 seconds)
5. Permit approves; Bind writes spec.nodeName

The provisioning latency is bounded by the CSI driver’s performance.

Custom Bind plugins

Custom Bind plugins extend what happens at Bind:

func (pl *MyBind) Bind(ctx context.Context, state *framework.CycleState, pod *v1.Pod, nodeName string) *framework.Status {
    // Custom binding logic; e.g., write to external system before the API bind
    return pl.externalSystem.Notify(pod, nodeName)
}

The framework allows the custom plugin to run before or after the default Bind (which writes to the API server).

Failure modes

Bind write fails

Error: timed out waiting for the condition

The API server does not respond to the binding. The scheduler treats this as a transient failure and retries the next cycle.

Permit webhook times out

Permit plugin VolumeBinding: timed out

The Permit plugin’s 30-second timeout fires; the Pod is rejected. The default is a 30-second Permit timeout.

Bind to unschedulable node

The chosen node becomes unschedulable (e.g., cordoned) between score and bind. The default PreBind plugin can detect this and reject the binding.

The cleanup of reservations

After a Bind succeeds, the reservation is released:

sequenceDiagram
    autonumber
    participant SC as Scheduler
    participant NC as Node cache
    SC->>NC: bind succeeded
    NC->>NC: release reservation
    Note over NC: cache is updated

If the Bind fails, the reservation is also released (eventually). The scheduler retries or rejects.

The post-bind hooks

After the Bind, the framework runs PostBind plugins:

1. Bind (writes spec.nodeName)
2. PostBind plugins run (custom hooks)

PostBind plugins are for actions that should happen after a successful bind — e.g., notifying an external system, updating metrics, etc.

The scheduler metrics for these phases

scheduler_pod_scheduling_duration_seconds (overall)
scheduler_permit_wait_duration_seconds (Permit phase)
scheduler_binding_duration_seconds (Bind phase)
scheduler_cache_size (overall cache)

Long Permit or Bind latency is the diagnostic.

$ kubectl get pod web -n prod -o jsonpath='{.spec.nodeName}'

cp-3

The Pod’s spec.nodeName is the bind result.

Quiz

Production discipline

• Reserve is advisory; act on it as such. A reservation is a coordination mechanism within the scheduler; the cluster’s authoritative state is etcd.
• Permit timeouts are operational risks. A slow or failing Permit webhook is a cluster-wide scheduling block.
• Bind latency is scheduler latency. Watch the scheduler_binding_duration_seconds histogram for the API server’s response time on the binding POST.
• The chosen node may change between score and bind. A Pod that was scored on node X may have to be rescheduled if X fails the PreBind check.
• PostBind is for hooks, not for gating. Don’t make PostBind plugins critical; the bind is already done.

The closing phases of the scheduling cycle are what make the decision observable. Operating them well is operating the cluster’s commit point.