Node allocatable, kubelet reservations, and capacity planning

A node’s “capacity” is what the hardware reports. The “allocatable” is what the kubelet tells the scheduler is available for Pods. The difference is kubelet reservations and eviction thresholds. This lesson covers the calculation, the kubelet flags that control it, and the capacity planning discipline.

Capacity vs allocatable

node Capacity:
  cpu: 4
  memory: 8Gi

node Allocatable:
  cpu: 3.6
  memory: 7Gi

The scheduler uses allocatable for placement. A node with 4 CPU capacity has 3.6 CPU allocatable after kubelet reservations.

flowchart LR
    Capacity[Node Capacity] --> Kube[kube-reserved]
    Capacity --> System[system-reserved]
    Capacity --> Evict[eviction-threshold]
    Kube --> Alloc[Allocatable]
    System --> Alloc
    Evict --> Alloc
    Alloc --> Sched[Scheduler uses Allocatable]

The formula:

allocatable = capacity - kube-reserved - system-reserved - eviction-threshold

Each component:

• kube-reserved: CPU and memory reserved for the kubelet, the runtime (containerd), and Kubernetes system processes.
• system-reserved: CPU and memory reserved for the OS and other system services (sshd, systemd, journald, etc.).
• eviction-threshold: the buffer that triggers node pressure eviction. Subtracted from allocatable so the kubelet starts evicting before the node is fully exhausted.

Kubelet reservation flags

# kubelet config
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
systemReserved:
  cpu: 500m
  memory: 1Gi
kubeReserved:
  cpu: 500m
  memory: 1Gi
evictionHard:
  memory.available: 500Mi
nodefs.available: 10%

The kubelet calculates allocatable from these flags:

allocatable_cpu = capacity_cpu - kubeReserved_cpu - systemReserved_cpu
allocatable_memory = capacity_memory - kubeReserved_memory - systemReserved_memory - evictionHard.memory.available

Note: eviction thresholds are only subtracted from allocatable for the relevant resource. CPU has no eviction-threshold subtraction (it’s a soft limit).

Sizing kubelet reservations

The reservation size depends on the node’s role:

General-purpose node (mixed workloads):

systemReserved:
  cpu: 500m
  memory: 1Gi
kubeReserved:
  cpu: 500m
  memory: 1Gi
evictionHard:
  memory.available: 500Mi
  nodefs.available: 10%

For a 4-CPU, 8Gi node: 1.5 CPU and 2.5Gi reserved. Allocatable: 2.5 CPU, 5.5Gi memory.

System node (control plane, monitoring agents):

systemReserved:
  cpu: 1
  memory: 2Gi
kubeReserved:
  cpu: 1
  memory: 2Gi
evictionHard:
  memory.available: 500Mi
  nodefs.available: 10%

Higher reservations because the node runs kube-proxy, CNI agents, log shippers, etc.

GPU node (specialised workloads):

systemReserved:
  cpu: 500m
  memory: 4Gi   # GPU driver overhead
kubeReserved:
  cpu: 500m
  memory: 2Gi

The GPU driver and CUDA libraries consume significant memory.

Eviction thresholds

evictionHard:
  memory.available: 500Mi
  nodefs.available: 10%
  nodefs.inodesFree: 5%
evictionSoft:
  memory.available: 1Gi
  nodefs.available: 15%
evictionSoftGracePeriod:
  memory.available: 30s
  nodefs.available: 30s

Two threshold types:

• evictionHard: when the threshold is hit, the kubelet starts evicting Pods immediately. No grace period.
• evictionSoft: when the threshold is hit, the kubelet starts evicting after the grace period. Allows for transient pressure to subside.

Eviction thresholds are subtracted from allocatable for the specific resource. If evictionHard.memory.available: 500Mi, then allocatable.memory is reduced by 500Mi.

Monitoring allocatable vs requested

kubectl describe node node-3 | grep -A 10 "Allocated resources"

Output:

Allocated resources:
  (Total limits may exceed 100%)
  Resource           Requests      Limits
  --------           --------      ------
  cpu                2100m (58%)   4100m (114%)
  memory             5120Mi (67%)  8192Mi (107%)
  ephemeral-storage  0 (0%)        0 (0%)

The Requests column shows total requests on the node vs allocatable. The percentage is utilisation.

Production discipline:

• Requests utilisation < 80%: healthy; room for new Pods.
• Requests utilisation 80-95%: tight; new Pods may not fit; consider adding capacity.
• Requests utilisation > 100%: the scheduler has placed more requests than allocatable (possible if nodes were added/resized after Pods were scheduled). Investigate.

Capacity planning

The discipline:

1. Sum the requests of all Pods in the cluster.
2. Multiply by the headroom factor (1.2-1.5x for growth, failure tolerance).
3. Divide by the allocatable fraction (typically 70-80% of capacity).
4. Plan node count: (total requests * headroom) / (allocatable per node).

Example:

• Cluster target: 100 CPU, 200Gi memory requested across all Pods.
• Headroom: 1.3x (30% for growth).
• Allocatable per node: 70% of 4 CPU = 2.8 CPU; 70% of 8Gi = 5.6Gi.
• Nodes needed: (100 * 1.3) / 2.8 = ~47 nodes for CPU; (200 * 1.3) / 5.6 = ~47 nodes for memory. CPU and memory align at 47 nodes.

Production patterns

Per-pool capacity planning:

Different node pools have different allocatable:

• Production pool (high-memory nodes, 16 CPU, 64Gi): allocatable ~12 CPU, 50Gi.
• Batch pool (high-CPU nodes, 16 CPU, 16Gi): allocatable ~12 CPU, 12Gi.
• System pool (control plane + add-ons): allocatable varies.

Each pool has its own capacity headroom. Production discipline: track utilisation per pool, not cluster-wide.

HPA + VPA + Cluster Autoscaler for elastic capacity:

# HPA scales horizontally
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: web-hpa
spec:
  scaleTargetRef:
    name: web
  minReplicas: 3
  maxReplicas: 30

# VPA right-sizes requests
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: web-vpa
spec:
  targetRef:
    name: web
  updatePolicy:
    updateMode: Auto

# Cluster Autoscaler scales nodes
# (configured via the cloud provider)

Three layers of autoscaling:

• HPA: more Pods when load increases.
• VPA: bigger Pods when requests are wrong.
• Cluster Autoscaler: more nodes when Pods can’t fit.

The combination keeps capacity right-sized.

Diagnosing capacity issues

Pods stuck Pending due to capacity:

kubectl describe pod web-7c8
# Events:
#   ... reason: FailedScheduling
#   ... message: 0/10 nodes are available: 1 Insufficient cpu, ...

The Pod cannot be scheduled because no node has enough allocatable. The fix: scale up (more nodes) or scale down (reduce requests).

Cluster fragmented:

The cluster has allocatable but no single node has enough for a specific Pod. This happens when:

• A few large Pods consume most of one node.
• The remaining Pods cannot fit anywhere.

The fix: rebalance with Pod topology spread constraints, or add larger nodes.

Cross-course references

• The Linux course part XXXVII-Linux-Resources covers cgroup resource management; allocatable is the cluster-level equivalent.
• The Proxmox course part XXXV-Linux-Storage covers cluster capacity planning; the same discipline applies to Kubernetes.
• The Observability course part LXXXIV-Kubernetes-CapacityPlanning covers capacity planning and forecasting.

Quiz

Production discipline

• Size reservations based on actual overhead. Monitor kubectl describe node for actual usage; size kube-reserved and system-reserved accordingly.
• Monitor allocatable vs requested utilisation. Alert when utilisation exceeds 80% — capacity is tight.
• Plan capacity from requests, not capacity. A node with 4 CPU has ~3 CPU allocatable; use the lower number.
• Use Cluster Autoscaler for elastic capacity. Combine with HPA + VPA for full elasticity.
• Set eviction thresholds with headroom. A 500Mi memory buffer is a small price for early warning before node exhaustion.