Capacity sizing — control plane and worker node sizing

Capacity sizing is the first decision in building a production cluster. Right-sizing balances cost (over-provisioned nodes) against safety (under-provisioned nodes that thrash). This lesson walks the minimums, the production targets, and the discipline of capacity planning.

The minimums vs production targets

flowchart LR
    M[Minimums: 2 vCPU + 4 GB] --> S[Small clusters]
    S --> P[Production: 4+ vCPU + 16+ GB]
    P --> L[Large: 8+ vCPU + 32+ GB]

The minimums are survival thresholds; the production targets are tuning to cluster scale.

Control plane sizing

The control plane runs:

• kube-apiserver (static pod).
• kube-controller-manager (static pod).
• kube-scheduler (static pod).
• etcd member (static pod).

Each consumes:

• kube-apiserver: ~500 MB idle, 1-2 GB under load (proportional to API request rate and cache size).
• kube-controller-manager: ~200 MB idle, 500 MB under load.
• kube-scheduler: ~200 MB idle, 500 MB under load.
• etcd: ~500 MB-1 GB (proportional to DB size and watch fan-out).

Total control plane baseline: ~1.5-2 GB
With cache + watch fan-out: ~4-8 GB
Production target: 16 GB+ (room for growth)

A 4 vCPU + 16 GB control plane is the standard production size for ≤500 nodes.

$ free -h | head

...

etcd storage requirements

The etcd data directory is sized for:

• Object count. A cluster with 100,000 objects stores ~100k * ~1 KB = 100 MB. A cluster with 1M objects is ~1 GB.
• Revision history. Each revision takes ~100 bytes. With 1B revisions = ~100 MB.
• Watch cache. In-memory mirroring of state; ~80% of the bbolt file size.

A typical production etcd data dir grows 1-5 GB per year.

Worker sizing

Workers run kubelet + containers. The size depends on workload:

General purpose

4 vCPU
8-16 GB memory
~20 GB local disk (for images, container logs)

Suitable for web servers, microservices.

Memory-intensive

8 vCPU
32-64 GB memory
~50 GB local disk

Suitable for databases, in-memory caches.

CPU-intensive

16 vCPU
8-16 GB memory
~50 GB local disk

Suitable for batch processing, ML inference.

GPU / accelerated

Specialised hardware
GPU drivers + runtime
Cloud-specific sizes (e.g., AWS p3, GCP a2)

The cloud LB + specialised infrastructure.

The capacity headroom

The cluster runs at:

Allocatable = Total - System reserved - Reserved (kube-system)
Used = sum of Pod resource requests
Headroom = Allocatable - Used

Production rules:

• Average utilisation < 70%. Headroom for spikes.
• Peak utilisation < 85%. Capacity for unexpected bursts.
• Headroom for node failure. A 100-node cluster needs ≥1/100 (or 1) node’s worth of headroom for redeploying failed nodes’ workloads.
• Headroom for growth. 30-50% growth before the next cluster resize.

The number of worker nodes

Cluster need: 1000 vCPU, 2 TB memory
Per worker: 8 vCPU, 64 GB
Workers needed: 1000 / 8 = 125 vCPU workers; 2 TB / 64 = 32 memory workers
Bottleneck: 125 workers (for vCPU)
With headroom: ~170 workers

Decision: 50-100 workers for adequate headroom

The aggregate capacity

The cluster’s first-deploy sizing

A first-time setup rarely needs the full production capacity. The right approach:

1. Estimate worst-case size. What’s the upper bound for this cluster?
2. Provision at ~70% of worst-case. Leave room for capacity planning errors.
3. Validate under load. A cluster at 50% utilised has more headroom than a cluster at 95%.
4. Scale up gradually. Add capacity as workload grows. Don’t over-provision from day one.

The sizing by cluster purpose

The sizing for new applications

When adding a new application to the cluster:

1. Estimate the application’s resource requests.
2. Compare to existing headroom.
3. Add nodes if needed before deploying.
4. Schedule with priority / preemption if appropriate.

A new application that consumes the cluster’s headroom causes pre-existing workloads to scale poorly.

The “memory-only” Pods

Some workloads run memory-heavy without needing CPU. A memory-optimised worker profile:

8 vCPU (modest)
128 GB memory
~50 GB disk

A cluster with these as a separate pool serves memory workloads efficiently.

The “general” profile tuning

A general-purpose worker’s CPU/memory ratio:

1 vCPU : 4 GB memory (modest)
1 vCPU : 8 GB memory (high)

The ratio depends on the workload mix. 1:8 (memory-rich) is becoming the standard for mixed workloads.

The disk sizing

Each worker’s disk holds:

• Container images: 5-10 GB each, cached.
• Container logs: rotation policy dependent.
• Ephemeral storage: Pod-level.
• kubelet data: secrets, projected volumes.

A 100 GB root disk is the typical minimum. SSD is strongly preferred for kubelet data.

The discipline

• Document the sizing rationale. The runbook should state: “4 vCPU + 16 GB control plane; 8 vCPU + 32 GB\n general workers; sized for X peak load at Y% utilisation”.
• Reserve capacity for growth. Don’t run at 95% utilisation.
• Plan for failure. A cluster must tolerate a 1/N node loss with 20% headroom for redeployment.
• Re-evaluate quarterly. Workloads change; the cluster’s sizing should be revisited.
• Test the worst-case sizing. A deployment that works at 70% utilisation may behave differently at 90%.

Quiz

Production discipline

• Document the sizing rationale. The runbook should state the cluster’s hardware, utilisation targets, and growth margin.
• Re-evaluate quarterly. Workloads change; sizing should follow.
• Reserve headroom. 50-70% peak utilisation; 20-30% for growth; 1 node for failure.
• Specialise pools. Different worker profiles for different workload types.
• Test at projected peak. A 70% utilisation cluster that fails at 80% is mis-sized.

Sizing is the foundation. Operating it well is matching the cluster to the workload with deliberate headroom.