OS and kernel tuning — sysctl, transparent huge pages, disk schedulers

Every production cluster runs on a Linux host. The host’s kernel and OS configuration directly affect Kubernetes stability: networking, disk I/O, memory management. This lesson walks the production tuning that makes a cluster reliable.

The kernel modules

A Kubernetes node requires:

# Networking
br_netfilter
overlay

# Storage
loop

The modules are loaded at boot via /etc/modules-load.d/:

cat > /etc/modules-load.d/k8s.conf <<EOF
br_netfilter
loop
overlay
EOF

For some CNIs (e.g., Calico), additional modules are required:

cat >> /etc/modules-load.d/k8s.conf <<EOF
ip_tables
ip_set
ip_set_hash
ip_set_hash_ip
xt_hashlimit
EOF

The sysctl parameters

The sysctl parameters to set on every node:

cat > /etc/sysctl.d/99-kubernetes-k8s.conf <<EOF
# IPv4 forwarding (mandatory for routing in CNI)
net.ipv4.ip_forward = 1

# Bridge netfilter (required for kube-proxy iptables)
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1

# Connection tracking for large workloads
net.netfilter.nf_conntrack_max = 1000000

# File descriptors
fs.file-max = 1000000
fs.nr_open = 1000000

# Process limits
kernel.pid_max = 4194304

# ARP cache size (dense cluster)
net.ipv4.neigh.default.gc_thresh1 = 4096
net.ipv4.neigh.default.gc_thresh2 = 8192
net.ipv4.neigh.default.gc_thresh3 = 16384

# TCP tuning for high connection rates
net.core.somaxconn = 32768
net.ipv4.tcp_max_syn_backlog = 32768
EOF

sudo sysctl --system

The parameters:

• net.ipv4.ip_forward=1 enables IPv4 routing (mandatory for CNI gateway).
• net.bridge.bridge-nf-call-iptables=1 ensures bridge traffic passes through iptables rules.
• net.netfilter.nf_conntrack_max bounds the connection tracking table size.

$ sudo sysctl net.ipv4.ip_forward net.bridge.bridge-nf-call-iptables

net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1

The transparent huge pages setting

Transparent Huge Pages (THP) is a memory optimization that can cause latency spikes in large workloads:

# Disable THP on a running system
echo madvise | sudo tee /sys/kernel/mm/transparent_hugepage/enabled

For kubelet etcd, the recommendation is THP off:

GRUB_CMDLINE_LINUX="transparent_hugepage=never"

Disable at boot:

sudo grubby --update-kernel=ALL --args="transparent_hugepage=never"
sudo reboot

The swap setting

Modern kubelet versions support limited swap:

# Disable swap (kubeadm default)
sudo swapoff -a
sed -i '/\sswap\s/ s/^/#/' /etc/fstab

For swap-permitted clusters (kubelet with --feature-gates=NodeSwap=true):

# Allow swap (with kubelet swap behavior)
# kubelet can run with swap enabled if configured

Production: disable swap unless the runtime specifically supports it.

The disk schedulers

The Linux kernel’s disk scheduler affects I/O latency:

For Kubernetes control-plane hosts:

# Use mq-deadline for SSD
echo mq-deadline | sudo tee /sys/block/sda/queue/scheduler

# For NVMe, none is appropriate (default)

Tune the queue depth:

echo 256 | sudo tee /sys/block/nvme0n1/queue/nr_requests

The ulimit settings

Process limits:

cat > /etc/security/limits.d/99-kubernetes.conf <<EOF
* soft nofile 1000000
* hard nofile 1000000
* soft nproc unlimited
* hard nproc unlimited
EOF

These apply to user sessions; systemd may need separate limits.

The tuned profile

The tuned daemon applies profile-based tuning:

sudo dnf install -y tuned
sudo systemctl enable --now tuned
sudo tuned-adm profile virtual-guest
# Or for high-throughput:
sudo tuned-adm profile throughput-performance

The available profiles:

The kubelet ulimit

The kubelet’s open file limit can be set via systemd:

cat > /etc/systemd/system/kubelet.service.d/99-k8s.conf <<EOF
[Service]
LimitNOFILE=1000000
LimitNPROC=infinity
EOF

sudo systemctl daemon-reload
sudo systemctl restart kubelet

The kernel command line

Some parameters are set at boot:

sudo grubby --update-kernel=ALL --args="transparent_hugepage=never cgroup_no_v1=all"

These take effect at next boot.

The log rotation

Container logs grow rapidly:

# /etc/logrotate.d/docker-containers
/var/log/containers/*.log {
    daily
    rotate 5
    missingok
    notifempty
    compress
    copytruncate
}

The kubelet’s log rotation may run via --container-log-max-files and --container-log-max-size.

The clock and time sync

Time sync is critical for certificate validity, log timestamps, and metric accuracy:

sudo dnf install -y chrony
sudo systemctl enable --now chronyd
sudo chronyc tracking

The NTP source is typically the cluster’s internal NTP server.

The firewall rules

Each node’s firewall must allow:

• Kubelet to API server: outbound TCP 6443.
• API server to kubelet: inbound TCP 10250.
• NodePort services: inbound TCP/UDP 30000-32767.
• Pod network: as required by CNI.
• Etcd peer: TCP 2380 (control plane only).

Cloud security groups often handle this; on-prem firewalls must be configured.

The SELinux / AppArmor status

Security profile enforcement:

# CentOS / RHEL
sudo setenforce 1
sudo sed -i 's/^SELINUX=.*/SELINUX=enforcing/' /etc/selinux/config

# Ubuntu AppArmor
sudo aa-enforce /etc/apparmor.d/*

The kubelet adjusts for SELinux / AppArmor modes; permissive mode is sometimes useful for debugging.

The summary checklist

KUBERNETES NODE OS TUNING CHECKLIST
====================================

Kernel modules:
  [ ] br_netfilter loaded
  [ ] overlay loaded
  [ ] CNI-specific modules loaded

Sysctl:
  [ ] net.ipv4.ip_forward = 1
  [ ] net.bridge.bridge-nf-call-iptables = 1
  [ ] net.netfilter.nf_conntrack_max = 1000000

Memory:
  [ ] THP off (deferred, then never)
  [ ] Swap off

Disk:
  [ ] Scheduler appropriate (mq-deadline / none)
  [ ] Queue depth tuned

Limits:
  [ ] ulimit nofile 1000000
  [ ] kubelet systemd override

Time:
  [ ] chrony / NTP

Firewall:
  [ ] Kubelet, API server, etcd, NodePort, Pod network

The discipline

• Apply the tuning before kubelet install. A node with bad sysctl config will fail to register.
• Test in staging. Validate the tuning before production.
• Use configuration management. Ansible or similar ensures consistency across hosts.
• Document every setting. A runbook entry on host tuning is essential.
• Re-evaluate with the kernel updates. New kernels may change recommendations.

Quiz

Production discipline

• Tuning before kubelet. Order matters.
• Configuration management. Ansible / cloud-init ensures consistency.
• Validate post-tune. A failed validation surfaces immediately.
• Document the tuning rationale. The runbook should explain why each parameter.
• Re-evaluate with kernel updates. New kernels may change recommendations.

OS tuning is the foundation. Operating it well is the cluster’s stability floor.