Node filesystem layout — /var/lib/kubelet, /var/log, /var/lib/containerd

A worker node’s filesystem is partitioned into a small number of directories that the cluster’s components use heavily. The three hot directories are /var/lib/kubelet (Pod state), /var/lib/containerd (image cache), and /var/log (container logs). All three grow with workload churn. This lesson walks the layout, the failure modes of disk pressure, and the operational patterns for keeping the node healthy.

The directory map

flowchart TD
    A[/var/lib/kubelet/] --> A1[Pod volumes]
    A --> A2[Plugin state]
    A --> A3[Resource manager state]
    B[/var/lib/containerd/] --> B1[Image blobs]
    B --> B2[Snapshotter layers]
    C[/var/log/] --> C1[Pod logs]
    C --> C2[Container logs]
    C --> C3[kube-proxy.log]
    A1 --> D{Disk pressure?}
    B1 --> D
    C1 --> D
    D -->|Yes| E[Eviction loop]
    D -->|No| F[Node healthy]

ls -la /var/lib/kubelet /var/lib/containerd /var/log

/var/lib/kubelet/
  config.json           # kubelet's configuration
  cpu_manager_state     # CPU manager's state
  memory_manager_state  # memory manager's state
  pods/                 # mounted volumes for active Pods
  plugins/              # volume plugins (CSI)
  pod-resources/        # pod resources socket
  device-plugins/       # device plugin sockets

/var/lib/containerd/
  io.containerd.content.v1.content/  # image blobs
  io.containerd.snapshotter.v1.overlayfs/  # image layers
  tmpmounts/             # ephemeral mounts

/var/log/
  pods/                 # symlinks to container logs
  containers/           # container log files
  kube-proxy.log        # kube-proxy service log
  audit.log             # audit log (if configured)

The runtime’s data directory may be different (the default is /var/lib/containerd for containerd, but the kubelet’s --root-dir flag overrides the kubelet’s directory).

The kubelet’s data directory

The kubelet’s --root-dir (default /var/lib/kubelet) holds:

• Pod volumes: each running Pod has a directory containing the mounted volumes (CSI, ConfigMap, Secret, emptyDir). The directory is created when the Pod starts and removed when the Pod is deleted.
• Plugin state: the kubelet’s volume plugins (CSI) store state in subdirectories under plugins/. The state is small (a few KB per volume) but accumulates.
• Resource manager state: the CPU and memory managers store their state in cpu_manager_state and memory_manager_state. The state is small (a few KB) but is rewritten on every Pod start.
• Pod resources: the pod-resources API endpoint listens on a socket under pod-resources/. The kubelet exposes the Pods’ resource assignments to device plugins.

The kubelet’s data directory grows with the number of Pods and the volume sizes. A node running 100 Pods with 1Gi volumes each consumes 100Gi of the directory.

The runtime’s data directory

The container runtime’s data directory (default /var/lib/containerd) holds:

• Image blobs: the actual image content. Each image layer is a separate blob; the same layer is shared across images.
• Image manifests: the image metadata. Sized in KB.
• Snapshotter layers: the overlayfs (or other snapshotter) layers. The runtime unpacks the image into a snapshotter layer for each container.

The runtime’s data directory grows with the number of images and the image sizes. A cluster with 100 distinct images, each 500Mi, consumes 50Gi of the directory.

The runtime’s garbage collection is optional. By default, the runtime does not garbage-collect images; the kubelet does. The kubelet’s image-gc-high-threshold applies.

The log directory

The container logs are written to /var/log/containers (by default) or /var/log/pods (the kubelet’s structured log path). Each container has a log file:

ls /var/log/containers | head -5

billing-1_prod-app_app-1.log
billing-2_prod-app_app-1.log
auth-service_prod-app_main-1.log

The log files are symlinks to the underlying files in /var/log/pods/<pod-id>/<container>/0.log. The kubelet symlinks the file and the systemd kubelet service rotates the log when it reaches --log-file-max-size (default 100Mi).

The log directory grows with the number of Pods and the log volume. A cluster with noisy Pods can fill the log directory in hours. The kubelet’s log rotation is the only protection.

The disk pressure failure mode

Disk pressure is the most common disk-related failure on a node. The kubelet detects it by checking the disk usage against the --disk-pressure-threshold (default 85% of the node’s filesystem). When the threshold is exceeded, the kubelet adds the disk-pressure NoSchedule taint and starts evicting Pods.

# Substitute your own value before running:
NODE=worker-03

kubectl describe node "$NODE" | grep -A 5 "Conditions"

Conditions:
  Type                 Status  Reason
  ----                 ------  ------
  DiskPressure         True    KubeletHasDiskPressure

The cluster’s view of the node is “no new Pods, evict running Pods.” The Pods are evicted gracefully (SIGTERM, then SIGKILL after 30s).

The root causes:

• Image cache. The runtime has too many images. The kubelet’s image GC is the protection.
• Log files. The container logs are not rotated. The kubelet’s log rotation is the protection.
• Orphaned volumes. The kubelet failed to remove a Pod’s volume directory. The kubelet’s volume GC is the protection.
• Ephemeral storage. A Pod is writing to the container’s writable layer. The Pod’s resources.requests.ephemeral-storage is the protection.

The eviction threshold

The kubelet’s eviction threshold is the limit at which it starts evicting Pods. The default is:

• memory.available < 100Mi — start evicting Pods when memory is below 100Mi.
• nodefs.available < 10% — start evicting Pods when the node’s filesystem is below 10% free.
• nodefs.inodesFree < 5% — start evicting Pods when the node’s filesystem is below 5% inodes free.
• imagefs.available < 15% — start evicting Pods when the image filesystem is below 15% free.

The kubelet’s --eviction-hard flag overrides the defaults. The --eviction-soft and --eviction-soft-grace-period flags add a soft threshold with a grace period; the kubelet sends a SIGTERM to the Pod’s containers when the soft threshold is exceeded for the grace period.

The ephemeral storage request

A Pod that writes to the container’s writable layer fills the node’s filesystem. The Pod’s resources.requests.ephemeral-storage declares the expected usage. The kubelet enforces the limit via the cgroup’s io controller (in newer kernel versions) or via the disk-quota controller.

resources:
  requests:
    ephemeral-storage: 1Gi
  limits:
    ephemeral-storage: 2Gi

A Pod that writes more than the limit is killed by the kubelet. The kubelet’s DiskPressure condition is set if the node’s filesystem is below the threshold.

The ephemeral-storage request is the production-grade way to declare expected usage. The kubelet’s monitoring of the node’s filesystem is the backstop.

The operational patterns

Size the node’s filesystem for the workload

A node with 100 Pods and 1Gi volumes each needs 100Gi plus the runtime’s image cache and the log files. The filesystem should be sized for the maximum workload, not the average.

Use a separate filesystem for the image cache

The runtime’s image cache is the largest directory on the node. Mounting /var/lib/containerd on a separate volume (a separate EBS volume, a separate LUN) isolates the image cache from the node’s root filesystem. The kubelet’s --image-gc-high-threshold applies to the image filesystem.

Use a separate filesystem for the logs

The container logs are the second-largest directory. Mounting /var/log on a separate volume isolates the logs from the node’s root filesystem. The kubelet’s log rotation applies.

Set the kubelet’s eviction thresholds

The defaults are conservative. Production clusters often set softer thresholds (e.g., 20% free) to give the operator more time to react. The kubelet’s --eviction-hard and --eviction-soft flags control the thresholds.

Monitor the kubelet’s metrics

The kubelet exposes the disk usage on its metrics endpoint. The relevant metrics:

• node_filesystem_size_bytes
• node_filesystem_avail_bytes
• node_filesystem_files_free
• container_fs_usage_bytes
• container_fs_limit_bytes

The metrics should be scraped by Prometheus; the operator should alert on the available bytes falling below the threshold.

Quiz

Production discipline

• Three hot directories: /var/lib/kubelet, /var/lib/containerd, /var/log. Each grows with workload churn; each needs monitoring.
• Disk pressure is a workload problem, not a node problem. Adding disk does not fix a workload that is writing too much. The fix is to identify the workload and address it.
• Use a separate filesystem for the image cache. The image cache is the largest directory; isolating it prevents the image cache from filling the node’s root filesystem.
• Use a separate filesystem for the logs. The logs are the second-largest directory; isolating them prevents the logs from filling the node’s root filesystem.
• Set the kubelet’s eviction thresholds. The defaults are conservative; production clusters should set them to the operator’s tolerable minimum.
• Audit the filesystem at every node repave. A new node that joins the cluster with a small filesystem is a node that will hit disk pressure early. The bootstrap should size the filesystem to the maximum workload.