kubelet garbage collection — images, containers, volumes

The kubelet runs three garbage collection subsystems: image GC (the cache of images), container GC (the dead containers), and the volume cleanup (the orphaned volume directories). Each has a threshold; each has a failure mode. A kubelet that is not garbage collecting is a node that is filling its disk. This lesson walks the three subsystems, the thresholds, and the operational patterns.

The image GC

flowchart LR
    A[kubelet housekeeping] --> B{Image cache<br/>>85% full?}
    B -->|Yes| C[ListImages]
    C --> D[Sort by last-used]
    D --> E[RemoveImage oldest]
    E --> F{<80% full?}
    F -->|No| E
    F -->|Yes| G[Done]
    B -->|No| G

The image GC subsystem removes images from the runtime’s cache when the disk usage exceeds the high threshold. The kubelet removes the least-recently-used images until the usage is below the low threshold.

# kubelet flags
--image-gc-high-threshold=85
--image-gc-low-threshold=80

The kubelet checks the disk usage every 5 minutes (the --image-gc-interval flag). If the usage is above the high threshold, the kubelet removes images until the usage is below the low threshold.

The kubelet’s image GC is the only protection against a runtime cache that grows without bound. A kubelet that is not image GC-ing is a node that fills its disk.

The runtime’s cache is the largest directory on the node. A cluster with 100 distinct images, each 500Mi, consumes 50Gi of the cache. The image GC keeps the cache below the threshold.

The container GC

The container GC subsystem removes dead containers. A dead container is a container that has exited and is no longer running. The kubelet keeps the container’s log file and the container’s metadata for a configurable period.

# kubelet flags
--containers-per-max-pod-count=1
--maximum-dead-containers-per-container=2

The kubelet keeps at most maximum-dead-containers-per-container dead containers per Pod. The kubelet removes the oldest dead containers when the limit is exceeded.

The container GC is independent of the image GC. The container GC is about the containers that have exited; the image GC is about the images that are cached.

The container GC’s threshold is a count, not a size. A Pod with 100 restarts and a maximum-dead-containers-per-container of 2 keeps 2 dead containers; the next 98 are removed. The container’s log files are kept; the container’s filesystem is removed.

The volume cleanup

The volume cleanup subsystem removes orphaned volume directories. An orphaned volume directory is a directory under /var/lib/kubelet/pods/<pod-id>/volumes that is not associated with an active Pod.

The kubelet’s volume cleanup is not configurable in modern versions. The Container Storage Interface (CSI) driver is responsible for cleaning up the volume’s state. The kubelet’s role is to remove the directory after the CSI driver has cleaned up.

The volume cleanup is the most common source of orphaned data. A CSI driver that fails to clean up the volume leaves the directory behind. The kubelet’s metrics expose the orphaned directory count.

The container log rotation

The kubelet rotates the container’s log files. The log file is concatenated to a new file when the size exceeds the --log-file-max-size (default 100Mi). The kubelet keeps at most --log-file-max-count (default 5) log files; the older files are removed.

# kubelet flags
--log-file-max-size=100Mi
--log-file-max-count=5

The rotation is per-container. A Pod with 100 containers has 100 log files. The maximum disk usage is 100 * 100Mi * 5 = 50Gi.

The log rotation is the only protection against a noisy container that fills the disk. A Pod that writes to stdout at 1Mi/s will fill the log file in 100s; the kubelet rotates the file.

The GC metrics

The kubelet exposes the GC metrics on the metrics endpoint:

# Substitute your own value before running:
NODE_IP=192.0.2.21   # the node whose kubelet you are querying

curl -k "https://$NODE_IP:10250/metrics" | grep kubelet_ | grep gc

The relevant metrics:

• kubelet_containers_per_max_pod_count
• kubelet_maximum_dead_containers_per_container
• kubelet_image_gc_high_threshold
• kubelet_image_gc_low_threshold
• kubelet_orphaned_pods_count
• kubelet_volume_dir_count
• kubelet_container_log_filesystem_usage_bytes

The metrics should be scraped by Prometheus. The operator should alert on:

• kubelet_orphaned_pods_count > 0 — orphaned volume directories.
• kubelet_volume_dir_count > 1000 — too many volume directories.
• kubelet_container_log_filesystem_usage_bytes > 80% — log directory is filling.

The GC failure modes

The kubelet’s GC failure modes:

The diagnostic:

df -h /var/lib/kubelet /var/lib/containerd /var/log

A disk that is filling is a kubelet that is not GC-ing. The fix is to investigate the kubelet’s flags and the CSI driver.

The GC operational patterns

Set the image GC thresholds

The defaults are 85% high and 80% low. A production cluster may want to lower the high threshold to 75% to give the operator more time to react. The cost is more frequent image pulls; the benefit is more disk free.

Set the container GC limits

The default is 2 dead containers per Pod. A production cluster may want to lower the limit to 1 to reduce the disk usage; the cost is fewer historical containers for debugging.

Set the log file size

The default is 100Mi per container. A noisy container may need a smaller size to prevent the disk from filling. The production rule is to set the size based on the container’s expected log volume.

Monitor the GC metrics

The kubelet’s metrics expose the GC state. The operator should monitor:

• The disk usage on the node’s filesystem.
• The image cache size.
• The orphan pod count.
• The volume directory count.

The metrics should be scraped by Prometheus; the operator should alert on the thresholds.

Quiz

Production discipline

• GC is the kubelet’s responsibility. The kubelet runs the GC; the operator configures the thresholds. A kubelet that is misconfigured is a node that fills its disk.
• Three GC subsystems: image GC, container GC, and volume cleanup. Each has a failure mode; each has a metric.
• Set the thresholds deliberately. A cluster that uses the defaults is a cluster that lives within the defaults. A production cluster may want to lower the thresholds to give the operator more time to react.
• Monitor the GC metrics. The disk usage is the primary signal. The operator should alert on the threshold.
• Audit the GC at every node repave. A new node that joins the cluster with the wrong GC configuration is a node that fills its disk. Validate the kubelet’s flags at bootstrap.