Node log paths — the files on the host

The node log paths are the files on the host. The container logs, the kubelet logs, the container runtime logs, and the system logs. The log rotation is the kubelet’s responsibility. This lesson walks the log paths, the inspection, the kubelet logs, and the log rotation.

The log paths

The log paths:

/var/log/pods/                 # container logs (per-namespace, per-pod, per-container)
/var/log/containers/           # symlinks to /var/log/pods/
/var/log/kubelet.log           # kubelet logs (if not journald)
/var/log/syslog                # system logs (Debian/Ubuntu)
/var/log/messages              # system logs (RHEL/Fedora)
/var/log/<runtime>/            # container runtime logs

The log paths are the host’s filesystem.

The container logs

The container logs:

# List the pod logs
ls /var/log/pods/

# The path is namespace_pod-name_pod-uid/container/0.log
ls /var/log/pods/default_nginx-7b9f8c5f6_abc123/nginx/

# The log file
cat /var/log/pods/default_nginx-7b9f8c5f6_abc123/nginx/0.log

The container logs are in /var/log/pods.

The kubelet logs

The kubelet logs:

# Via journald (systemd)
journalctl -u kubelet -n 100

# Via the log file
tail -f /var/log/kubelet.log

The kubelet logs are the kubelet’s diagnostics.

The runtime logs

The container runtime logs:

# containerd
journalctl -u containerd -n 100

# CRI-O
journalctl -u crio -n 100

# Kubernetes version detection
kubectl get nodes -o wide

The runtime logs are the container runtime’s diagnostics.

The system logs

The system logs:

# Debian/Ubuntu
ls /var/log/syslog

# RHEL/Fedora
ls /var/log/messages

# The journald
journalctl -n 100

The system logs are the host’s diagnostics.

The log rotation

The log rotation:

# kubelet config
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
containerLogMaxSize: 10Mi
containerLogMaxFiles: 5

The kubelet rotates the logs based on the max size and max files.

The log rotation strategy

The log rotation strategy:

flowchart LR
    A[Current log: 0.log] --> B[Size > 10Mi]
    B --> C[Rotate: 0.log -> 4.log]
    C --> D[New 0.log created]
    E{More than 5 files?}
    F --> E
    E -->|yes| G[Delete oldest]
    E -->|no| H[Keep]

The rotation is the kubelet’s responsibility.

The kubectl logs

The kubectl logs queries the kubelet:

kubectl logs nginx-1-abc
kubectl logs nginx-1-abc -c nginx
kubectl logs nginx-1-abc --previous
kubectl logs nginx-1-abc --tail=100 --since=1h

The kubectl logs uses the kubelet’s log API.

The kubelet’s log API

The kubelet’s log API:

GET /containerLogs/<namespace>/<pod>/<container>

The kubelet exposes the log files via the API. The kubectl logs queries the API.

The log collection

The log collection:

# Promtail
promtail -config.file=promtail.yaml

# Fluent Bit
fluent-bit -c fluent-bit.conf

# Filebeat
filebeat -e -c filebeat.yaml

The collection tails the log files and ships them to the storage.

The log inspection

The log inspection:

# Substitute your own node address before running:
NODE_IP=192.0.2.11

# On the node
sudo tail -f /var/log/pods/default_nginx-*/nginx/0.log

# Via kubectl
kubectl logs nginx-1-abc -f

# Via the kubelet API
curl -k "https://$NODE_IP:10250/containerLogs/default/nginx-1-abc/nginx"

The inspection is per pod, per container.

The log rotation and the collection

The log rotation and the collection:

sequenceDiagram
    participant C as Container
    participant R as Runtime
    participant K as Kubelet
    participant L as Log file
    participant P as Promtail
    C->>R: stdout/stderr
    R->>K: log line
    K->>L: write
    L->>P: tail
    P->>L: ship to Loki
    K->>L: rotate when full

The collection must happen before the rotation.

The cross-course references

• The Loki course covers the log storage.
• The Grafana course covers the dashboards.
• The systemd course covers the journald.

Quiz

Production discipline

• Know the log paths. /var/log/pods, /var/log/containers, /var/log/kubelet.log.
• Use the kubelet’s log rotation. 10Mi max size, 5 max files.
• Use the log collection. Promtail or Fluent Bit.
• Use journalctl for the kubelet logs. The systemd integration.
• Document the log paths. The per-node inspection.
• Test the log rotation. Verify the files are rotated.

The node log paths are the host’s logs. Operating it well is knowing the paths, configuring the rotation, and shipping the logs to the storage.