Container stdout to logs — the logging pipeline

Container logging in Kubernetes flows from the container stdout/stderr to the kubelet’s log files, then to the log collection. The structured logs are queryable; the unstructured logs are not. This lesson walks the logging pipeline, the log paths, the structured logs, and the collection.

The logging pipeline

The logging pipeline:

flowchart LR
    A["Container stdout"] --> B[Kubelet]
    B --> C["/var/log/pods"]
    C --> D["Promtail/Fluent Bit"]
    D --> E["Loki/ELK"]
    E --> F[Grafana]

The pipeline is the log path.

The container runtime

The container runtime captures the logs:

container runtimes:
  - containerd
  - CRI-O
  - Docker (legacy)

The runtime captures the container’s stdout/stderr to a log file.

The kubelet integration

The kubelet captures the logs:

# The kubelet's log files
ls /var/log/pods/

# Per-namespace, per-pod, per-container
# /var/log/pods/<namespace>_<pod-name>_<pod-uid>/<container>/0.log

The kubelet captures the logs to the host’s filesystem.

The log paths

The log paths:

/var/log/pods/
  default_nginx-7b9f8c5f6_abc123/
    nginx/
      0.log
  kube-system_kube-apiserver-cp-1_def456/
    kube-apiserver/
      0.log

The path includes the namespace, the pod name, the pod UID, the container, and the log file.

The kubectl logs

The kubectl logs:

kubectl logs nginx-1-abc
kubectl logs nginx-1-abc -c nginx
kubectl logs nginx-1-abc --previous
kubectl logs nginx-1-abc --tail=100
kubectl logs nginx-1-abc --since=1h
kubectl logs -f nginx-1-abc

The kubectl logs queries the kubelet’s log files.

The structured logs

The structured logs:

{
  "timestamp": "2026-08-16T10:00:00.000Z",
  "level": "INFO",
  "message": "Request processed",
  "service": "nginx",
  "request_id": "abc123",
  "duration_ms": 100
}

The structured logs are JSON; the fields are queryable.

The JSON formatter

The JSON formatter:

log.SetFormatter(&log.JSONFormatter{
  FieldMap: log.FieldMap{
    log.FieldKeyTime: "@timestamp",
    log.FieldKeyLevel: "level",
    log.FieldKeyMsg: "message",
  },
})

The Go log library uses the JSON formatter.

import logging
import json

class JSONFormatter(logging.Formatter):
    def format(self, record):
        return json.dumps({
            "timestamp": self.formatTime(record),
            "level": record.levelname,
            "message": record.getMessage(),
            "module": record.module,
        })

The Python log library uses the JSON formatter.

The collection

The collection (Promtail):

# Promtail config
server:
  http_listen_port: 9080

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
- job_name: kubernetes-pods
  kubernetes_sd_configs:
    - role: pod
  relabel_configs:
    - source_labels: [__meta_kubernetes_pod_label_app]
      target_label: app
    - source_labels: [__meta_kubernetes_pod_phase]
      regex: (Failed|Pending|Unknown)
      action: drop

The Promtail ships the logs to Loki.

The Loki integration

The Loki integration:

sequenceDiagram
    participant C as Container
    participant K as Kubelet
    participant P as Promtail
    participant L as Loki
    C->>K: stdout/stderr
    K->>K: write to /var/log/pods
    P->>K: tail /var/log/pods
    P->>L: push logs
    L->>L: store in chunks
    L-->>Grafana: query

The Loki integration is the log path.

The observability

The observability:

flowchart LR
    A[Container logs] --> B[Promtail]
    B --> C[Loki]
    C --> D[Grafana]
    D --> E[Cluster dashboard]
    D --> F[Workload dashboard]
    D --> G[Service dashboard]

The observability is the Grafana dashboard.

The production patterns

The production patterns:

• Structured logs: JSON with relevant fields.
• Log rotation: the runtime rotates the logs.
• Log retention: the cluster’s log retention.
• Log-based alerts: the Loki alerts.

The cross-course references

The Observability course covers the signals in detail.

• The Loki course covers the logs storage.
• The Grafana course covers the dashboards.
• The Events course (Part XCI) covers the Kubernetes events.

Quiz

Production discipline

• Use stdout/stderr for container logs. The canonical path.
• Use structured logs (JSON). Queryable in Loki.
• Deploy a log collector. Promtail, Fluent Bit.
• Use a log storage. Loki, ELK.
• Configure the LogQL queries. The log queries.
• Configure the log-based alerts. The LogQL alerts.

The container logging pipeline is the cluster’s logs. Operating it well is via stdout/stderr, with structured logs, and the collector and storage.