Lab 23: Ship container logs off the node

Objective

By the end of this lab you will have followed one log line from a container’s stdout to a file on a node, watched the kubelet delete that file while the workload was still running, and built the pipeline that would have kept a copy.

The thing worth taking away is smaller and more useful than “install a logging stack”. It is that kubectl logs is not a log system. It is a read of a file on one node, with a size cap, a file-count cap, and a lifetime tied to the Pod object. Every one of those three limits is reachable in a normal week, and each one loses data silently — no error, no event, no gap marker, just a line that is no longer there when somebody asks for it.

You will finish by taking the sink down on purpose and measuring exactly how many lines you lost, because “we have centralised logging” is a claim that deserves a number.

Architecture

There is no log agent inside the container and no log API in Kubernetes. The entire pipeline is a file, a symlink, and a process that reads it.

flowchart TB
    APP["container process<br/>writes to stdout"] --> CRI["containerd<br/>adds timestamp, stream, tag"]
    CRI --> FILE["/var/log/pods/NAMESPACE_POD_UID/CONTAINER/0.log"]
    FILE --> LINK["/var/log/containers/*.log<br/>symlink"]
    FILE --> KLOG["kubelet log API<br/>serves kubectl logs"]
    LINK --> FB["Fluent Bit DaemonSet<br/>tail + position DB"]
    FB --> ENRICH["kubernetes filter<br/>asks the API server whose Pod this is"]
    ENRICH --> LOKI["Loki<br/>labels + chunks"]
    KUBELET["kubelet rotation<br/>containerLogMaxSize / MaxFiles"] -.->|deletes| FILE
    GC["Pod deletion + kubelet GC"] -.->|deletes the directory| FILE

The two dotted arrows are the whole problem. Both of them remove data that kubectl logs was serving a moment earlier, neither of them tells anybody, and the collector is in a race with both.

Requirements

A disposable kubeadm cluster: one control-plane node and two workers, Kubernetes 1.34.x, containerd as the CRI. The cluster built in Lab 01 is exactly right.
kubectl 1.34.x, jq and curl on your workstation.
SSH to at least one worker node, with sudo. Tasks 2 and 3 read files under /var/log that no API exposes.
At least 1 GiB of free memory on one worker for Loki, and roughly 1 GiB of free disk on every worker. Task 1 deliberately makes one container emit 200,000 log lines; the kubelet’s default caps hold at most five files of 10 MiB per container, so the space it occupies at any moment is bounded even though the total written is larger.
Cluster-admin, because the collector needs a ClusterRole and the logging namespace needs a Pod Security exception.
The manifests below pin busybox:1.36, grafana/loki:3.5 and fluent/fluent-bit:3.2. Confirm all three tags resolve before you start, and record the versions you actually got — a logging pipeline is mostly configuration, and configuration keys move between major versions.

Scenario

An application fails at 02:14 with a stack trace. The on-call engineer is paged at 02:40, opens a terminal at 02:45, and runs kubectl logs.

Three things have happened in those thirty-one minutes. The container restarted, so the interesting output is behind --previous. The restart loop produced enough output to roll the log file over twice. And at 02:52 a cluster autoscaler replaced the node, which deleted the Pod object, which means --previous now returns nothing at all.

Nothing was broken. No component misbehaved. Every one of those three behaviours is documented, default, and correct. The logs are simply gone, and the incident review will record that the root cause could not be established.

Your job is to reproduce each of the three losses on purpose, so that you can say what your cluster’s real retention is, and then to build the pipeline that changes the answer.

Tasks

Task 1: Deploy something that logs in a countable way

Most logging labs use a workload that emits English. Use one that emits sequence numbers instead: the point of everything below is to count what arrived, and you cannot count prose.

chatty.yaml:

apiVersion: v1
kind: Namespace
metadata:
  name: logging-lab
  labels:
    # This workload has no reason to fail restricted, so hold it there.
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: chatty
  namespace: logging-lab
spec:
  replicas: 1
  selector:
    matchLabels:
      app: chatty
  template:
    metadata:
      labels:
        app: chatty
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: chatty
          image: busybox:1.36
          command: ["/bin/sh", "-c"]
          args:
            - |
              # A bounded burst: enough to force rotation, not enough to
              # fill the node. Each line carries its own sequence number so
              # that a missing line is countable rather than a feeling.
              i=1
              budget=200000
              pad="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
              while [ "$i" -le "$budget" ]; do
                echo "seq=$i level=info msg=synthetic-log-line pad=$pad"
                i=$((i + 1))
              done
              echo "seq=BURST-COMPLETE budget=$budget"
              # Then a slow heartbeat, so there is always something recent.
              while true; do
                echo "seq=heartbeat ts=$(date -u +%Y-%m-%dT%H:%M:%SZ)"
                sleep 15
              done
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
            readOnlyRootFilesystem: true
          resources:
            requests:
              cpu: 50m
              memory: 16Mi
            limits:
              memory: 64Mi

WORKDIR="$HOME/k8s-logging-lab"
mkdir -p "$WORKDIR"
cd "$WORKDIR"

kubectl apply -f chatty.yaml
kubectl rollout status deployment/chatty -n logging-lab --timeout=120s

POD=$(kubectl get pod -n logging-lab -l app=chatty -o jsonpath='{.items[0].metadata.name}')
NODE=$(kubectl get pod -n logging-lab "$POD" -o jsonpath='{.spec.nodeName}')
echo "POD=$POD NODE=$NODE" | tee pod-and-node.txt

Write those two values down. Every command in the next two tasks needs them, and the Pod name changes the moment anything restarts.

Task 2: Find the file, and read what is actually in it

Wait until the burst has finished — kubectl logs -n logging-lab "$POD" | tail -1 shows the BURST-COMPLETE line or a heartbeat — then go and look at the storage.

cd "$HOME/k8s-logging-lab"

# Substitute the address of the node named in pod-and-node.txt:
NODE_ADDR=192.0.2.12

ssh "$NODE_ADDR" 'sudo ls -l /var/log/containers/ | grep chatty' | tee log-path-trace.txt

Read-only / Safeworker— Everything in /var/log/containers is a symlink; the name encodes pod, namespace and container

$ sudo ls -l /var/log/containers/ | grep chatty

lrwxrwxrwx 1 root root 96 Aug 19 09:12 chatty-7d4b8f9c6-lm2xq_logging-lab_chatty-2f1a....log -> /var/log/pods/logging-lab_chatty-7d4b8f9c6-lm2xq_9c3f.../chatty/0.log

Illustrative output

Two directories, one file. /var/log/containers holds symlinks whose filenames encode pod, namespace and container, which is why every log collector on earth globs that directory rather than the real one — the metadata is in the name. /var/log/pods holds the actual file, in a directory named for the namespace, the Pod and the Pod’s UID.

Now read a raw line. This is the part people are surprised by:

cd "$HOME/k8s-logging-lab"

# Substitute the address of the node named in pod-and-node.txt:
NODE_ADDR=192.0.2.12

# Resolve the symlink to the file it actually points at, then read one line
# of that file. Two steps, so you can see both halves.
LOGFILE=$(ssh "$NODE_ADDR" 'sudo readlink -f "$(sudo ls -d /var/log/containers/chatty*.log | head -1)"')
echo "$LOGFILE" | tee -a log-path-trace.txt

ssh "$NODE_ADDR" "sudo head -1 '$LOGFILE'" | tee -a log-path-trace.txt

Read-only / Safeworker— The CRI log format: timestamp, stream, partial/full tag, then the application line

$ sudo head -1 /var/log/pods/logging-lab_chatty-7d4b8f9c6-lm2xq_9c3f.../chatty/0.log

2026-08-19T09:12:41.118273941Z stdout F seq=1 level=info msg=synthetic-log-line pad=xxxxxxxx...

Illustrative output

The application wrote seq=1 level=info .... What is on disk has three fields in front of it: an RFC 3339 timestamp with nanoseconds, the stream name (stdout or stderr), and a tag that is F for a full line or P for a partial one. That is the CRI logging format, written by containerd, and it is why a collector needs a parser: without one, every field you care about is buried inside a string that also contains a timestamp you already have.

The P tag matters more than it looks. A line longer than the runtime’s buffer — commonly 16 KB — is split, and the pieces are tagged P until the last, which is tagged F. A stack trace or a large JSON document arrives as several records unless the collector reassembles them, which is the single most common reason a logging pipeline shows truncated errors.

Task 3: Measure how far back your cluster can actually look

The burst wrote 200,000 lines, and on disk each one is larger than what the application emitted because the CRI prefix is added to every line. Before looking at the result, find out what the kubelet’s caps on this node really are — do not assume the documented defaults, because a node built by somebody else may not have them.

The kubelet serves its own effective configuration, and you can read it through the API server without touching the node:

cd "$HOME/k8s-logging-lab"

NODE=$(kubectl get pod -n logging-lab -l app=chatty \
  -o jsonpath='{.items[0].spec.nodeName}')

kubectl get --raw "/api/v1/nodes/$NODE/proxy/configz" \
  | jq '.kubeletconfig | {containerLogMaxSize, containerLogMaxFiles}' \
  | tee rotation-evidence.txt

Read-only / Safeworkstation— The kubelet's live configuration, read through the API server rather than from a file

$ kubectl get --raw "/api/v1/nodes/$NODE/proxy/configz" | jq '.kubeletconfig | {containerLogMaxSize, containerLogMaxFiles}'

{
"containerLogMaxSize": "10Mi",
"containerLogMaxFiles": 5
}

Illustrative output

Those two numbers are the entire local retention policy for every container on that node. Now look at what they produced:

# Substitute the address of the node named in pod-and-node.txt:
NODE_ADDR=192.0.2.12

ssh "$NODE_ADDR" 'sudo ls -lh /var/log/pods/logging-lab_chatty-*/chatty/' \
  | tee -a rotation-evidence.txt

Read the listing rather than trusting a description of it. Record the exact names your kubelet produced and whether the rotated files are compressed — both are implementation details that have changed between releases, and compression matters, because a compressed rotated file holds far more history than its size suggests when the log lines are repetitive. What is stable is the shape: one current file, a bounded number of older ones, and nothing at all beyond that.

Now ask what kubectl logs can still show you:

cd "$HOME/k8s-logging-lab"
POD=$(kubectl get pod -n logging-lab -l app=chatty -o jsonpath='{.items[0].metadata.name}')

echo "first line kubectl can return:" | tee -a rotation-evidence.txt
kubectl logs -n logging-lab "$POD" | head -1 | tee -a rotation-evidence.txt

echo "lines kubectl can return:" | tee -a rotation-evidence.txt
kubectl logs -n logging-lab "$POD" | wc -l | tee -a rotation-evidence.txt

The application emitted 200,000 numbered lines and said so on the last line of the burst. Compare that with the sequence number on the first line kubectl logs gives you back.

Whatever number you get is the answer for this cluster, and it is the number worth recording — not the one in anybody’s design document. It is a product of three things: the container’s log rate, the two caps you just read, and whether your kubelet’s log reader follows rotated files at all. That last point is a version-dependent implementation detail, which is exactly why the task is to measure it rather than to be told it.

If the first line you get back is not seq=1, the missing lines existed, were correct, were never replicated anywhere, and are now unrecoverable — removed by a mechanism that exists to stop a chatty container filling a node’s disk, which is a mechanism you want. If you do get seq=1, the burst simply did not exceed this node’s local budget, and the useful exercise is to work out how many lines it would have taken.

Now the failure that no cap protects you from:

cd "$HOME/k8s-logging-lab"
POD=$(kubectl get pod -n logging-lab -l app=chatty -o jsonpath='{.items[0].metadata.name}')

kubectl delete pod -n logging-lab "$POD"
sleep 5

kubectl logs -n logging-lab "$POD" 2>&1 | tee -a rotation-evidence.txt

NotFound. Not “no logs” — the Pod object is gone, so there is nothing to ask about. The kubelet’s garbage collection removes the directory under /var/log/pods shortly afterwards. Confirm it:

# Substitute the address of the node named in pod-and-node.txt:
NODE_ADDR=192.0.2.12

ssh "$NODE_ADDR" 'sudo ls -d /var/log/pods/logging-lab_chatty-* 2>/dev/null || echo "directory removed"'

This is the loss that matters, because the events that delete Pods are exactly the events you investigate afterwards: an eviction, a node replacement, a rollout, a kubectl delete pod by the person debugging. The log’s lifetime is tied to the object whose failure you are trying to explain.

Task 4: Build the sink

Loki first, so the collector has somewhere to send to.

loki.yaml:

apiVersion: v1
kind: Namespace
metadata:
  name: logging
  labels:
    # The collector in Task 5 mounts /var/log from the host, which baseline
    # forbids. The exception is deliberate, scoped to this namespace, and
    # left visible: audit and warn stay on at baseline so every Pod creation
    # here announces what it is doing.
    pod-security.kubernetes.io/enforce: privileged
    pod-security.kubernetes.io/audit: baseline
    pod-security.kubernetes.io/warn: baseline
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: loki
  namespace: logging
spec:
  replicas: 1
  selector:
    matchLabels:
      app: loki
  template:
    metadata:
      labels:
        app: loki
    spec:
      securityContext:
        # The upstream image runs as UID 10001 and needs to own its data
        # directory. fsGroup makes the emptyDir writable by that GID.
        runAsUser: 10001
        runAsGroup: 10001
        fsGroup: 10001
      containers:
        - name: loki
          image: grafana/loki:3.5
          # No -config.file argument: the image ships a single-binary
          # filesystem configuration at /etc/loki/local-config.yaml and uses
          # it by default. Correct for a lab. Wrong for production, where
          # storage, retention and limits are all decisions you must make.
          ports:
            - name: http
              containerPort: 3100
          readinessProbe:
            httpGet:
              path: /ready
              port: http
            initialDelaySeconds: 20
            periodSeconds: 10
          volumeMounts:
            - name: data
              mountPath: /loki
          resources:
            requests:
              cpu: 100m
              memory: 256Mi
            limits:
              memory: 1Gi
      volumes:
        - name: data
          # An emptyDir: the log store dies with the Pod. That is honest for
          # a lab and is the single biggest difference from a real install.
          emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: loki
  namespace: logging
spec:
  selector:
    app: loki
  ports:
    - name: http
      port: 3100
      targetPort: http

cd "$HOME/k8s-logging-lab"

kubectl apply -f loki.yaml
kubectl rollout status deployment/loki -n logging --timeout=180s
kubectl get pods -n logging -o wide

If the Pod crash-loops, read its logs and then read the configuration it is actually using — the bundled file names the paths it expects to own, and a permissions failure will name one of them:

kubectl logs -n logging deploy/loki --tail=20
kubectl exec -n logging deploy/loki -- cat /etc/loki/local-config.yaml

That second command is the habit worth keeping. When a container ships its own default configuration, read the default rather than guessing at it.

Task 5: Build the collector

The collector needs three things that this course has already covered separately: a ServiceAccount with RBAC, because the Kubernetes filter asks the API server which Pod each log file belongs to; a hostPath mount, which is why the namespace carries a Pod Security exception; and a position database on a writable path, which is what makes a restart resume instead of duplicating or skipping.

fluent-bit.yaml:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluent-bit
  namespace: logging
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluent-bit
rules:
  # The minimum the kubernetes filter needs to turn a filename into
  # namespace, pod, container and labels. No write verbs, no secrets.
  - apiGroups: [""]
    resources: ["pods", "namespaces"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: fluent-bit
subjects:
  - kind: ServiceAccount
    name: fluent-bit
    namespace: logging
roleRef:
  kind: ClusterRole
  name: fluent-bit
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: logging
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush         1
        Log_Level     info
        Daemon        Off
        Parsers_File  parsers.conf

    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*.log
        Parser            cri
        DB                /var/lib/fluent-bit/positions.db
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Refresh_Interval  5

    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_Tag_Prefix     kube.var.log.containers.
        Merge_Log           On
        Keep_Log            Off

    [OUTPUT]
        Name       loki
        Match      kube.*
        Host       loki.logging.svc.cluster.local
        Port       3100
        labels     job=fluent-bit
        label_keys $kubernetes['namespace_name'],$kubernetes['pod_name'],$kubernetes['container_name']
  parsers.conf: |
    [PARSER]
        Name        cri
        Format      regex
        Regex       ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) ?(?<log>.*)$
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L%z
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: logging
spec:
  selector:
    matchLabels:
      app: fluent-bit
  template:
    metadata:
      labels:
        app: fluent-bit
    spec:
      serviceAccountName: fluent-bit
      # A log collector must run everywhere, including on nodes that are
      # tainted for other reasons - a node you cannot collect from is a node
      # whose incidents you cannot investigate.
      tolerations:
        - operator: Exists
      containers:
        - name: fluent-bit
          image: fluent/fluent-bit:3.2
          volumeMounts:
            - name: config
              mountPath: /fluent-bit/etc/fluent-bit.conf
              subPath: fluent-bit.conf
            - name: config
              mountPath: /fluent-bit/etc/parsers.conf
              subPath: parsers.conf
            - name: varlog
              mountPath: /var/log
              readOnly: true
            - name: positions
              mountPath: /var/lib/fluent-bit
          resources:
            requests:
              cpu: 50m
              memory: 64Mi
            limits:
              memory: 200Mi
      volumes:
        - name: config
          configMap:
            name: fluent-bit-config
        - name: varlog
          # /var/log covers both the symlinks in /var/log/containers and
          # their targets in /var/log/pods. Read-only: the collector has no
          # business writing here.
          hostPath:
            path: /var/log
            type: Directory
        - name: positions
          # Writable, and on the host on purpose. A position database in an
          # emptyDir is lost when the collector restarts, and the collector
          # then either re-sends everything or skips to the end.
          hostPath:
            path: /var/lib/fluent-bit
            type: DirectoryOrCreate

Note the two subPath mounts rather than mounting the ConfigMap over /fluent-bit/etc. Mounting the directory would hide the other files the image ships there, and the failure is confusing: the process starts, reads a configuration that is missing pieces, and behaves almost correctly.

cd "$HOME/k8s-logging-lab"

kubectl apply -f fluent-bit.yaml 2>&1 | tee -a fluent-bit-apply.txt
kubectl rollout status daemonset/fluent-bit -n logging --timeout=180s
kubectl get pods -n logging -o wide

kubectl logs -n logging -l app=fluent-bit --tail=20

The apply prints a Pod Security warning naming the hostPath volume, and then succeeds. That is the exception doing its job: enforcement is off in this namespace, visibility is not, and the warning appears every time a Pod is created here.

Task 6: Query the sink, and prove the logs outlive the Pod

Open a port-forward and leave it running in its own terminal:

kubectl port-forward -n logging svc/loki 3100:3100

In a second terminal:

cd "$HOME/k8s-logging-lab"

curl -s http://127.0.0.1:3100/ready; echo

# Which labels exist? These are the ones label_keys produced, plus job.
curl -s http://127.0.0.1:3100/loki/api/v1/labels | jq -r '.data[]'

# Values for one of them, to confirm the enrichment worked.
curl -s http://127.0.0.1:3100/loki/api/v1/label/namespace_name/values | jq -r '.data[]'

If namespace_name is missing, the Kubernetes filter did not enrich the records — go back to the collector’s logs before doing anything else. A pipeline that ships lines without identity is a pipeline you cannot query.

Now read some lines back:

cd "$HOME/k8s-logging-lab"

NOW=$(date +%s)
START=$(( NOW - 3600 ))

curl -sG http://127.0.0.1:3100/loki/api/v1/query_range \
  --data-urlencode 'query={namespace_name="logging-lab"}' \
  --data-urlencode "start=${START}000000000" \
  --data-urlencode "end=${NOW}000000000" \
  --data-urlencode 'limit=5' \
  | jq -r '.data.result[].values[][1]'

Loki’s range API takes Unix nanoseconds, which is why the shell arithmetic above pads with nine zeroes. Every timestamp in this pipeline is nanosecond precision, from the CRI line onward, and mixing units is the most common reason a query returns nothing when the data is definitely there.

Now the payoff. Delete the Pod, then ask both systems the same question:

cd "$HOME/k8s-logging-lab"

POD=$(kubectl get pod -n logging-lab -l app=chatty -o jsonpath='{.items[0].metadata.name}')
echo "deleting $POD"
kubectl delete pod -n logging-lab "$POD"
sleep 10

# The cluster has forgotten.
kubectl logs -n logging-lab "$POD" 2>&1 | head -1

# The sink has not.
NOW=$(date +%s); START=$(( NOW - 3600 ))
curl -sG http://127.0.0.1:3100/loki/api/v1/query_range \
  --data-urlencode "query={pod_name=\"$POD\"}" \
  --data-urlencode "start=${START}000000000" \
  --data-urlencode "end=${NOW}000000000" \
  --data-urlencode 'limit=3' \
  | jq -r '.data.result[].values[][1]'

kubectl logs returns NotFound for a Pod that no longer exists. The same lines come back from Loki, addressed by the dead Pod’s name, because the identity was attached to the record at collection time rather than looked up at query time. That is the entire value proposition of the pipeline, in two commands.

Task 7: Break the sink and measure the gap

“We have centralised logging” should come with a number. Get one.

cd "$HOME/k8s-logging-lab"

# Count what Loki holds for the namespace right now.
curl -sG http://127.0.0.1:3100/loki/api/v1/query \
  --data-urlencode 'query=count_over_time({namespace_name="logging-lab"}[1h])' \
  | jq -r '.data.result[] | .value[1]' | tee gap-measurement.txt

count_over_time is a LogQL range aggregation: it counts the entries matching the stream selector over the window, server-side. Use it rather than paging through query_range — the range API is capped at a few thousand entries per request, so counting by fetching is both slow and wrong.

Now take the sink away and generate a burst while it is down:

Service impact possibleworkstation— Simulates a sink outage: the collector keeps tailing, and has nowhere to send

$ kubectl scale deployment/loki -n logging --replicas=0

cd "$HOME/k8s-logging-lab"

kubectl rollout restart deployment/chatty -n logging-lab
kubectl rollout status deployment/chatty -n logging-lab --timeout=120s

# Let the burst run against a dead sink.
sleep 60
kubectl logs -n logging -l app=fluent-bit --tail=10

kubectl scale deployment/loki -n logging --replicas=1
kubectl rollout status deployment/loki -n logging --timeout=180s

Restart the port-forward — it died with the Pod — then count again:

cd "$HOME/k8s-logging-lab"

curl -sG http://127.0.0.1:3100/loki/api/v1/query \
  --data-urlencode 'query=count_over_time({namespace_name="logging-lab"}[1h])' \
  | jq -r '.data.result[] | .value[1]' | tee -a gap-measurement.txt

# What is the lowest sequence number that survived the outage?
NOW=$(date +%s); START=$(( NOW - 1800 ))
curl -sG http://127.0.0.1:3100/loki/api/v1/query_range \
  --data-urlencode 'query={namespace_name="logging-lab"} |= "seq="' \
  --data-urlencode "start=${START}000000000" \
  --data-urlencode "end=${NOW}000000000" \
  --data-urlencode 'direction=forward' \
  --data-urlencode 'limit=1' \
  | jq -r '.data.result[].values[][1]' | tee -a gap-measurement.txt

Whether you lost anything depends on a race you can name

The Loki store in this lab is an emptyDir, so scaling to zero destroyed everything it had already ingested — that part of the gap is an artefact of the lab’s own shortcut, and you should say so in your notes rather than counting it as pipeline loss.

The interesting question is what happened to the lines written during the outage. Fluent Bit was still tailing the file and recording its position. When the output failed it retried, and with Mem_Buf_Limit reached it paused the input rather than reading further. Whether those lines arrive after recovery comes down to one race: the file the position database points into must still exist. If the kubelet rotated it away first, the position refers to a file that is gone and the lines in it were never anywhere else.

That is why local rotation settings and sink availability are not independent knobs. Your local buffer — containerLogMaxSize multiplied by containerLogMaxFiles, divided by the workload’s log rate — is the length of sink outage you can survive, and it is a number you can compute for your own cluster in about a minute.

Compute it and write it down. For the defaults, 5 files of 10 MiB is 50 MiB per container; a container logging 1 MiB per minute survives about fifty minutes of sink outage, and one logging 10 MiB per minute survives five.

What could go wrong?

What happens if the collector Pod restarts?

It reads its position database and resumes where it stopped. That is the entire purpose of the DB setting, and it is why the database is on a hostPath here rather than in an emptyDir — a position file that dies with the Pod turns every collector restart into either a flood of duplicates or a silent gap, depending on the plugin’s default.

What happens if two collectors run on the same node?

Both tail the same files and both ship. Loki accepts the duplicates, your line counts double, and nothing warns you. This is a real outcome of installing a second logging stack “to evaluate it” without removing the first.

What happens if an application logs to a file inside the container instead of stdout?

Nothing in this pipeline sees it. The file lives in the container’s writable layer, it is not in /var/log/pods, kubectl logs does not show it, and it is destroyed with the container. Collecting it needs a sidecar that tails the file and echoes it to stdout, or a shared volume — both of which are workarounds for an application that should log to stdout.

What happens if a log line is larger than the runtime’s buffer?

containerd splits it and tags the pieces P until the final F. Without multiline reassembly in the collector, a 40 KB stack trace arrives as three unrelated records, and the query that would have found it matches only the fragment containing the search term.

What happens if every Pod label becomes a Loki label?

Loki builds one stream per unique label combination, and the number of streams is the thing that governs its memory and query cost. Turning every Kubernetes label into a Loki label — including per-deployment hashes — is the canonical way to make a Loki cluster fall over. The label_keys line in the configuration above names three labels deliberately; everything else stays inside the log line, where a filter expression can still find it.

Validation

Run these against the finished state. Each proves an outcome rather than restating a step.

cd "$HOME/k8s-logging-lab"

# 1. The collector is running on every node.
kubectl get daemonset fluent-bit -n logging \
  -o jsonpath='{.status.numberReady}/{.status.desiredNumberScheduled}'; echo

# 2. Identity is attached to the records, not inferred at query time.
curl -s http://127.0.0.1:3100/loki/api/v1/labels | jq -r '.data[]' \
  | grep -q 'namespace_name' && echo "PASS: enrichment worked"

# 3. Logs exist in the sink for a Pod that no longer exists.
kubectl get pods -n logging-lab -o name
grep -q 'seq=' gap-measurement.txt && echo "PASS: shipped lines are readable"

# 4. The sink can count, server-side.
curl -sG http://127.0.0.1:3100/loki/api/v1/query \
  --data-urlencode 'query=count_over_time({job="fluent-bit"}[5m])' \
  | jq -r '.data.result[] | .value[1]'

# 5. The collector's RBAC is read-only.
kubectl auth can-i create pods \
  --as=system:serviceaccount:logging:fluent-bit && echo "FAIL: collector can write"

# 6. The exception namespace still warns.
kubectl apply -f fluent-bit.yaml --dry-run=server 2>&1 | grep -q 'Warning' \
  && echo "PASS: the hostPath exception is still visible"

Check 4 must return a rising number on repeated runs — a static count means the collector has stopped and nothing has told you. Check 5 must print nothing before the echo, because kubectl auth can-i exits non-zero on no; if you see FAIL, the collector has write access it does not need.

Expected Outcome

k8s-logging-lab/
├── chatty.yaml
├── fluent-bit.yaml
├── fluent-bit-apply.txt
├── gap-measurement.txt
├── log-path-trace.txt
├── loki.yaml
├── pod-and-node.txt
└── rotation-evidence.txt

On the cluster: a logging-lab namespace at enforce: restricted running one workload that logs to stdout; a logging namespace at enforce: privileged with baseline audit and warn, running one Loki and one Fluent Bit per node; and a query that returns log lines for a Pod the cluster has forgotten.

Production notes

This lab maps onto two changes with different risk profiles, and a third thing that is not a change at all.

Deploying the collector is a per-node change with host access. A DaemonSet with a hostPath mount and a cluster-wide read of Pods and Namespaces touches every node and reads metadata from every workload. It belongs in a change window, with the RBAC in the change record — a log collector is a legitimate reason for a cluster-wide read grant and it is still a cluster-wide read grant. The rollout should be staged: one node first, verify lines arrive with correct labels, then the rest. tolerations matter here in a way they do not for most workloads, because the node you cannot collect from is the node whose incident you cannot reconstruct.

Deploying the sink is a capacity decision dressed as an install. The emptyDir in this lab hides every question that matters: where chunks live, how long they are kept, what the per-tenant ingestion limit is, and what happens when the object store is unreachable. Answer those before the first production log line, because retention applied retroactively does not bring anything back.

Measuring your local buffer is not a change, and it is the highest-value hour in this list. Take your containerLogMaxSize, multiply by containerLogMaxFiles, divide by the log rate of your noisiest container, and you have the length of sink outage your cluster survives without loss. Most teams have never computed it, and most are surprised by how small it is.

Hold is a legitimate outcome here too. If the collector cannot be given its hostPath mount because a policy forbids it, do not weaken the policy under time pressure to get the rollout done. Record the blocker, name the namespace exception that would be needed, name the owner, and stop. An exception granted in a hurry and never reviewed is how privileged namespaces multiply.

Troubleshooting

Fluent Bit runs but ships nothing. Check the tail input matched files: kubectl exec -n logging ds/fluent-bit -- ls /var/log/containers | head. An empty listing means the hostPath mount is wrong or the node uses a different path.

Records arrive without namespace_name or pod_name. The Kubernetes filter could not enrich them, usually because Kube_Tag_Prefix does not match the tag the tail input generated. The tag is derived from the path, so Path /var/log/containers/*.log with Tag kube.* produces kube.var.log.containers.FILENAME, and the prefix must match exactly, trailing dot included.

Loki returns an empty result for a query you know should match. Check the time range first — the range API takes nanoseconds, and a range in seconds silently matches nothing. Then check the label name with /loki/api/v1/labels rather than assuming it.

Loki crash-loops on startup. Read its logs, then read /etc/loki/local-config.yaml inside the container to see which paths it expects to own. A permissions error names the directory; fsGroup on the Pod is what makes the emptyDir writable by the image’s user.

The port-forward keeps dying. It is one TCP session to one Pod. If Loki restarts, the forward drops. Repeated drops mean Loki is being OOM-killed; check the restart count before blaming the network.

Log lines appear twice. Either two collectors are running — check for a second logging stack — or the position database was lost and Fluent Bit re-read from the start of the file.

A multi-line stack trace arrives as several unrelated records. That is the CRI P/F split. Fluent Bit’s multiline support reassembles them; the minimal configuration in this lab does not, which is a deliberate omission so that you see the raw behaviour first.

Cleanup

Order matters slightly: remove the collector before the sink, so it does not spend the interval retrying against a Service that no longer exists.

Step 1. Remove the workloads:

cd "$HOME/k8s-logging-lab"

kubectl delete -f fluent-bit.yaml --ignore-not-found
kubectl delete -f loki.yaml --ignore-not-found
kubectl delete -f chatty.yaml --ignore-not-found

kubectl get ns logging logging-lab 2>&1 | grep -q NotFound \
  && echo "namespaces removed"

Deleting fluent-bit.yaml removes the ClusterRole and ClusterRoleBinding as well as the namespaced objects, because they are in the same file. Confirm it, because a cluster-wide read grant left behind by a deleted collector is exactly the orphan that an RBAC audit finds two years later:

kubectl get clusterrole fluent-bit 2>&1 | grep -q NotFound \
  && kubectl get clusterrolebinding fluent-bit 2>&1 | grep -q NotFound \
  && echo "cluster-scoped RBAC removed"

Step 2. The collector left a position database on every node, and the kubelet’s own log files are unaffected by any of this. Remove the first:

This deletes a directory on every node, over SSH, as root

/var/lib/fluent-bit was created by the DaemonSet with type: DirectoryOrCreate, so it exists on every node the collector ran on and nothing else will ever remove it. The command below deletes it.

/var/lib on a Kubernetes node also holds /var/lib/kubelet and /var/lib/containerd. A mistyped path here is not recoverable. Print the directory before removing it, and read what you printed:

# Substitute each node address in turn:
NODE_ADDR=192.0.2.12
ssh "$NODE_ADDR" 'ls -la /var/lib/fluent-bit'

Destructiveworker— Removes the collector's position database; verify the path first, and repeat per node

$ sudo rm -rf /var/lib/fluent-bit

Step 3. Confirm the cluster is back where it started:

# Substitute the address of the node named in pod-and-node.txt:
NODE_ADDR=192.0.2.12

ssh "$NODE_ADDR" 'sudo ls -d /var/log/pods/logging* 2>/dev/null || echo "no lab pod log directories remain"'
kubectl get namespaces -o name | grep -E 'logging' || echo "no lab namespaces remain"

Keep log-path-trace.txt, rotation-evidence.txt and gap-measurement.txt — they are the deliverables, and the third one is the number to quote next time someone says the logs will be there.

Under the hood

There is no logging subsystem in Kubernetes. What exists is a convention and two garbage collectors.

The convention is the CRI logging format. When containerd starts a container it attaches to the process’s stdout and stderr and writes each line to a file as TIMESTAMP STREAM TAG MESSAGE, where the timestamp is RFC 3339 with nanoseconds, the stream is stdout or stderr, and the tag is F for a complete line or P for a fragment of one that exceeded the read buffer. The kubelet chooses the path: /var/log/pods/NAMESPACE_NAME_UID/CONTAINER/N.log, where N increments each time the container restarts — which is precisely what kubectl logs --previous reads, and why it can only ever go back one restart.

/var/log/containers is a second view of the same files. The kubelet maintains symlinks there whose names encode pod, namespace and container id, because a collector that globs a directory can extract metadata from the filename without asking anybody. Everything downstream depends on that naming convention, which is why the Kubernetes filter’s Kube_Tag_Prefix has to match the tail input’s path exactly.

The first garbage collector is the kubelet’s container log manager. It polls the current log file, and when it exceeds containerLogMaxSize it rotates: the current file is renamed with a timestamp suffix and compressed, a new current file is opened, and files beyond containerLogMaxFiles are removed. A collector holding an open file descriptor on a rotated file can keep reading it until it is unlinked; one that has fallen behind loses the difference.

The second is Pod garbage collection. When the Pod object is deleted the kubelet removes the whole /var/log/pods/NAMESPACE_NAME_UID directory, because the UID in the path will never be reused. This is not configurable by retention time — there is no “keep logs for a deleted Pod for an hour” setting — and it is the reason the log’s lifetime is bounded by the object whose failure you are usually investigating.

Loki’s side is a different shape again. It does not index log content; it indexes labels, and stores the lines for each unique label combination as a compressed chunk stream. A query is therefore a stream selector — which chunks to open — followed by a filter expression applied to the decompressed lines. That design is why label cardinality is the dominant cost: each new unique combination is a new stream with its own index entry and its own chunks, and a label derived from a Pod name or a request id multiplies both without bound.

What You Learned

kubectl logs is a file read, not a log system. One file per container instance, on one node, with a size cap, a file-count cap, and a lifetime tied to the Pod object.
The three losses are independent and all silent. Rotation drops the oldest lines, a container restart pushes the previous instance behind --previous, and Pod deletion removes everything. None of them produces an error, an event, or a gap marker.
The stored format is not what the application wrote. Timestamp, stream, and an F/P tag come first, which is why a parser is mandatory and why long lines arrive in pieces.
Identity is attached at collection time. That is the reason the shipped copy can be queried by the name of a Pod that no longer exists, and the reason the collector needs RBAC.
Your local buffer is a computable number. Size times file count divided by log rate is how long a sink outage your cluster survives, and it is usually much shorter than people assume.
A log collector is the one system that cannot report its own failure. Monitor its output rate, because silence from a logging pipeline is indistinguishable from quiet.
Label cardinality is the sink’s cost model. Three deliberate labels, and everything else inside the line where a filter can still reach it.

Why this matters

The engineer in the Scenario did everything right and finished the incident review with “root cause could not be determined”. Nothing had failed. The logs were deleted by mechanisms that exist to protect the cluster, on a schedule nobody had ever measured, and their absence was discovered at the exact moment they were needed and not one second earlier.

That is the characteristic failure of observability: it is not detected by the thing it is supposed to detect. A monitoring gap shows up as a blind spot on a dashboard; a logging gap shows up as a question you cannot answer, months later, under time pressure, with an audience. The number you computed in Task 7 — how long a sink outage your cluster survives, and how long a deleted Pod’s output remains recoverable — is not a nice-to-have metric. It is the honest answer to “can we investigate this”, and the only time to work it out is before somebody asks.

Objectives

Prerequisites

Objective

Architecture

Requirements

Scenario

Tasks

Task 1: Deploy something that logs in a countable way

Task 2: Find the file, and read what is actually in it

Task 3: Measure how far back your cluster can actually look

Task 4: Build the sink

Task 5: Build the collector

Task 6: Query the sink, and prove the logs outlive the Pod

Task 7: Break the sink and measure the gap

Validation

Expected Outcome

Production notes

Troubleshooting

Cleanup

What You Learned

Deliverables

Verification status