Inspect object → events → logs — the evidence-gathering arc

The evidence-gathering arc is the cluster’s way of telling the operator what happened. The arc is: inspect the object, read the events, read the logs. The arc converts a symptom into a candidate cause; the discipline is to follow every step.

Step 3: Inspect the object

The object is the Kubernetes resource that is misbehaving. The inspection is kubectl get and kubectl describe.

kubectl get pod billing-7d8f-abcde -n prod -o wide
kubectl describe pod billing-7d8f-abcde -n prod

The output is the object’s spec, status, conditions, and events. The output is the input to step 4.

flowchart TD
    A[kubectl get] --> B[Spec, status, conditions]
    B --> C[kubectl describe] --> D[Events, references]
    D --> E[Output]

The kubectl get -o yaml is the canonical first command. The YAML output is the source of truth; the describe output is the human-readable summary.

Step 4: Events

The events are the cluster’s history of the object. Events are time-ordered, scoped to the namespace, and have a type (Normal, Warning) and a reason. The events are the narrative of how the object got to its current state.

kubectl get events -n prod --sort-by=.lastTimestamp \
  --field-selector involvedObject.name=billing-7d8f-abcde

The output is a list of events:

LAST SEEN   TYPE      REASON              OBJECT                           MESSAGE
2m          Warning   FailedScheduling    pod/billing-7d8f-abcde          0/12 nodes are available: 3 Insufficient memory, 9 Insufficient cpu.
3m          Normal    Scheduled           pod/billing-7d8f-abcde          Successfully assigned billing-7d8f-abcde to node-03
3m          Normal    Pulling             pod/billing-7d8f-abcde          Pulling image "billing:1.2.3"
3m          Normal    Pulled              pod/billing-7d8f-abcde          Successfully pulled image "billing:1.2.3"
3m          Normal    Created             pod/billing-7d8f-abcde          Created container billing
3m          Warning   CrashLoopBackOff    pod/billing-7d8f-abcde          Back-off restarting failed container

The events are the cluster’s memory. They are the input to step 5.

Step 5: Logs

The logs are the container’s history. The logs are ordered by line, scoped to the container, and contain the application’s output. The logs are the narrative of how the container got to its current state.

kubectl logs -n prod billing-7d8f-abcde -c billing --previous
kubectl logs -n prod billing-7d8f-abcde -c billing --tail=200

The --previous flag is the key: it shows the logs of the previous container instance, which is the one that crashed before the current restart. The --tail=200 flag limits the output to the last 200 lines.

The logs are the application’s memory. They are the input to step 6.

Why the arc is in this order

The arc is in this order because each step’s output is the next step’s input:

• kubectl get shows the object’s current state. The state is the what.
• kubectl describe shows the events and references. The events are the how (the cluster’s view).
• kubectl logs shows the application’s output. The logs are the why (the application’s view).

The three views are different lenses on the same object. The operator who reads all three converges on the candidate cause faster than the operator who reads one.

Failure modes of the arc

The most common failure modes:

• Skipping kubectl describe. The operator reads the logs and begins to interpret the application’s view. The interpretation is incomplete; the events are the cluster’s view of the same object.
• Skipping --previous. The operator reads the current logs of the running container. The current logs are the application starting up; the previous logs are the application crashing.
• Reading the wrong container. A Pod can have multiple containers; the operator reads the wrong container’s logs. The -c flag is the remediation.

Production discipline

The evidence-gathering arc is the cluster’s way of telling the operator what happened. The discipline is to run the arc in order, never skip a step, and use the same commands every time. The arc’s output is the input to step 6.

• Use --previous for CrashLoopBackOff. The failing container is the previous one.
• Use -c for multi-container Pods. The wrong container is a common failure mode.
• Use -o yaml for the canonical view. The describe is the summary; the YAML is the source of truth.

Quiz