Lab 3: Deploy and inspect Pods

Objective

By the end of this lab you will have built one Pod that is deliberately more than a container — an init container, an application container and a sidecar, sharing one IP and one volume — and you will have inspected it from three different vantage points: the API server, the inside of the containers, and the node’s container runtime.

The artefact that matters is the diff in Task 3. You will write about forty lines of YAML and the API server will store about a hundred and forty. Every one of the lines you did not write is a default, a mutation or a projection that something in the cluster decided on your behalf, and being unable to account for them is the reason people find Pod objects intimidating to read.

The second thing you will leave with is a habit: kubectl logs is not the first command. On a healthy Pod it tells you almost nothing that describe and get -o yaml did not already say, and on several classes of broken Pod it has nothing to tell you at all.

Architecture

Everything lives in one namespace. Nothing touches a node’s configuration, the control plane, or any existing workload.

kubeadm cluster (1.34.x)
  k8s-cp-1     control plane
  k8s-w-1      worker      the Pod lands on one of these
  k8s-w-2      worker

namespace podinspect
  web                        Pod
    initContainers:
      seed-content           busybox, writes index.html into the shared volume, exits 0
    containers:
      app                    nginx, serves /usr/share/nginx/html from that volume
      sidecar                busybox, appends a heartbeat line to the same volume
    volumes:
      content                emptyDir, mounted by all three

  web-managed                Deployment, 2 replicas, for Task 9 only

workstation
  ~/k8s-podinspect/manifests/   what you wrote
  ~/k8s-podinspect/evidence/    what the cluster said

The shape is deliberate. seed-content proves ordering, sidecar proves the shared network namespace and the shared volume, and app proves that a container’s root filesystem is still its own.

Requirements

• A kubeadm cluster on Kubernetes 1.34.x with at least one schedulable worker. B-nested is sufficient. A single-node cluster works if the control-plane taint has been removed, but Task 9 is less interesting on one.
• kubectl 1.34.x with permission to create and delete a namespace, and to create Pods, Deployments and ephemeral containers in it.
• Shell access to the worker node with sudo, for Task 7. If you cannot reach a node, Task 7 is the only one you must skip; everything else runs from the workstation.
• Outbound access from the nodes to docker.io for nginx:1.27.2 and busybox:1.36. Together they are under 80 MB.
• Roughly 150 MiB of memory and 0.2 CPU of cluster headroom. The Pods carry explicit requests so that the numbers are visible rather than implicit.
• No out-of-band access requirement. Nothing in this lab reconfigures networking, SSH or a firewall, and nothing it does can lock you out of a node.

Scenario

Somebody hands you a Pod name and says it is “up but weird”. Before you can form a hypothesis you need to know what the Pod actually is: how many containers, which one is serving, what it mounts, which node it is on, what it was asked for versus what it got.

The reflex is to run kubectl logs and read whatever comes out. That habit fails in a specific and common way: it answers a question about one container’s stdout while the interesting facts — that a second container is holding the port, that the volume is empty because the init container failed, that the image is not the tag you think — live in the object.

This lab builds a Pod complicated enough for that distinction to matter, and then asks the object rather than the log.

Tasks

Task 1 — Workspace, namespace, and a baseline you can diff against

NS=podinspect

mkdir -p "$HOME/k8s-podinspect/manifests" "$HOME/k8s-podinspect/evidence"
cd "$HOME/k8s-podinspect"

kubectl get namespace "$NS" > evidence/00-namespace-before.txt 2>&1
cat evidence/00-namespace-before.txt

kubectl create namespace "$NS"
kubectl config set-context --current --namespace="$NS"
kubectl version -o yaml > evidence/00-versions.yaml
kubectl get nodes -o wide > evidence/00-nodes.txt
kubectl get nodes -o custom-columns='NAME:.metadata.name,VERSION:.status.nodeInfo.kubeletVersion,UNSCHEDULABLE:.spec.unschedulable' \
  > evidence/00-nodes-stable.txt

00-namespace-before.txt is the file Cleanup consults. If it reads Error from server (NotFound), deleting the namespace at the end destroys only what this lab made. If it reads anything else, the namespace was already there and deleting it takes somebody else’s work with it.

Setting the context’s default namespace is a convenience with a sharp edge: every unqualified kubectl command for the rest of this session targets podinspect, including ones you run after you have stopped thinking about the lab. Cleanup sets it back.

Task 2 — Write the Pod

manifests/web-pod.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: web
  namespace: podinspect
  labels:
    app: web
    tier: frontend
spec:
  initContainers:
    - name: seed-content
      image: busybox:1.36
      command:
        - sh
        - -c
        - |
          echo "served from a volume the init container wrote" > /work/index.html
          echo "pod: $(hostname)" >> /work/index.html
      volumeMounts:
        - name: content
          mountPath: /work
  containers:
    - name: app
      image: nginx:1.27.2
      ports:
        - name: http
          containerPort: 80
      volumeMounts:
        - name: content
          mountPath: /usr/share/nginx/html
      resources:
        requests:
          cpu: 50m
          memory: 32Mi
        limits:
          memory: 64Mi
    - name: sidecar
      image: busybox:1.36
      command:
        - sh
        - -c
        - |
          while true; do
            date -Is >> /work/heartbeat.log
            sleep 5
          done
      volumeMounts:
        - name: content
          mountPath: /work
      resources:
        requests:
          cpu: 10m
          memory: 16Mi
        limits:
          memory: 32Mi
  volumes:
    - name: content
      emptyDir: {}

Count what you wrote before you apply it — the number is half of Task 3:

cd "$HOME/k8s-podinspect"
wc -l manifests/web-pod.yaml

Task 3 — Ask the API server what it would store, and diff

--dry-run=server sends the object through the whole request pipeline — authentication, authorisation, mutating admission, validation, defaulting — and returns the result without persisting it. It is the only way to see what admission did to your object separately from what the controllers did to it afterwards.

cd "$HOME/k8s-podinspect"

kubectl apply -f manifests/web-pod.yaml --dry-run=server -o yaml \
  > evidence/03-server-defaulted.yaml

wc -l manifests/web-pod.yaml evidence/03-server-defaulted.yaml

The stored object is several times the size of the manifest. Go through the difference deliberately rather than scrolling past it:

cd "$HOME/k8s-podinspect"

grep -nE 'restartPolicy|dnsPolicy|schedulerName|serviceAccount|terminationGracePeriodSeconds|imagePullPolicy|enableServiceLinks|preemptionPolicy|priority:' \
  evidence/03-server-defaulted.yaml

Each of those is a field you did not write and now own the consequences of. terminationGracePeriodSeconds: 30 is the budget a container gets between SIGTERM and SIGKILL. restartPolicy: Always is why a crashing container in this Pod will restart forever rather than stopping. imagePullPolicy: IfNotPresent follows from having pinned a tag; had you written nginx:latest, the default would have been Always instead, and every Pod start would have contacted the registry.

Two more additions are worth finding by name:

cd "$HOME/k8s-podinspect"

grep -nA6 'tolerations:' evidence/03-server-defaulted.yaml
grep -nB2 -A8 'kube-api-access' evidence/03-server-defaulted.yaml

The tolerations were added by an admission plugin: they say this Pod will tolerate its node going NotReady or unreachable for a bounded number of seconds before the node controller evicts it. That number is the delay between a node dying and its Pods being rescheduled, and almost nobody who complains about that delay knows it is written into every Pod they create.

The kube-api-access volume is the projected ServiceAccount token: a short-lived, audience-bound credential mounted at /var/run/secrets/kubernetes.io/serviceaccount. Your Pod can talk to the API server whether or not you wanted it to.

Now apply it for real and watch it start:

cd "$HOME/k8s-podinspect"

kubectl apply -f manifests/web-pod.yaml
kubectl wait pod/web --for=condition=Ready --timeout=120s
kubectl get pod web -o wide

Task 4 — The four questions, on one healthy Pod

Four commands, four different answers. Run each and record, in evidence/triage.md, one fact it gave you that the others did not.

cd "$HOME/k8s-podinspect"

# 1. What is the system doing right now?
kubectl get pods -o wide | tee evidence/04-get.txt

# 2. What does the controller see on this object?
kubectl describe pod web | tee evidence/04-describe.txt

# 3. What does the object actually say?
kubectl get pod web -o yaml > evidence/04-yaml.txt

# 4. What is the field called?
kubectl explain pod.spec.containers.volumeMounts | tee evidence/04-explain.txt

The READY column in the first output reads 2/2, not 3/3. Init containers are not counted: they have run and exited, and the column reports containers that are expected to be running now. That single mismatch between what you wrote and what the column shows is a routine source of confusion.

describe is the only one of the four that shows events, and the events are the Pod’s own history in order:

cd "$HOME/k8s-podinspect"

sed -n '/^Events:/,$p' evidence/04-describe.txt

Scheduled, then Pulling/Pulled/Created/Started for seed-content, then the same four for app and sidecar. That ordering is the proof that init containers run to completion first; nothing in kubectl get shows it.

Task 5 — Get the same facts in a form you can script

describe is for humans. Everything above it is for pipelines, and the difference matters as soon as you have more than a handful of Pods.

cd "$HOME/k8s-podinspect"

kubectl get pod web -o jsonpath='{.status.podIP}{"\n"}'
kubectl get pod web -o jsonpath='{range .spec.containers[*]}{.name}{"\t"}{.image}{"\n"}{end}'
kubectl get pod web -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}{.ready}{"\t"}{.restartCount}{"\n"}{end}'

kubectl get pods -o custom-columns='NAME:.metadata.name,NODE:.spec.nodeName,IP:.status.podIP,QOS:.status.qosClass' \
  | tee evidence/05-columns.txt

Note that .status.podIP is singular. Three containers, one address: the Pod is the unit of network identity, and there is nowhere in the object to put a per-container IP because there is not one.

The QOS column is a field you never wrote either. The API server derived it from the resources you set: this Pod has requests and limits that are not equal, so it is Burstable. Had you set no resources at all it would be BestEffort and first in line for eviction under node memory pressure.

Selectors are the other half of scripting:

cd "$HOME/k8s-podinspect"

kubectl get pods -l app=web
kubectl get pods -l 'tier in (frontend,backend)'
kubectl get pods --field-selector=status.phase=Running -o name

Label selectors read labels you control. Field selectors read a short, indexed list the API server maintains per kind — metadata.name, metadata.namespace, status.phase and spec.nodeName for Pods — and asking for anything else is an error rather than a slow query.

Task 6 — Prove what the containers do and do not share

This is the task that makes “a Pod is not a container” concrete.

First, notice what happens when you do not name a container:

$ kubectl logs web --tail=3

Defaulted container "app" out of: app, sidecar, seed-content (init)
/docker-entrypoint.sh: Configuration complete; ready for start up
2026/08/18 09:14:20 [notice] 1#1: start worker processes

It did not fail. It picked the first container, told you in one line above the output, and gave you logs that may have nothing to do with your problem. On a Pod where a mutating webhook has injected a proxy sidecar as the first container, that default is actively misleading. Name the container:

kubectl logs web -c sidecar --tail=3
kubectl logs web -c seed-content

The init container’s log still exists even though the container exited, because the kubelet keeps the log file for the Pod’s lifetime, not the container’s.

Now the network namespace. app listens on port 80; sidecar does not listen on anything. If they share a namespace, the sidecar can reach the app on loopback:

$ kubectl exec web -c sidecar -- wget -qO- --timeout=3 http://127.0.0.1/

served from a volume the init container wrote
pod: web

Two facts fell out of one command. The sidecar reached nginx over loopback, so the two containers share a network namespace. And the page nginx served is the file the init container wrote, so the volume outlived the container that populated it.

The volume is live, not a copy. The sidecar is appending to it right now:

kubectl exec web -c sidecar -- wget -qO- --timeout=3 http://127.0.0.1/heartbeat.log | tail -3
kubectl exec web -c app -- tail -3 /usr/share/nginx/html/heartbeat.log

The same lines, fetched through nginx and read from the filesystem. One emptyDir, two mount paths, two containers.

Now the boundary. Ask the sidecar for something that exists only in the nginx image:

kubectl exec web -c sidecar -- ls /etc/nginx || echo "not visible from the sidecar, as expected"
kubectl exec web -c app -- ls /usr/share/nginx/html

Root filesystems are per-container. Network namespace and mounted volumes are per-Pod. That is the whole model, and it is why a sidecar that needs to read the app’s configuration file needs a shared volume rather than proximity.

Task 7 — Find the same Pod from the node

Everything so far came through the API server. Go to the node and look at the same Pod as the runtime sees it.

cd "$HOME/k8s-podinspect"

kubectl get pod web -o jsonpath='{.spec.nodeName}{"\t"}{.metadata.uid}{"\n"}' \
  | tee evidence/07-pod-identity.txt

SSH to that node, then:

sudo crictl pods --name web

One entry, and its ID is the sandbox. crictl pods lists sandboxes; crictl ps lists application containers and does not show the sandbox at all, which is why a Pod with two containers shows two rows here and three in your manifest:

SANDBOX="$(sudo crictl pods --name web -q | head -1)"
echo "sandbox: $SANDBOX"

sudo crictl ps --pod "$SANDBOX"
sudo crictl ps -a --pod "$SANDBOX"

The second command adds the exited init container. Now find the log files that kubectl logs has been reading through the API server all along. The directory is named from the namespace, the Pod name and the Pod UID you captured above:

LOGDIR="$(sudo find /var/log/pods -maxdepth 1 -type d -name 'podinspect_web_*' | head -1)"
echo "$LOGDIR"

sudo ls -R "$LOGDIR"
sudo tail -3 "$LOGDIR/sidecar/0.log"

The directory name ends in the Pod UID from evidence/07-pod-identity.txt — compare the two. The UID, not the name, is what makes the path unique: delete and recreate a Pod with the same name and you get a different directory, which is how the kubelet keeps two generations of logs apart.

0.log is the current instance. A container that has restarted leaves the previous instance’s file behind, and that file is what kubectl logs --previous returns — which is the only place the output of a crash is kept.

Task 8 — Attach an ephemeral container

kubectl exec requires that the container is running and has the binary you want. Neither is guaranteed: a distroless image has no shell, and a crashing container is not there to exec into. Ephemeral containers solve both by adding a container to a Pod that is already running.

kubectl debug web -it --image=busybox:1.36 --target=app -- sh -c 'wget -qO- --timeout=3 http://127.0.0.1/ ; ls -l /proc/1/'

The ephemeral container shares the Pod’s network namespace, which is why loopback reaches nginx from it. --target=app additionally joins the named container’s process namespace, so the processes you see under /proc belong to nginx rather than to your debug shell.

Look at what this did to the Pod:

cd "$HOME/k8s-podinspect"

kubectl get pod web -o jsonpath='{range .spec.ephemeralContainers[*]}{.name}{"\t"}{.image}{"\n"}{end}' \
  | tee evidence/08-ephemeral.txt
kubectl get pod web -o wide

Task 9 — Delete the Pod, and find out whether anything cares

cd "$HOME/k8s-podinspect"

kubectl get pod web -o jsonpath='{.metadata.ownerReferences}{"\n"}'
kubectl delete pod web --wait=true
kubectl get pods

The ownerReferences output is empty. Nothing owns this Pod, so nothing notices it is gone, and it does not come back. That is the whole reason production workloads are not created this way.

Now the same containers under a controller. manifests/web-deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-managed
  namespace: podinspect
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web-managed
  template:
    metadata:
      labels:
        app: web-managed
    spec:
      containers:
        - name: app
          image: nginx:1.27.2
          ports:
            - containerPort: 80
          resources:
            requests:
              cpu: 50m
              memory: 32Mi
            limits:
              memory: 64Mi

cd "$HOME/k8s-podinspect"

kubectl apply -f manifests/web-deployment.yaml
kubectl rollout status deployment/web-managed --timeout=120s
kubectl get pods -l app=web-managed -o wide | tee evidence/09-before.txt

VICTIM="$(kubectl get pods -l app=web-managed -o jsonpath='{.items[0].metadata.name}')"
kubectl get pod "$VICTIM" -o jsonpath='{.metadata.ownerReferences[0].kind}{"/"}{.metadata.ownerReferences[0].name}{"\n"}'

kubectl delete pod "$VICTIM"
sleep 10
kubectl get pods -l app=web-managed -o wide | tee evidence/09-after.txt

diff evidence/09-before.txt evidence/09-after.txt || true

Read the diff. The replacement has a different name and a different IP, and it is owned by the same ReplicaSet. The ReplicaSet did not restore the Pod you deleted; it observed that it had one replica and wanted two, and created a new one. Nothing about the old Pod survived except the template it was made from.

That is the distinction the whole Deployment model rests on, and it is why anything that needs a stable name or a stable address needs a StatefulSet or a Service rather than a Pod.

Validation

cd "$HOME/k8s-podinspect"

wc -l manifests/web-pod.yaml evidence/03-server-defaulted.yaml
grep -c 'kube-api-access' evidence/03-server-defaulted.yaml
grep -E 'app|sidecar' evidence/05-columns.txt
cat evidence/08-ephemeral.txt
diff evidence/09-before.txt evidence/09-after.txt || true

The lab succeeded when all of the following hold:

• evidence/03-server-defaulted.yaml is substantially longer than manifests/web-pod.yaml, and you can name what restartPolicy, terminationGracePeriodSeconds, the two default tolerations and the kube-api-access volume each do.
• evidence/triage.md has four rows, and each names one thing that command cannot tell you.
• You captured the Defaulted container line from kubectl logs web with no -c, and can say why silently choosing the first container is worse than refusing.
• The sidecar fetched nginx’s page over 127.0.0.1, and that page contained the text the init container wrote.
• kubectl exec web -c sidecar -- ls /etc/nginx failed while kubectl exec web -c app -- ls /usr/share/nginx/html succeeded.
• crictl pods showed one sandbox and crictl ps -a --pod showed three containers for a Pod whose READY column said 2/2.
• evidence/08-ephemeral.txt names the debug container, and you can state what it would take to remove it.
• The diff between 09-before.txt and 09-after.txt shows a changed Pod name and a changed IP, not a restored one.

Expected Outcome

~/k8s-podinspect/
├── manifests/
│   ├── web-pod.yaml
│   └── web-deployment.yaml
└── evidence/
    ├── 00-namespace-before.txt, 00-versions.yaml
    ├── 00-nodes.txt, 00-nodes-stable.txt
    ├── 03-server-defaulted.yaml
    ├── 04-get.txt, 04-describe.txt, 04-yaml.txt, 04-explain.txt
    ├── 05-columns.txt
    ├── 07-pod-identity.txt
    ├── 08-ephemeral.txt
    ├── 09-before.txt, 09-after.txt
    └── triage.md

In the cluster: one namespace holding a two-replica Deployment, and no web Pod — you deleted it in Task 9 and nothing brought it back, which was the point.

Troubleshooting

The Pod stays Init:0/1. The init container has not exited. kubectl logs web -c seed-content and kubectl describe pod web will show whether it is still pulling, still running, or failing. An init container that never exits holds the Pod there indefinitely; there is no timeout unless you set activeDeadlineSeconds.

kubectl wait --for=condition=Ready times out but the Pod looks fine. Check kubectl get pod web -o jsonpath='{.status.containerStatuses[*].ready}'. All containers must report ready for the Pod condition to flip, so a sidecar that is crash-looping keeps a perfectly healthy app container out of every Service.

wget from the sidecar returns nothing and exits 1. nginx has not bound yet, or the init container did not write index.html, in which case nginx returns a 403 for a directory with no index. Check kubectl exec web -c app -- ls -l /usr/share/nginx/html.

kubectl exec web -c app -- tail ... says tail: command not found. You are on a different nginx image than the one this lab pins. nginx:1.27.2 is Debian-based and has a full userland; the -alpine variants have busybox equivalents and the -distroless variants have no shell at all, which is exactly the case ephemeral containers exist for.

crictl pods --name web returns nothing. You are on the wrong node. The node name is in evidence/07-pod-identity.txt; the Pod runs on exactly one.

kubectl debug reports that ephemeral containers are not enabled. The feature has been on by default since 1.25, so on a 1.34 cluster this points at an admission policy rejecting the subresource rather than at the feature gate. kubectl auth can-i create pods/ephemeralcontainers distinguishes the two.

The replacement Pod in Task 9 has the same IP as the deleted one. Possible and not a fault: Pod CIDRs are reused, and a freed address can be handed out again immediately. Check the name, which is always different.

Cleanup

The lab created one namespace, one working directory, and one changed kubectl context.

Step 1. See what is about to be removed, and confirm the namespace was yours:

cd "$HOME/k8s-podinspect"

kubectl get all
cat evidence/00-namespace-before.txt

Step 2. Delete the namespace:

$ kubectl delete namespace podinspect --wait=true

Step 3. Put the kubectl context back, or every later command silently targets a namespace that no longer exists:

kubectl config set-context --current --namespace=default
kubectl config view --minify -o jsonpath='{..namespace}{"\n"}'

Step 4. Confirm the cluster is as you found it:

cd "$HOME/k8s-podinspect"

kubectl get namespace podinspect || echo "namespace gone, as expected"
diff <(kubectl get nodes -o custom-columns='NAME:.metadata.name,VERSION:.status.nodeInfo.kubeletVersion,UNSCHEDULABLE:.spec.unschedulable') \
     evidence/00-nodes-stable.txt \
  && echo "node inventory unchanged"

The diff is against the stable projection rather than -o wide, whose AGE column changes between the two captures and would report a difference on a lab that changed nothing.

Step 5. Keep the deliverables, then remove the working directory.

ls -la "$HOME/k8s-podinspect"

mkdir -p "$HOME/k8s-lab-deliverables/pod-inspection"
cp -a "$HOME/k8s-podinspect/evidence" "$HOME/k8s-podinspect/manifests" \
      "$HOME/k8s-lab-deliverables/pod-inspection/"

rm -rf "$HOME/k8s-podinspect"

Nothing was left on the node: the log directory under /var/log/pods is removed by the kubelet when the Pod’s containers are garbage-collected, and the images pulled into the node’s content store are shared cache rather than lab state.

Production notes

This lab is a read-heavy exercise with two writes in it, and it maps onto a real change window in three specific ways.

The dry-run is the review. In production, --dry-run=server belongs in the pull request, not in the incident. Capturing the server-defaulted object beside the manifest is how a reviewer sees the sidecar a webhook injects, the node selector a policy adds, and the registry an image rewrite substitutes — none of which appear in the file being reviewed.

The Pod is the wrong unit for anything durable. Task 9 is a two-minute demonstration of a rule that costs real outages: a Pod created directly is not replaced when its node is drained, when it is evicted under memory pressure, or when someone deletes it. If the exercise in front of you is “get this running quickly”, the extra six lines of Deployment are cheaper than the incident.

Ephemeral containers need a policy before they need a procedure. Attaching a debug container to a production Pod is an audited, irreversible change to that Pod, and the image you attach runs on a production node with the workload’s network. Decide in advance which images are permitted and who may create pods/ephemeralcontainers, so that at 03:00 the question is which image, not whether.

The honest limit of this lab: everything here is a healthy Pod. Reading a broken one is a different skill with a different order of operations, and it is what Lab 4 is for.

What You Learned

• You write a fraction of the object. The API server stores defaults, admission mutations, a projected token and two tolerations you never asked for, and each has an operational consequence you now own.
• --dry-run=server shows admission; --dry-run=client shows nothing. One goes through the pipeline, the other checks a local schema.
• The READY column counts running containers, not containers. Init containers are absent from it and present in the events.
• kubectl logs picks a container for you and says so quietly. On a Pod with an injected sidecar, the default is the wrong container.
• A Pod shares a network namespace and its volumes; it does not share root filesystems. One IP, one emptyDir, three separate images.
• The sandbox is where the IP lives. crictl pods shows it, crictl ps does not, and its lifetime is why a restarted container keeps its address.
• An ephemeral container cannot be removed. Only deleting the Pod removes it.
• Deleting an unowned Pod is permanent; deleting an owned one is a replacement. Different name, different IP, same template.