kubectl logs, exec, cp, and debug — runtime inspection

Read commands (get, describe, explain) tell you what the controller sees. Runtime commands (logs, exec, cp, debug) tell you what the container sees. They are the on-call engineer’s stethoscope when a Pod is misbehaving in a way the controller cannot see — a bug, a missing config, a stuck process. This lesson covers the runtime-inspection toolkit every production operator carries.

kubectl logs — container logs

kubectl logs <pod> reads the container’s stdout/stderr from the kubelet’s log file (or the runtime’s stdout if the runtime is containerd v1.7+ with the kubelet pipe enabled).

kubectl logs web-7c8                                # first container
kubectl logs web-7c8 -c nginx                      # named container
kubectl logs web-7c8 --previous                    # previous instance
kubectl logs web-7c8 --tail=200                    # last 200 lines
kubectl logs web-7c8 --since=10m                   # last 10 minutes
kubectl logs web-7c8 --since-time=2026-08-16T12:00:00Z
kubectl logs web-7c8 -f                           # follow (tail -f)
kubectl logs web-7c8 --timestamps                  # prepend RFC3339 timestamps
kubectl logs -l app=web --tail=50                  # all pods with label

--previous is critical. When a container restarts, the kubelet retains the previous container’s log file in /var/log/pods/<ns>_<pod>/<container>/previous.log. --previous reads that file. Without it, you only see logs from the current instance.

For multi-container Pods, -c <name> selects the container. If you omit -c, kubectl returns an error if there is more than one container; passing -c is mandatory.

Logging drivers and what you actually get

The kubelet passes --log-driver to the runtime. The default is json-file (writes to disk). With json-file, you get:

• One log file per container per restart
• Stdout and stderr merged
• A 10 MB rotation limit (configurable)
• Logs from previous instances retained in previous.log

If the runtime is journald or fluentd (rare), kubectl logs reads from the journal or talks to the fluentd socket respectively. Most production clusters use json-file and pipe through a node-level log shipper (Fluent Bit, Vector).

kubectl exec — run a command in a container

kubectl exec opens an interactive or one-shot session in a container. It uses the kubelet’s container runtime exec API (which runs runc exec on containerd or nsenter on Docker).

kubectl exec -it web-7c8 -- /bin/bash               # interactive bash
kubectl exec -it web-7c8 -- /bin/sh                 # for distroless images
kubectl exec web-7c8 -- ls /var/log                 # one-shot command
kubectl exec web-7c8 -c nginx -- curl localhost:8080/healthz
kubectl exec -it web-7c8 -c nginx -- /bin/bash      # multi-container
kubectl exec -n kube-system etcd-node-1 -- etcdctl ...   # control plane

-i keeps stdin open, -t allocates a TTY. Together they give you an interactive shell. Without -it, you get a one-shot command.

The command after -- runs inside the container, not on the host. The container’s filesystem, environment, and PID namespace are what you see. To run on the host, SSH to the node or use kubectl debug --profile=sysadmin (see below).

Limitations of exec

• The container must be running. If the container is in Waiting (e.g., CrashLoopBackOff), exec fails. Use kubectl debug instead.
• The container must have a shell or the command you want to run. Distroless images have no shell; you must exec the exact binary you need.
• kubectl exec does not work for initContainers that have completed.
• The command runs as the container’s user. If the container has runAsNonRoot: true, you cannot run as root.

kubectl cp — file transfer

kubectl cp copies files between the local filesystem and a container. It uses tar over the exec channel: kubectl starts a tar process in the container, streams the data over stdin/ stdout.

kubectl cp ./local-file.txt web-7c8:/tmp/remote-file.txt
kubectl cp web-7c8:/var/log/app.log ./app.log
kubectl cp ./config.yaml web-7c8:/etc/app/config.yaml -c app
kubectl cp -n team-a-prod web-7c8:/var/log/audit.log ./audit.log

cp has limitations:

• It uses tar in the container; if the container has no tar (very minimal images), cp fails. Use kubectl exec cat < /file > local-file as a fallback.
• It is not transactional. If the transfer fails partway, you have a partial file in the container. For configuration files, prefer mounting a ConfigMap as a volume update rather than copying.
• It does not preserve ownership or special permissions.

For production config updates, never use kubectl cp as the normal workflow. Use it for ad-hoc extraction (e.g., grabbing a config file to inspect), not as a deployment mechanism.

kubectl debug — ephemeral debug containers

kubectl debug was added in Kubernetes 1.23 to address the “how do I debug a Pod whose main container has no shell, or is in CrashLoopBackOff, or is otherwise undebuggable” problem. It attaches an ephemeral container to a running Pod.

kubectl debug web-7c8 -it --image=busybox --target=app
# attaches an ephemeral busybox container that shares the
# network namespace with the main 'app' container

kubectl debug web-7c8 -it --image=nicolaka/netshoot --copy-to=web-7c8-debug
# copies the Pod spec, swaps the container image, and runs a
# debuggable replica

There are several modes:

• --target=<container> — attach an ephemeral container that shares the network namespace with the named container. This is the standard “debug the network” mode.
• --image=<image> — choose the debug image. Common choices: busybox, nicolaka/netshoot, alpine, ghcr.io/ kubernetes-debug-tools/debug-tools.
• --copy-to=<new-pod> — clone the Pod spec into a new Pod with one container swapped. Useful when the original Pod cannot accept an ephemeral container (e.g., it has its own resource quota).
• --profile=sysadmin (1.30+) — runs the debug container with host namespaces, giving you chroot, mount, and network visibility at the node level. Powerful, requires explicit RBAC.

When to use debug vs exec vs logs

Cross-course references

• The Docker course part XXIX-Docker-Build covers docker exec, docker logs, docker cp — the same set of runtime-inspection tools, before they were lifted to the cluster level.
• The Linux course part XXII-Linux-NetTroubleshoot covers ss, tcpdump, strace — the system-level tools you run inside a container (via exec/debug) when triaging.
• The Observability course part LXXXIX-Kubernetes-Logging discusses the cluster log pipeline; kubectl logs reads the same data the pipeline ships.

Quiz

Production discipline

• Read-only triage first. Logs, then exec, then debug. Each step is more invasive than the last.
• Always pass --previous when a container is restarting. The current logs only show the new instance’s startup; the crash message is in previous.
• Use kubectl debug with --target for network-level inspection. Netshoot + --target is the standard “what does this Pod’s network actually look like” tool.
• Restrict and audit pods/exec. A shell in a production container is a privilege; production RBAC should scope it to specific identities and audit-log every invocation.
• Avoid kubectl cp as a deployment workflow. For configuration changes, mount a ConfigMap or update the image; cp is for one-off extraction during triage.