Container states — Waiting, Running, Terminated

status.phase is the Pod-level summary; containerStatuses[*].state is the per-container truth. Every triage starts at the phase, drills into container state, and then reads the events. This lesson covers the three container states, the fields inside each, and the diagnostic patterns they enable.

The three container states

status:
  containerStatuses:
  - name: nginx
    state:
      running:
        startedAt: "2026-08-16T12:00:00Z"
    ready: true
    restartCount: 0
    image: nginx:1.27.2
    imageID: docker.io/library/nginx@sha256:abc123
    containerID: containerd://...
    started: true

A container is in one of three states at any moment:

• Waiting: the container is not yet running. It is in some pre-start phase (image pull, sandbox creation, volume mount) or is in a backoff loop after repeated crashes. The state.waiting.reason field tells you why.
• Running: the container’s process is active. The state.running.startedAt field tells you when it started.
• Terminated: the container’s process has exited. The state.terminated field carries the exit code, reason, finishedAt timestamp.

stateDiagram-v2
    [*] --> Waiting: created, not yet started
    Waiting --> Running: process started
    Waiting --> Waiting: backoff loop (CrashLoopBackOff)
    Running --> Waiting: process exited, restart triggered
    Running --> Terminated: process exited, no restart
    Terminated --> Waiting: restart triggered
    Terminated --> [*]: Pod terminated (restartPolicy: Never)

Waiting — common reasons

state:
  waiting:
    reason: CrashLoopBackOff
    message: "back-off 5m0s restarting failed container=nginx pod=web-7c8"

The reason field is the diagnostic key. Common reasons:

ImagePullBackOff vs ErrImagePull

ErrImagePull is the first failure. The kubelet records the reason and retries. After several failures, it transitions to ImagePullBackOff and applies exponential backoff.

The diagnosis:

kubectl describe pod web-7c8
# Events:
#   ... reason: Failed   ... message: Failed to pull image "registry.example.com/web:v1.2.3":
#                            rpc error: ... tag not found

Common causes:

• Tag does not exist (tag not found, manifest unknown).
• Auth failure (unauthorized, access denied).
• Network unreachable (dial tcp: i/o timeout).
• ImagePullSecret missing or invalid (could not find secret).

CrashLoopBackOff

The container starts, exits non-zero, restarts, exits again. The kubelet applies exponential backoff: 10s, 20s, 40s, 80s, 160s, capped at 5 minutes.

The diagnosis:

kubectl logs web-7c8 --previous
kubectl describe pod web-7c8

Common causes:

• Application error (config missing, database unreachable, panic on startup).
• OOMKilled — the container exceeded its memory limit; the process is killed with SIGKILL. Exit code 137. See Part XII for resource limits.
• Liveness probe failure — repeated failures trigger container restart; the liveness probe may be too strict.
• Readiness probe vs liveness probe: a failing readiness probe does NOT restart the container; only liveness does.

Running — the healthy state

state:
  running:
    startedAt: "2026-08-16T12:00:00Z"

The container’s process is active. startedAt is the timestamp the kubelet first observed the process running.

Even in Running, a container may not be ready. The ready field on the container status indicates whether the readiness probe has succeeded. A Running-but-not-ready container is excluded from the Service’s Endpoints list.

kubectl get pod web-7c8 -o jsonpath='{.status.containerStatuses[*].ready}'
# false false

This means both containers are running but neither is ready (probes failing). The Pod is technically Running; traffic should not be sent.

Terminated — exit codes and reasons

state:
  terminated:
    exitCode: 0
    reason: Completed
    startedAt: "2026-08-16T12:00:00Z"
    finishedAt: "2026-08-16T12:05:00Z"
    containerID: containerd://...

Common terminated.reason values:

The exit code carries the kernel signal information for signal-killed processes:

• 137: 128 + 9 (SIGKILL). OOMKill or forced termination.
• 143: 128 + 15 (SIGTERM). Graceful termination, e.g., drain.
• 129: 128 + 1 (SIGHUP). Rare; usually the runtime’s hangup signal.

flowchart LR
    Exit[Container exits] --> Code{Exit code?}
    Code -- 0 --> Success[Completed]
    Code -- 1-127 --> Error[Error: app fault]
    Code -- 137 --> OOM[OOMKilled]
    Code -- 143 --> Term[SIGTERM/graceful]

Reading container statuses

The full triage command:

kubectl get pod web-7c8 -o jsonpath='
{.status.containerStatuses[*].name} = {.status.containerStatuses[*].state}{"\n"}
ready: {.status.containerStatuses[*].ready}
restartCount: {.status.containerStatuses[*].restartCount}
image: {.status.containerStatuses[*].image}
' | jq

For each container, output:

{
  "name": "nginx",
  "state": {
    "waiting": {
      "reason": "CrashLoopBackOff"
    }
  },
  "ready": false,
  "restartCount": 3,
  "image": "nginx:1.27.2"
}

The combination of state + ready + restartCount tells the story: container is in CrashLoopBackOff, has restarted 3 times, is not ready. The diagnosis starts with kubectl logs --previous and kubectl describe.

Cross-course references

• The Linux course part VI-Linux-Processes covers POSIX exit codes and signal handling; container states map onto the same concepts.
• The Docker course part XXX-Docker-Lifecycle covers container states (created, running, exited, dead, paused); Kubernetes container states are the same with richer reasons.
• The Ansible course part XXXV-Ansible-Scripting covers service health states; container states are the cluster-level equivalent.

Quiz

Production discipline

• Read containerStatuses[*].state as the second triage step. Phase is the first signal; container state is the ground truth.
• Empty --previous logs + CrashLoopBackOff = OOMKill. The cgroup killed the process before it could log.
• Distinguish OOMKilled from Error and Completed. The terminated.reason field is the diagnostic key.
• Set memory limits on every container. A Pod without a memory limit can be OOMKilled at the node level (not QoS-aware).
• Monitor container restart count. A high restartCount is the early signal of an unstable workload.