Runtime failure modes — ErrImagePull, CrashLoopBackOff, OOMKilled

The runtime’s failure modes are reflected in the Pod’s status and events. The kubelet surfaces the runtime’s errors as Pod events; the operator reads the events to diagnose the failure. This lesson walks the most common runtime failure modes, the diagnostic pattern for each, and the operational moves.

The kubelet’s view of runtime failures

The kubelet’s per-Pod goroutine surfaces the runtime’s errors as Pod events. The events are recorded in the Pod’s Status.Conditions and the cluster’s events.

# Substitute your own value before running:
POD=app-7d9f4c6b58-nkq2r

kubectl describe pod "$POD" | grep -A 20 "Events"

The events are the primary diagnostic. The events include the failure type, the timestamp, and the message.

ErrImagePull

A Pod that is failing to pull the image has the ErrImagePull event. The kubelet’s PullImage call returned an error.

Events:
  Type     Reason          Age   From     Message
  ----     ------          ----  ----     -------
  Warning  Failed          2m    kubelet  Failed to pull image "registry.example.com/app:1.0.0":
                                       rpc error: code = Unknown desc = Error response from daemon:
                                       pull access denied for registry.example.com/app

The diagnostic:

# Substitute your own value before running:
POD=app-7d9f4c6b58-nkq2r

kubectl describe pod "$POD" | grep -A 10 "Events"

The message identifies the failure. The common causes:

• Registry unreachable: the network or DNS is blocking the registry. The fix is to check the network.
• Image not found: the tag or digest is wrong. The fix is to correct the tag.
• Authentication failed: the imagePullSecret is missing or wrong. The fix is to provide the correct secret.
• Timeout: the registry is slow or the network is slow. The fix is to investigate the latency.

The kubelet retries the pull with exponential backoff. The ImagePullBackOff event indicates the retry cycle.

ImagePullBackOff

A Pod that has failed to pull the image multiple times has the ImagePullBackOff event. The kubelet retries every 5s, 10s, 20s, …, up to 5 minutes between retries.

Events:
  Type     Reason          Age   From     Message
  ----     ------          ----  ----     -------
  Warning  BackOff         2m    kubelet  Back-off pulling image "registry.example.com/app:1.0.0"
  Warning  Failed          2m    kubelet  Error: ImagePullBackOff

The ImagePullBackOff is the kubelet’s backoff. The underlying error is the same as ErrImagePull; the backoff is just the timing.

The fix is to address the underlying error. The kubelet stops the backoff when the pull succeeds.

ContainerCreating

A Pod that is stuck in ContainerCreating is waiting for the kubelet to create the container. The Pod is still in the kubelet’s sync loop; the container has not been started.

Events:
  Type     Reason          Age   From     Message
  ----     ------          ----  ----     -------
  Warning  Failed          2m    kubelet  Failed to create container
  Warning  Failed          2m    kubelet  Error: container create failed: container_linux.go: ...

The diagnostic:

# Substitute your own value before running:
POD=app-7d9f4c6b58-nkq2r

kubectl describe pod "$POD" | grep -A 10 "Events"

The message identifies the failure. The common causes:

• Volume mount failed: the CSI driver is failing. The fix is to investigate the CSI driver.
• Network setup failed: the CNI plugin is failing. The fix is to investigate the CNI plugin.
• Image pull failed: the runtime could not pull the image. The fix is to address the pull failure.
• Resource limits exceeded: the container’s resource limits exceed the node’s capacity. The fix is to reduce the limits or scale the node.

RunContainerError

A Pod that has been created but failed to start has the RunContainerError event. The kubelet’s StartContainer call returned an error.

Events:
  Type     Reason          Age   From     Message
  ----     ------          ----  ----     -------
  Warning  Failed          2m    kubelet  Failed to start container
  Warning  Failed          2m    kubelet  Error: container_linux.go:345: starting container process caused
                                       "exec: \"/bin/sh\": executable file not found in $PATH"

The diagnostic:

# Substitute your own value before running:
POD=app-7d9f4c6b58-nkq2r

kubectl describe pod "$POD" | grep -A 10 "Events"

The message identifies the failure. The common causes:

• Command not found: the container’s command is wrong. The fix is to correct the command.
• Entrypoint not executable: the container’s entrypoint is not executable. The fix is to correct the entrypoint.
• Volume mount failed: the container’s volume mount failed during the start. The fix is to investigate the volume.

CrashLoopBackOff

A Pod that is repeatedly crashing has the CrashLoopBackOff event. The kubelet restarts the container; the container crashes; the kubelet waits with exponential backoff.

Events:
  Type     Reason          Age   From     Message
  ----     ------          ----  ----     -------
  Warning  BackOff         2m    kubelet  Back-off restarting failed container
  Normal   Pulled          2m    kubelet  Successfully pulled image
  Normal   Created         2m    kubelet  Created container app
  Warning  Failed          2m    kubelet  Error: container exited with code 1

The diagnostic:

# Substitute your own value before running:
POD=app-7d9f4c6b58-nkq2r

kubectl logs "$POD" --previous

The previous container’s logs show the reason for the crash. The common causes:

• Application error: the application exited with a non-zero status. The fix is to investigate the application’s logs.
• Configuration error: the application cannot read the configuration. The fix is to provide the correct configuration.
• Database connection failed: the application cannot connect to the database. The fix is to investigate the database.
• Out of memory: the application was OOMKilled. The fix is to increase the memory limit.

OOMKilled

A container that is killed by the kernel for exceeding its memory limit has the OOMKilled reason. The kubelet’s last termination was OOMKilled.

# Substitute your own value before running:
POD=app-7d9f4c6b58-nkq2r

kubectl get pod "$POD" -o jsonpath='{.status.containerStatuses[].lastState.terminated.reason}'

OOMKilled

The diagnostic:

# Substitute your own value before running:
POD=app-7d9f4c6b58-nkq2r

kubectl describe pod "$POD" | grep -A 5 "Last State"

Last State:     Terminated
Reason:         OOMKilled
Exit Code:      137
Started:        ...
Finished:       ...

The exit code 137 is the standard for OOMKilled (128 + 9, where 9 is SIGKILL). The fix is to increase the memory limit or to investigate the application’s memory usage.

ImageInspectError

A Pod that is failing because the runtime cannot inspect the image has the ImageInspectError event. The runtime can pull the image but cannot read its metadata.

Events:
  Type     Reason          Age   From     Message
  ----     ------          ----  ----     -------
  Warning  Failed          2m    kubelet  Failed to inspect image "registry.example.com/app:1.0.0":
                                       rpc error: code = Unknown

The cause is usually a corrupted image or a runtime bug. The fix is to restart the runtime or to use a different runtime.

The diagnostic workflow

The runtime’s failure modes share a common diagnostic workflow:

flowchart TD
    A[Pod failing] --> B[kubectl describe pod]
    B --> C{Pod in ContainerCreating?}
    C -->|Yes| D[Read events<br/>ErrImagePull, ContainerCreating]
    C -->|No| E{Pod in CrashLoopBackOff?}
    E -->|Yes| F[kubectl logs --previous]
    E -->|No| G{Container OOMKilled?}
    G -->|Yes| H[Increase memory limit]
    G -->|No| I[Read kubelet logs]
    I --> J[Read runtime logs]

The workflow:

1. Run kubectl describe pod. Read the events.
2. Identify the failure mode. The event name is the primary signal.
3. Run kubectl logs --previous. The previous container’s logs show the application’s failure.
4. Run journalctl -u kubelet. The kubelet’s logs show the CRI errors.
5. Run journalctl -u containerd. The runtime’s logs show the runtime’s errors.

The runtime’s metrics

The kubelet exposes the runtime’s metrics on its metrics endpoint. The relevant metrics:

• kubelet_containers_per_max_pod_count
• kubelet_running_containers
• kubelet_running_pods
• kubelet_containers_failed_total
• runtime_container_status_phase

The operator should monitor the metrics and alert on the failure rate.

Quiz

Production discipline

• Read the Pod’s events first. The events are the primary diagnostic. The message identifies the failure.
• Distinguish application failures from runtime failures. A CrashLoopBackOff is an application failure; an ErrImagePull is a runtime failure.
• OOMKilled is a memory limit, not a memory amount. The fix is to increase the limit or fix the leak.
• CrashLoopBackOff is a symptom. The fix is to read the application’s logs and address the failure.
• Use kubectl logs --previous to debug. The previous container’s logs are the only way to see why the application crashed.
• Monitor the kubelet’s runtime metrics. The metrics expose the runtime’s failure rate. The operator should alert on the threshold.
• Audit the runtime at every release. A new application version that is not compatible with the runtime is a Pod that fails. The audit catches the incompatibility.