Image pulls — registry, layers, and the kubelet's role

Image pulls are the cluster’s primary mechanism for loading workload artifacts. The kubelet calls the runtime’s CRI to pull the image; the runtime contacts the registry, downloads the layers, and stores them in the cache. This lesson walks the registry protocol, the ImagePullPolicy, the image-pull secrets, and the operational patterns for diagnosing pull failures.

The image pull flow

The kubelet’s image pull flow:

sequenceDiagram
    autonumber
    participant K as kubelet
    participant R as Runtime
    participant Reg as Registry
    participant Cache as Local cache

    K->>R: PullImage(image, auth)
    R->>Reg: GET /v2/<name>/manifests/<tag>
    Reg-->>R: manifest
    R->>Reg: GET /v2/<name>/blobs/<digest>
    Reg-->>R: layer
    R->>Cache: store layer
    R->>Reg: GET /v2/<name>/blobs/<digest> (next layer)
    Reg-->>R: layer
    R->>Cache: store layer
    R-->>K: image pulled

The kubelet calls the runtime’s PullImage API. The runtime contacts the registry using the registry protocol (HTTP/HTTPS). The runtime downloads the manifest, then the layers. The layers are stored in the local cache.

The runtime’s cache is shared across Pods. The same layer is not stored twice; the runtime uses content-addressable storage (the layer’s digest is the key).

The ImagePullPolicy

The Pod’s spec.containers[].imagePullPolicy controls when the kubelet pulls the image. The three values:

• Always: the kubelet always pulls the image. The runtime checks the cache; if the cached image is not the latest, the runtime pulls.
• IfNotPresent: the kubelet pulls the image only if the cache does not have it. The default for the :latest tag is Always; the default for other tags is IfNotPresent.
• Never: the kubelet never pulls the image. The image must be in the cache; the kubelet fails to start the container if the image is missing.

The default is determined by the image tag. The :latest tag defaults to Always; other tags default to IfNotPresent.

spec:
  containers:
    - name: app
      image: registry.example.com/app:1.0.0
      imagePullPolicy: IfNotPresent

The IfNotPresent policy is the production default. It avoids the network round-trip when the image is already cached. The Always policy is appropriate for :latest and for production rollouts where the cache may be stale.

The image-pull secrets

The kubelet uses the Pod’s imagePullSecrets to authenticate to private registries. The secret is a kubernetes.io/dockerconfigjson or kubernetes.io/ dockercfg Secret.

spec:
  containers:
    - name: app
      image: registry.example.com/app:1.0.0
  imagePullSecrets:
    - name: registry-credentials

The secret is a base64-encoded JSON file containing the registry URL, the username, and the password (or token).

kubectl create secret docker-registry registry-credentials \
  --docker-server=registry.example.com \
  --docker-username=ops \
  --docker-password=secret

The kubelet reads the secret and passes the credentials to the runtime. The runtime uses the credentials to authenticate to the registry.

The production pattern: the secret is created by the cluster’s bootstrap automation; the Pod references the secret by name. The secret’s content is managed by the cluster’s secret-management system (e.g., External Secrets Operator, Vault).

The image-pull failure modes

The image-pull failure modes:

The diagnostic:

# Substitute your own value before running - the Pod that will not pull:
POD=web-7c8d9f4b5-qr2mn

kubectl describe pod "$POD" | grep -A 10 "Events"

Events:
  Type     Reason          Age   From              Message
  ----     ------          ----  ----              -------
  Normal   BackOff         2m    kubelet           Back-off pulling image
  Warning  Failed          2m    kubelet           Error: ImagePullBackOff
  Normal   Pulling         2m    kubelet           Pulling image "registry.example.com/app:1.0.0"
  Warning  Failed          2m    kubelet           Failed to pull image: rpc error: code = Unknown

The ImagePullBackOff is the kubelet’s exponential backoff for retries. The kubelet retries the pull every 5s, then 10s, then 20s, with a maximum of 5 minutes between retries.

The image’s digest

The image’s digest is the content-addressable identifier. The digest is sha256:...; the digest is computed from the image’s content. The same digest always refers to the same content.

spec:
  containers:
    - name: app
      image: registry.example.com/app@sha256:abc123...

The image’s digest is the immutability guarantee. The registry cannot change the digest’s content; the digest refers to a specific image version. The digest is the production-grade way to specify an image.

The image cache size

The runtime’s image cache is the largest directory on the node. The cache size is bounded by the kubelet’s --image-gc-high-threshold (default 85% of the filesystem).

A cluster that pulls large images frequently is a cluster that fills the cache. The fix is to:

• Use --image-gc-high-threshold to lower the threshold.
• Use image-pull automation to remove the images.
• Use a separate filesystem for the image cache.

The cache size is reported by the kubelet’s metrics:

# Substitute your own value before running - the kubelet's address:
NODE_IP=192.0.2.31

curl -k "https://$NODE_IP:10250/metrics" | grep container_fs

The operator should monitor the cache size and alert on the threshold.

The image pull performance

The image pull performance is bounded by the network’s bandwidth and the registry’s latency. A cluster that pulls from a remote registry is a cluster that has network latency in the Pod’s startup.

The production pattern:

• Co-locate the registry with the cluster. A registry in the same region as the cluster reduces the latency.
• Use a pull-through cache. A registry cache (e.g., distribution’s pull-through cache, Harbor’s proxy cache) acts as a local mirror. The kubelet pulls from the cache; the cache pulls from the upstream registry.
• Pre-pull images. A DaemonSet that pre-pulls the cluster’s images on every node ensures the cache is warm when the Pods are scheduled.
• Use the :latest tag carefully. The Always policy ensures the cache is fresh; the cost is the network round-trip.

The image pull secrets in production

The image-pull secrets are typically managed by the cluster’s secret-management system. The pattern:

1. The secret is stored in the secret-management system (Vault, AWS Secrets Manager).
2. The External Secrets Operator (or equivalent) syncs the secret to the cluster.
3. The Pod references the secret by name.
4. The secret is rotated by the secret-management system.

The Pod’s imagePullSecrets is a reference to the secret; the secret’s content is not in the Pod’s spec.

Quiz

Production discipline

• Always tag images by digest. The tag is for humans; the digest is for machines. The production rule is to use the digest for production deploys.
• Use the IfNotPresent policy for production. The policy avoids the network round-trip when the image is cached. The Always policy is appropriate for :latest tags.
• Co-locate the registry with the cluster. A regional registry is faster than a global one. Use a pull-through cache for an extra layer of caching.
• Manage the image-pull secrets with the cluster’s secret-management system. The Pod’s imagePullSecrets is a reference; the secret’s content is managed by the secret-management system.
• Monitor the image cache size. The cache is the largest directory on the node. The kubelet’s metrics expose the cache size; the operator should alert on the threshold.
• Audit the image pull at every release. A new image that is not pullable is a Pod that fails. The audit catches the missing image.