Containers, images and OCI — what the kubelet actually pulls

Kubernetes is an orchestrator of containers. It does not run containers itself — kubelet delegates to a runtime (containerd by default) via the Container Runtime Interface. The runtime’s job is to pull, unpack, and start an image that conforms to the OCI Image Specification. This lesson covers the pieces the runtime hands to the kernel and the ways the abstraction can break in production.

What an OCI image is

The OCI Image Spec defines an image as three things:

1. A manifest (index.json for multi-platform, manifest.json for a single-platform image) listing the layers and config
2. A config (<digest>.json) describing the image’s runtime parameters: entrypoint, environment, working directory, exposed ports, volumes, OS, architecture
3. Layers (<digest> tarballs) that are overlaid to produce the root filesystem

The whole image is content-addressed by digest (a SHA-256 over each component). The image has no inherent name — it has a manifest digest, and tags are mutable pointers to a digest.

flowchart TB
    R[Registry] -->|pull by tag| M[Manifest<br/>sha256:abc123]
    M --> C[Config<br/>sha256:def456]
    M --> L1[Layer 0<br/>sha256:ghi789]
    M --> L2[Layer 1<br/>sha256:jkl012]
    M --> L3[Layer N<br/>sha256:mno345]
    C --> FS[Unpacked root filesystem]
    L1 --> FS
    L2 --> FS
    L3 --> FS
    FS --> Runc[runc starts container]

When kubelet asks containerd to pull nginx:1.27.1:

1. Containerd talks to the registry, fetches the manifest for the platform matching the node’s architecture.
2. It compares the manifest digest with what is already on disk.
3. For each layer digest in the manifest, it fetches only the layers it does not have. Layers are content-addressed, so they are deduped across images.
4. It unpacks the layers into a snapshotter’s directory (typically under /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/).
5. It starts the runtime (runc) with the unpacked root and the image’s config as the OCI runtime spec.

Tags are mutable, digests are not

This is the single most important supply-chain concept in production:

• nginx:1.27.1 is a tag. The registry owner can re-point this tag at a different image at any time. (Container registries usually forbid overwriting a tag’s digest silently, but it has happened and continues to happen.)
• nginx@sha256:abc123... is a digest reference. It points at one specific manifest and cannot be re-pointed without changing the digest.

Production Kubernetes should anchor on digests:

spec:
  containers:
  - name: web
    image: nginx@sha256:abc123def456...
    # NOT image: nginx:1.27.1 — that tag could change under you

kubectl set image and kustomize will rewrite digests back to tags if the manifest is mutated carelessly. Production change management requires digest pinning in version control and admission control that rejects image: <name>:<mutable-tag> patterns.

The runtime contract: CRI

Kubelet does not talk to runc directly. It talks to the Container Runtime Interface (CRI), a gRPC API. The default implementation is containerd; alternatives include CRI-O.

sequenceDiagram
    autonumber
    participant K as kubelet
    participant C as containerd (CRI)
    participant Reg as Registry
    participant S as Snapshotter
    participant R as runc

    K->>C: PullImage(image=nginx:1.27.1, sandbox=...)
    C->>Reg: GET /v2/nginx/manifests/1.27.1
    Reg-->>C: manifest + config + layer digests
    C->>Reg: GET /v2/nginx/blobs/<layer-digest>
    Reg-->>C: layer tarball
    C->>S: Unpack layers into snapshot
    S-->>C: snapshot path
    K->>C: CreateContainer(sandbox, container, mounts)
    C->>R: runc create (OCI runtime spec)
    K->>C: StartContainer(containerID)
    C->>R: runc start
    R-->>C: container PID
    C-->>K: status updates

The CRI surface kubelet uses:

• PullImage — fetches and unpacks an image
• CreateSandbox — creates the pause namespace holder
• CreateContainer / StartContainer / StopContainer / RemoveContainer — container lifecycle
• ContainerStatus — runtime-side status
• UpdateRuntimeConfig — applies Pod-level resources to the runtime

In production, the CRI error surface is where many kubelet incidents originate. The Pod event Failed with reason ErrImagePull or ImagePullBackOff is a CRI-layer error.

How kubelet authenticates to a registry

If the registry is anonymous (docker.io/library/*), kubelet needs no credentials. For private registries, kubelet uses credentials from imagePullSecrets on the ServiceAccount, the node’s ~/.docker/config.json, or (in 1.34+) the kubelet’s credential provider plugins.

kubectl get secret regcred -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d

{
  "auths": {
    "registry.example.com": {
      "username": "ci-bot",
      "password": "REDACTED",
      "auth": "BASE64(user:pass)"
    }
  }
}

Image pull policy: when kubelet actually pulls

The Pod’s imagePullPolicy controls whether kubelet trusts the cache:

Production defaults:

• Always for any image referenced by mutable tag — without it, kubelet uses the cached layer set even if the tag has changed
• IfNotPresent for any image referenced by digest — the cache check is by digest, so a cached image is authoritative
• Never only in air-gapped clusters where the registry is deliberately unreachable and the cache is provisioned out of band

Kubelet’s defaults:

• imagePullPolicy: Always if the tag is latest or absent
• imagePullPolicy: IfNotPresent otherwise

Where image operations fail

Every ImagePullBackOff in production comes from one of:

1. Registry unreachable. DNS, routing, firewall, TLS — all the usual suspects. Check from the node:

   crictl pull nginx:1.27.1

   This uses the same runtime kubelet uses; if it fails here, it fails for kubelet.
2. Authentication failure. Wrong secret, expired token, registry rejecting the ServiceAccount token. Look for unauthorized in kubelet logs:

   journalctl -u kubelet --since "5 min ago" | grep -i pull

3. Layer corrupt or partial. Disk full mid-pull, network blip during layer fetch. The snapshotter will retry, but if the cache is left in a partial state, crictl rmpi (remove pod images) clears it.
4. Architecture mismatch. Image built for amd64, node is arm64 (Graviton, Apple Silicon in dev clusters). The manifest does not advertise a matching platform entry; pull fails with no manifest found.
5. Registry rate limit. Docker Hub throttles anonymous pulls. Production clusters should mirror or use a paid Docker Hub account.

Image lifecycle in the runtime cache

Containerd keeps pulled images in its content store. Kubelet garbage-collects images it has not used in --image-gc-high-threshold percent of disk (default 85%) and down to --image-gc-low-threshold (default 80%). When the runtime’s disk crosses the high threshold, kubelet deletes the oldest unused images until it reaches the low threshold.

This means a node can lose its image cache if the disk pressure is high. Production nodes should:

• Have enough ephemeral storage for the full image set plus headroom
• Be monitored on container_image_size_bytes and container_fs_inodes_free
• Have an alert when kubelet_image_garbage_collected is non-zero (kubelet emits an event when it GCs an image)

How to inspect what kubelet has cached

From any node:

crictl images

Output shows image ID (digest), tag, size, and the repository the image was pulled from. This is the source of truth for “is this image present on this node?” — not docker images, which queries a different store if dockerd is also installed.

To list layers per image:

# Image ID from the `crictl images` output above:
IMAGE_ID=sha256:c2c45d506085d300b72a6d4b10e3dce104228080a2cf095fc38333afe0f70a1e

crictl inspecti "$IMAGE_ID"

Output includes info.imageSpec.config, info.imageSpec.rootfs.diffIds, and info.imageSpec.os/info.imageSpec.architecture.

Multi-platform images and node selection

A multi-platform image (often a “fat manifest” or OCI image index) lists platform-specific manifests. Kubelet, via containerd, picks the manifest that matches the node’s kubernetes.io/arch and kubernetes.io/os labels.

For production:

• Multi-arch images built for both linux/amd64 and linux/arm64 are the norm for clusters with mixed architecture.
• A Pod that explicitly sets nodeSelector: kubernetes.io/arch: arm64 is the standard way to land a workload on Graviton nodes.
• Mismatches fail at pull with no obvious error; the Pod event says “no matching manifest for linux/arm64” or similar.

Cross-course references

• The Docker course parts XXVIII-Docker-Images and XXIX-Docker-Build cover image layers, manifests, and multi-stage builds — the same model Kubernetes inherits.
• The Docker course part XXXVII-Docker-Registries covers private registries, authentication, retention, and backup — the same surface kubelet integrates with.
• The Linux course part LXXVIII-Linux-Containers covers the kernel primitives the runtime composes into the container/Pod abstraction.
• The Observability course part IX-Observability-Exporters covers cAdvisor and node_exporter metrics that surface image pull times and runtime cache state into the monitoring stack.

Quiz

Production discipline

• Pin digests in production manifests; reserve tags for humans and CI pipelines that rewrite digests back to tags.
• Set imagePullPolicy: Always for any image referenced by a mutable tag; IfNotPresent is acceptable for digest-pinned images.
• Provision node ephemeral storage with headroom for the full image set; image GC at 85% is too late if your cache is the deployment fast-path.
• Monitor ImagePullBackOff events and alert on sustained failures; most are auth or scope, not network.
• Run a private registry (or a paid Docker Hub plan) with audit logs for tag mutations; image supply-chain incidents are detected at the registry, not in the cluster.