Registry cache — pull-through proxy and local kubelet cache

Registry caching is essential for production performance and resilience. This lesson walks pull-through proxy caches, kubelet local cache, and the discipline.

The registry cache architecture

flowchart LR
    A[kubelet] -->|pull| B[Pull-through proxy cache]
    B -->|cache hit| C[Return cached image]
    B -->|cache miss| D[Upstream registry]
    D -->|return image| B
    B -->|cache and return| A
    E[Upstream down] -->|cache hit serves| A

The cache architecture:

• Pull-through proxy. A registry that proxies pulls from an upstream; caches the result.
• kubelet local cache. Per-node image storage; caches pulled images for reuse.

Both layers are essential. The proxy cache reduces upstream load; the kubelet cache reduces pull latency.

distribution pull-through proxy

# distribution config.yaml
version: 0.1
log:
  level: info
storage:
  filesystem:
    rootdirectory: /var/lib/registry
proxy:
  remoteurl: https://registry-1.docker.io
  username: [username]
  password: [password]

The distribution registry configuration:

• storage. Where images are stored (local filesystem, S3, etc.).
• proxy.remoteurl. The upstream registry URL.
• proxy.username/password. Credentials for the upstream (Docker Hub, private registry).

When a Pod pulls an image, distribution checks its local cache; if missing, it pulls from the upstream, caches, and returns to the Pod.

Harbor as proxy cache

# Harbor proxy cache project
harbor-cli project create \
  --name dockerhub-cache \
  --public false \
  --proxy-cache \
  --proxy-url https://registry-1.docker.io

Harbor’s proxy cache:

• A project is configured as a proxy cache.
• The project’s URL is harbor.example.com/dockerhub-cache/library/<image>.
• Pods pull from this URL; Harbor proxies to Docker Hub and caches.

# Pod spec
containers:
  - name: app
    image: harbor.example.com/dockerhub-cache/library/nginx:1.25

The Pod pulls nginx:1.25 from Harbor’s proxy cache. Harbor pulls from Docker Hub on first request, caches, and serves subsequent requests.

The kubelet local cache

flowchart LR
    A[kubelet] -->|pull image| B["Local cache: /var/lib/containerd"]
    B --> C{Image present?}
    C -->|Yes| D[Use cached image]
    C -->|No| E[Pull from registry]
    E --> B
    F[Disk pressure] -->|GC at 80%| B

The kubelet local cache:

• Storage. /var/lib/containerd (containerd), /var/lib/docker (docker).
• Hit. The kubelet uses the cached image; no pull.
• Miss. The kubelet pulls from the registry.
• GC. At 80% disk pressure, the kubelet garbage collects least-recently-used images.

The configuration

# /etc/containerd/config.toml (containerd)
[plugins."io.containerd.grpc.v1.cri"]
  sandbox_image = "k8s.gcr.io/pause:3.9"

[plugins."io.containerd.grpc.v1.cri".containerd]
  [plugins."io.containerd.grpc.v1.cri".containerd.mirrors]
    [plugins."io.containerd.grpc.v1.cri".containerd.mirrors."docker.io"]
      endpoint = ["https://harbor.example.com/dockerhub-cache"]
    [plugins."io.containerd.grpc.v1.cri".containerd.mirrors."gcr.io"]
      endpoint = ["https://gcr.example.com"]

The containerd configuration:

• mirrors. Map upstream registries to mirror endpoints.
• The kubelet (via containerd) tries the mirror first; falls back to the upstream if the mirror fails.

The garbage collection

# kubelet configuration
--image-gc-high-threshold=85
--image-gc-low-threshold=80
--minimum-image-ttl-duration=2m

The kubelet GC:

• high threshold (85%). When disk usage exceeds this, GC starts.
• low threshold (80%). GC stops when disk usage drops below this.
• minimum-image-ttl-duration (2m). Images must be at least 2 minutes old before they are eligible for GC.

Quiz

The operational discipline

Registry caching in production rests on five non-negotiable elements:

• Pull-through proxy cache. Always; for both performance and resilience.
• kubelet local cache. Default; tune GC thresholds.
• Multi-registry mirrors. Configure multiple endpoints.
• Monitor cache hit rate. High hit rate means less upstream load.
• Document the cache topology. In the runbook.

Registry caching is production performance and resilience. The discipline is pull-through cache with monitoring.