Immutable ConfigMaps and Secrets — preventing runtime drift

ConfigMaps and Secrets support an immutable: true flag that prevents the API server from accepting any changes after creation. Once set, the data is fixed for the lifetime of the object; updates are rejected with an error. This lesson covers what immutability guarantees, why kubelet benefits from it, and when it is the right design choice.

What immutable means

apiVersion: v1
kind: ConfigMap
metadata:
  name: web-config
data:
  log_level: info
immutable: true

With immutable: true:

• The API server rejects any attempt to modify data or binaryData. The response is a 422 with a message: “the object is immutable; spec.immutable is set to true.”
• The object can still be deleted and recreated (with the same name).
• The object’s metadata (labels, annotations) is mutable — immutability applies to the data only.

flowchart TB
    A[kubectl apply -f new.yaml] --> B{immutable?}
    B -->|true| C["API server:<br/>422 Unprocessable Entity"]
    B -->|false| D[Object updated]
    C --> E["Operator must<br/>delete + recreate"]

The kubelet benefit

The kubelet mounts ConfigMaps and Secrets into Pods. With mutable objects, the kubelet watches them and re-syncs the mount periodically (within syncPeriod). With immutable objects, the kubelet knows the data will not change; it mounts the volume once at Pod start and does not watch.

flowchart TB
    A[Pod starts] --> B{ConfigMap immutable?}
    B -->|yes| C["Kubelet reads once<br/>mounts volume"]
    B -->|no| D["Kubelet watches<br/>re-syncs every syncPeriod"]
    C --> E[No syncPeriod traffic]
    D --> F["ConfigMap watch traffic<br/>+ sync traffic"]

For clusters with thousands of Pods consuming the same ConfigMap, the watch traffic and the periodic sync add up. Immutable objects reduce this to a single read at Pod start. The kubelet’s startup time is faster; the API server’s watch traffic is lower.

When immutable is correct

The right use case: configuration that should not change at runtime. Examples:

• Release-baked config. The configuration is shipped with the release; a change requires a new release and a rollout. The ConfigMap is a snapshot of the release’s config.
• TLS certificates issued by an external CA. The cert is rotated via a new Secret object (same name, deleted and recreated); the existing Pods use the old cert until restarted.
• Audit-log config. The audit log destinations are baked at release; changing them at runtime would split the audit trail.
• Compliance config. Some compliance regimes require configuration to be versioned and unchanging; immutable ConfigMaps enforce this at the cluster level.

When immutable is wrong

The wrong use case: configuration that is expected to roll. Examples:

• Feature flags toggled weekly. An immutable ConfigMap requires deletion + recreation for every flag toggle; the Pod must be restarted.
• Rate limits and quotas. A config change that should take effect immediately (e.g., a tighter rate limit during an incident) does not work with immutable objects.
• External service URLs. A database failover that requires URL change requires the old Pods to terminate.
• Any config consumed as env vars. Env vars are immutable in the running Pod anyway; immutable ConfigMaps add no value here.

Migration from mutable to immutable

A cluster has many mutable ConfigMaps. The migration to immutable:

flowchart TB
    A["Current state:<br/>mutable ConfigMap"] --> B[Identify consumers]
    B --> C{Can consumers handle<br/>delete + recreate?}
    C -->|yes| D["Set immutable: true<br/>via patch"]
    C -->|no| E[Stay mutable]
    D --> F["Roll consumers<br/>if env-var based"]

# Patch to immutable
kubectl patch configmap web-config -n prod --type=merge \
  -p '{"immutable":true}'

# Reverse (requires recreation; cannot be reversed on the same object)
kubectl delete configmap web-config -n prod
kubectl create configmap web-config -n prod --from-file=...

The patch to immutable: true is itself a write. Existing Pods do not see a config change; the data has not changed. The immutability applies to future writes.

To make a previously-immutable object mutable again, you must delete and recreate it.

Combination with hash annotations

Immutable ConfigMaps work well with the hash annotation pattern for Deployments:

spec:
  template:
    metadata:
      annotations:
        config-checksum/web: <sha256 of web-config>

A change to web-config requires:

1. Delete the old immutable ConfigMap.
2. Create the new immutable ConfigMap with new data.
3. The hash annotation changes; the Deployment rolls.

The combination is deterministic: every change produces a new object, every change triggers a rollout.

Performance: kubelet startup

For Pods that mount many ConfigMaps and Secrets, the immutable flag reduces kubelet startup latency:

flowchart LR
    A[Pod scheduled] --> B[Kubelet starts Pod]
    B --> C{Volumes mount}
    C --> D["ConfigMap volume:<br/>read once"]
    C --> E["Secret volume:<br/>read once"]
    C --> F["EmptyDir:<br/>instant"]
    C --> G["PVC:<br/>CSI call"]
    D -->|immutable| H[No watch]
    E -->|immutable| H
    D -->|mutable| I["Watch starts<br/>syncPeriod loop"]
    E -->|mutable| I

For a Pod that mounts 10 ConfigMaps and 5 Secrets, the immutable flag eliminates 15 watch loops. The kubelet’s memory footprint is smaller; the API server’s watch load is lower.

Quiz

Production discipline

• Default to immutable for release-baked config. A ConfigMap that does not change at runtime should be immutable.
• Default to mutable for runtime-tunable config. A ConfigMap that is expected to roll (feature flags, rate limits, URLs) should remain mutable.
• Audit immutable ConfigMaps. A query for configmaps with immutable=true and metadata.age < 1d catches recent promotions; a query for configmaps with immutable=false and metadata.age > 30d surfaces candidates for immutability.
• Document the immutability contract. A change to an immutable ConfigMap requires delete + recreate. The runbook must spell this out.
• Validate the migration. A patch to immutable: true is a one-way door (in terms of the same object). Test the migration on a single ConfigMap first.

Immutability is a discipline, not a default. Operators who understand when it applies have ConfigMaps that align with the workload’s actual change cadence.