Lab 7: ConfigMaps and Secrets in a running workload

Objective

By the end of this lab you will be able to answer, for any Pod in any cluster, the question that generates more wasted incident time than any other configuration question: I changed the ConfigMap — has the running container seen it yet, and if not, when will it?

You will answer it with command output rather than belief. You will have a container consuming one ConfigMap twice over — as environment variables and as files — plus a Secret the same two ways, and you will have watched the file surface and the environment surface diverge in real time on a clock you started yourself.

Architecture

One namespace, one ConfigMap, one Secret, two Deployments. The second Deployment exists only to demonstrate one thing the first cannot.

flowchart LR
    CM["ConfigMap<br/>web-config"] --> E["envFrom<br/>whole object"]
    CM --> V["volume mount<br/>/etc/web"]
    S["Secret<br/>db-credentials"] --> SE["env via<br/>secretKeyRef"]
    S --> SV["volume mount<br/>/etc/db"]
    E --> P["Deployment web<br/>busybox container"]
    V --> P
    SE --> P
    SV --> P
    CM --> SP["subPath mount<br/>/etc/web/app.conf"]
    SP --> P2["Deployment web-subpath"]

The point of the shape is that both Deployments read the same ConfigMap object. Every difference you observe between them is a difference in how the kubelet projects it, not a difference in the data.

Requirements

A cluster you are willing to create and destroy a namespace in, with kubectl configured against it. Kubernetes 1.34.x was the reference version for this course; nothing here depends on a version newer than 1.21.
One schedulable node is enough. No storage class, no ingress, no metrics server.
The ability to pull busybox:1.36 from Docker Hub, or a mirror of it. It is used because it has a shell, env, cat and ls, which is the whole requirement — the lab is about the kubelet, not about the workload.
Roughly 8 MiB of memory per replica and no CPU to speak of. This lab cannot destabilise a node.
No out-of-band access is needed. Nothing here touches networking, the kubelet configuration, or the node.

Scenario

At 02:40 an on-call engineer raises the log level of the checkout service from info to debug by editing its ConfigMap, because the errors in the logs are not detailed enough to say what is failing. Twenty minutes later the logs are still at info. The engineer edits the ConfigMap again, carefully, and watches kubectl get configmap -o yaml show the new value. The logs stay at info.

At 03:20 a second engineer rotates the database password in the same service’s Secret, as part of a credential rotation that had been scheduled for the maintenance window. The application keeps authenticating with the old password for another nine minutes, and then, without anyone doing anything, starts authenticating with the new one.

Both of these are the same mechanism seen from two ends, and both are completely deterministic. This lab builds the smallest system that exhibits both.

Tasks

Task 1: Capture the starting state

Nothing here is destructive, but the last command tells you whether Cleanup will be safe.

NS=config-lab
mkdir -p "$HOME/k8s-config-lab"
cd "$HOME/k8s-config-lab"

# Which cluster is this? Do not skip: an unnoticed context switch is the
# single most common way a lab lands somewhere it should not.
kubectl config current-context | tee context.txt
kubectl version -o yaml | tee versions.yaml

# Does the namespace already exist? Empty output means it does not,
# which is what you want.
kubectl get namespace "$NS" --ignore-not-found

If that last command printed a namespace, choose another name and use it everywhere below. If it printed nothing, create it:

Configuration changeworkstation— Creates the namespace this lab lives in

$ kubectl create namespace config-lab

namespace/config-lab created

Illustrative output

Task 2: Write the ConfigMap and the Secret

Save this as configmap.yaml. Read the three keys before you apply it — the lab turns on the difference between them.

apiVersion: v1
kind: ConfigMap
metadata:
  name: web-config
  namespace: config-lab
data:
  LOG_LEVEL: info
  GREETING: "config revision 1"
  app.conf: |
    [server]
    port = 8080
    workers = 4
    log_level = info

LOG_LEVEL and GREETING are valid shell identifiers. app.conf is not — a dot cannot appear in an environment variable name. That single character is what makes this ConfigMap worth studying: it will behave differently on the two consumption surfaces, and the difference is not an error.

Save this as secret.yaml:

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
  namespace: config-lab
type: Opaque
stringData:
  username: app
  password: rev1-lab-only-not-a-real-credential

stringData takes plain text and the API server encodes it; data requires you to have encoded it already. Both end up stored identically. The password above is deliberately self-describing: nothing in this lab should ever be reused, and a credential that announces itself is easier to spot if it escapes into a terminal history.

Apply both:

NS=config-lab
kubectl apply -f configmap.yaml -f secret.yaml
kubectl get configmap,secret -n "$NS"

Task 3: Write the Deployment that consumes both, four ways at once

Save this as deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  namespace: config-lab
  labels:
    app: web
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: web
          image: busybox:1.36
          command: ["sh", "-c", "while true; do sleep 3600; done"]
          envFrom:
            - configMapRef:
                name: web-config
          env:
            - name: DB_USERNAME
              valueFrom:
                secretKeyRef:
                  name: db-credentials
                  key: username
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: db-credentials
                  key: password
          volumeMounts:
            - name: web-config
              mountPath: /etc/web
              readOnly: true
            - name: db-credentials
              mountPath: /etc/db
              readOnly: true
      volumes:
        - name: web-config
          configMap:
            name: web-config
        - name: db-credentials
          secret:
            secretName: db-credentials
            defaultMode: 0400

Four consumptions of two objects: envFrom takes the whole ConfigMap as environment variables, the web-config volume projects the same ConfigMap as files, secretKeyRef names two Secret keys as environment variables, and the db-credentials volume projects the same Secret as files.

Apply it and wait for the rollout to finish rather than assuming it did:

NS=config-lab
kubectl apply -f deployment.yaml
kubectl rollout status deployment/web -n "$NS" --timeout=120s

Task 4: Read the four surfaces and account for every key

This is the accounting step, and it is the one people skip. Do not skip it — the whole rest of the lab is a set of changes to these four surfaces, and you cannot see a change you never measured a baseline for.

NS=config-lab
POD="$(kubectl get pod -n "$NS" -l app=web \
  -o jsonpath='{.items[0].metadata.name}')"
echo "pod: $POD"

# Surface 1: environment variables from the ConfigMap
kubectl exec -n "$NS" "$POD" -- env | sort | grep -E 'LOG_LEVEL|GREETING|app'

# Surface 2: files from the ConfigMap
kubectl exec -n "$NS" "$POD" -- ls /etc/web
kubectl exec -n "$NS" "$POD" -- cat /etc/web/app.conf

# Surface 3: environment variables from the Secret
kubectl exec -n "$NS" "$POD" -- env | grep -E '^DB_'

# Surface 4: files from the Secret
kubectl exec -n "$NS" "$POD" -- ls -l /etc/db

Read-only / Safeworkstation— What arrived on the environment surface

$ kubectl exec -n config-lab deploy/web -- env | sort | grep -E 'LOG_LEVEL|GREETING|app'

GREETING=config revision 1
LOG_LEVEL=info

Illustrative output

Two keys went in as environment variables. Three keys came out as files. Write the accounting table now — it is a deliverable:

ConfigMap key	Environment variable	File under /etc/web
`LOG_LEVEL`	yes	yes
`GREETING`	yes	yes
`app.conf`	no	yes

app.conf is missing from the environment because envFrom copies keys verbatim — it does not uppercase them and it does not rewrite punctuation — and a key that is not a valid environment variable name is skipped rather than mangled. The container starts anyway. Check the Pod’s events, because the skip is reported there rather than in the container:

NS=config-lab
kubectl get events -n "$NS" --sort-by=.lastTimestamp | tail -20

Note also what surface 3 tells you about Secrets: DB_PASSWORD is sitting in the container’s environment in plain text, readable by every process in that container through /proc/1/environ, and it will appear in a core dump if the process ever writes one. Surface 4 is the same credential with a file mode on it. Compare the two:

NS=config-lab
kubectl exec -n config-lab deploy/web -- sh -c 'tr "\0" "\n" < /proc/1/environ | grep DB_'
kubectl exec -n config-lab deploy/web -- ls -l /etc/db

Task 5: Change both objects and time the divergence

Start a clock, make one write to each object, and watch. This is the measurement the whole lab exists for.

In one terminal, start the observer. It runs for four minutes and stops on its own:

NS=config-lab
POD="$(kubectl get pod -n "$NS" -l app=web \
  -o jsonpath='{.items[0].metadata.name}')"

for i in $(seq 1 24); do
  printf '%s  file=%-8s env=%-8s secret_file=%s\n' \
    "$(date -u +%H:%M:%S)" \
    "$(kubectl exec -n "$NS" "$POD" -- cat /etc/web/LOG_LEVEL 2>/dev/null)" \
    "$(kubectl exec -n "$NS" "$POD" -- printenv LOG_LEVEL 2>/dev/null)" \
    "$(kubectl exec -n "$NS" "$POD" -- cat /etc/db/password 2>/dev/null)"
  sleep 10
done

In a second terminal, note the time and make the change:

NS=config-lab
date -u +%H:%M:%S

kubectl patch configmap web-config -n "$NS" --type=merge \
  -p '{"data":{"LOG_LEVEL":"debug","GREETING":"config revision 2"}}'

kubectl patch secret db-credentials -n "$NS" --type=merge \
  -p '{"stringData":{"password":"rev2-lab-only-not-a-real-credential"}}'

Go back to the observer and watch. What you should see, and what you must record in the timing table:

file= flips from info to debug some tens of seconds after the write.
secret_file= flips at roughly the same time, by the same mechanism.
env= never changes. Not in four minutes, not in four hours.

Why the file lags, and why the lag is bounded

The kubelet does not push the change. It re-reads the ConfigMap and Secret objects a Pod refers to on its periodic sync, and updates the projected volume if the content differs. The Kubernetes documentation states the propagation bound directly: the total delay from updating a ConfigMap to the new keys being projected into the Pod can be as long as the kubelet sync period — one minute by default — plus the cache propagation delay.

That is a bound, not a schedule. A change can appear in five seconds or in seventy. If you measured twelve seconds, you have not discovered that the sync period is twelve seconds; you have discovered that you wrote the ConfigMap shortly before a sync was due.

The environment surface is not lagging. It is not participating. Environment variables are part of the process’s initial state, handed to it by the container runtime at exec time. Nothing in Kubernetes can reach into a running process and change them, which is why the answer is never “wait longer”.

Task 6: Look under the mount point

Now find out how a projected file can change underneath a running process without that process ever seeing a half-written file.

NS=config-lab
kubectl exec -n config-lab deploy/web -- ls -la /etc/web

Read-only / Safeinside the container— The projected volume is a directory of symlinks, not a directory of files

$ kubectl exec -n config-lab deploy/web -- ls -la /etc/web

total 0
drwxrwxrwt    3 root     root           120 Aug 18 09:12 .
drwxr-xr-x    1 root     root            60 Aug 18 09:12 ..
drwxr-xr-x    2 root     root           100 Aug 18 09:14 ..2026_08_18_09_14_31.2418306715
lrwxrwxrwx    1 root     root            32 Aug 18 09:14 ..data -> ..2026_08_18_09_14_31.2418306715
lrwxrwxrwx    1 root     root            15 Aug 18 09:12 GREETING -> ..data/GREETING
lrwxrwxrwx    1 root     root            16 Aug 18 09:12 LOG_LEVEL -> ..data/LOG_LEVEL
lrwxrwxrwx    1 root     root            15 Aug 18 09:12 app.conf -> ..data/app.conf

Illustrative output

Every key you can see is a symlink into ..data, and ..data is itself a symlink into a timestamped directory. To publish an update, the kubelet writes a new timestamped directory containing the complete new content and then re-points ..data at it in a single operation, after which the old directory is removed. A reader either follows the old link and gets the complete old content, or follows the new link and gets the complete new content. There is no window in which it can read half of each.

This also explains the failure mode you are about to reproduce.

Prove it. Save this as deployment-subpath.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-subpath
  namespace: config-lab
  labels:
    app: web-subpath
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web-subpath
  template:
    metadata:
      labels:
        app: web-subpath
    spec:
      containers:
        - name: web
          image: busybox:1.36
          command: ["sh", "-c", "while true; do sleep 3600; done"]
          volumeMounts:
            - name: web-config
              mountPath: /etc/web/app.conf
              subPath: app.conf
              readOnly: true
      volumes:
        - name: web-config
          configMap:
            name: web-config

NS=config-lab
kubectl apply -f deployment-subpath.yaml
kubectl rollout status deployment/web-subpath -n "$NS" --timeout=120s

# Baseline: both Deployments agree
kubectl exec -n "$NS" deploy/web         -- cat /etc/web/app.conf
kubectl exec -n "$NS" deploy/web-subpath -- cat /etc/web/app.conf

# Change the key both of them read
kubectl patch configmap web-config -n "$NS" --type=merge \
  -p '{"data":{"app.conf":"[server]\nport = 8080\nworkers = 16\nlog_level = debug\n"}}'
sleep 90

# The volume mount moved. The subPath mount did not.
kubectl exec -n "$NS" deploy/web         -- cat /etc/web/app.conf
kubectl exec -n "$NS" deploy/web-subpath -- cat /etc/web/app.conf
kubectl exec -n "$NS" deploy/web-subpath -- ls -la /etc/web

The ls -la on the second Deployment is the tell: there is no ..data, no timestamped directory and no symlink — just a plain file bind-mounted into place. It was correct when the container started and it will be correct forever after, whatever the ConfigMap says.

Task 7: Prove that base64 is not a security control

NS=config-lab

# What the API stores
kubectl get secret db-credentials -n "$NS" -o jsonpath='{.data.password}'; echo

# What it means
kubectl get secret db-credentials -n "$NS" \
  -o jsonpath='{.data.password}' | base64 -d; echo

# The convenience form, which does the same thing in one step
kubectl get secret db-credentials -n "$NS" -o jsonpath='{.data.password}' \
  | base64 -d | wc -c

One command and no credentials of any kind beyond the ones you already used to talk to the API. Now ask the question that actually matters — not “is it encoded” but “who is allowed to ask”:

NS=config-lab
kubectl auth can-i get secrets -n "$NS"
kubectl auth can-i get secrets -n "$NS" \
  --as=system:serviceaccount:config-lab:default

The encoding is a transport convenience; the control is RBAC

Base64 exists in the Secret API because data values are arbitrary bytes and JSON is text. It stops nothing. The three things that do stop something are: RBAC that scopes get on secrets to the identities that genuinely need it, encryption at rest so a stolen etcd snapshot is not a credential dump, and keeping the high-value credentials outside the cluster entirely so that the Kubernetes object is a reference rather than the secret.

A Secret is worth using over a ConfigMap anyway — it is the object those three controls are built around, it is excluded from ordinary object dumps, and it makes the intent auditable. But if your threat model is “someone read our YAML”, a Secret and a ConfigMap are the same object with different spelling.

Task 8: Make the change actually reach the Pod

You now have a Pod whose LOG_LEVEL environment variable still says info while the ConfigMap says debug. There are exactly two ways out, and they differ in who has to remember.

The first is the manual rollout. Every replica is replaced, so the new process starts with the new environment:

Service impact possibleworkstation— Replaces every Pod in the Deployment; brief capacity reduction during the roll

$ kubectl rollout restart deployment/web -n config-lab && kubectl rollout status deployment/web -n config-lab --timeout=120s

deployment.apps/web restarted
Waiting for deployment "web" rollout to finish: 1 old replicas are pending termination...
deployment "web" successfully rolled out

Illustrative output

NS=config-lab
kubectl exec -n "$NS" deploy/web -- printenv LOG_LEVEL
kubectl exec -n "$NS" deploy/web -- printenv GREETING

Both now report revision 2. The cost of this route is that it is a human step: it lives in a runbook, and the 02:40 engineer in the Scenario is the proof that runbook steps get missed.

The second way removes the human. Put a checksum of the ConfigMap’s data into the Pod template, so that changing the ConfigMap changes the template and the Deployment controller rolls on its own:

NS=config-lab
SUM="$(kubectl get configmap web-config -n "$NS" \
  -o jsonpath='{.data}' | sha256sum | cut -d' ' -f1)"
echo "checksum: $SUM"

kubectl patch deployment web -n "$NS" --type=merge \
  -p "{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"config-checksum/web-config\":\"$SUM\"}}}}}"

kubectl rollout status deployment/web -n "$NS" --timeout=120s
kubectl get deployment web -n "$NS" \
  -o jsonpath='{.spec.template.metadata.annotations}'; echo

Change the ConfigMap, recompute, patch, and the roll happens because the template hash moved. In production a controller such as Reloader, Argo CD or Flux does the recompute-and-patch step; the mechanism it uses is exactly the one you just performed by hand.

Validation

Each of these is a command whose output settles the claim, not a restatement of the claim.

kubectl exec -n config-lab deploy/web -- env | grep -c app returns 0, and kubectl exec -n config-lab deploy/web -- ls /etc/web | grep -c app.conf returns 1. The same key, present on one surface and absent from the other.
Your timing table has three rows and the third one is empty: the ConfigMap write has a time, the file change has a time roughly tens of seconds later, and the environment change never happened.
kubectl exec -n config-lab deploy/web -- ls -la /etc/web shows a ..data symlink pointing at a timestamped directory, and every key as a symlink into ..data.
kubectl exec -n config-lab deploy/web-subpath -- ls -la /etc/web shows no ..data and no symlinks, and cat /etc/web/app.conf in that Pod still shows workers = 4 while the same file in deploy/web shows workers = 16.
kubectl get secret db-credentials -n config-lab -o jsonpath='{.data.password}' | base64 -d prints the current password in plain text, with no additional credential.
After the rollout in Task 8, kubectl exec -n config-lab deploy/web -- printenv LOG_LEVEL prints debug.
kubectl get deployment web -n config-lab -o jsonpath='{.spec.template.metadata.annotations}' contains config-checksum/web-config with a 64-character hex value.

Expected Outcome

$HOME/k8s-config-lab/
├── configmap.yaml
├── context.txt
├── deployment.yaml
├── deployment-subpath.yaml
├── secret.yaml
├── timing-table.md
└── versions.yaml

In the cluster: a namespace config-lab holding one ConfigMap, one Secret and two Deployments, with the web Deployment carrying a config-checksum/web-config annotation on its Pod template.

More importantly, you can now answer the opening question without guessing. Given any Pod and any ConfigMap, you can say which consumption surfaces are in play by reading the Pod spec, you can say which of them will move on a write and which will not, and you know that “wait longer” is a valid answer for exactly one of them.

Troubleshooting

The Pod stays in ContainerCreating and events mention the ConfigMap or Secret. A configMapKeyRef or secretKeyRef is optional: false — the default — and the object or the key does not exist. Check the exact spelling, including the namespace: a ConfigMap in default is invisible to a Pod in config-lab. kubectl describe pod names the missing object.

The Pod is in CreateContainerConfigError. Same family of cause, surfaced at a later stage: the kubelet found the object but could not assemble the container’s configuration from it. Read the message on kubectl describe pod; it names the key.

kubectl exec fails with “container not found” or “unable to upgrade connection”. You are executing against a Pod that has been replaced — likely after the Task 8 rollout. Re-resolve POD from the label selector, or use deploy/web, which resolves at call time.

The file never changes, even after five minutes. Confirm you patched the object your Pod actually mounts: kubectl get configmap web-config -n config-lab -o yaml should show the new value. If it does, check whether the mount is a subPath — that is Task 6, and it is working as designed. If it is neither, check whether the ConfigMap is marked immutable: true, in which case the patch would have been rejected and you would have seen the error.

The Secret value has a trailing newline you did not expect. This lab uses stringData in a manifest, which does not add one. Creating a Secret with --from-file preserves the file’s bytes exactly, newline included, and that newline is the single most common cause of a credential that is correct to the eye and rejected by the server. Compare byte counts with wc -c rather than reading the value.

base64 -d prints “invalid input”. Some base64 implementations want -D or --decode, and some object to the absence of a trailing newline. kubectl get secret ... -o go-template='{{.data.password | base64decode}}' avoids the local tool entirely.

Cleanup

The lab created one namespace and one directory. Both go.

Confirm what you are about to remove before you remove it:

NS=config-lab
kubectl get all,configmap,secret -n "$NS"

Then delete the namespace, which removes both Deployments, their ReplicaSets, their Pods, the ConfigMap and the Secret:

Destructiveworkstation— Deletes the namespace and every object in it, including any object you added by hand

$ kubectl delete namespace config-lab

namespace "config-lab" deleted

Illustrative output

Namespace deletion is asynchronous. Confirm it finished rather than assuming it did, because a namespace stuck in Terminating will block the name being reused:

kubectl get namespace config-lab --ignore-not-found

The lab changed nothing outside that namespace: no kubelet configuration, no node state, no cluster-scoped object, and no context switch. Your workstation files can go too, once you have copied the timing table somewhere you keep notes:

rm -rf "$HOME/k8s-config-lab"

What You Learned

A ConfigMap key can exist on one consumption surface and not the other. envFrom copies keys verbatim and skips any key that is not a valid environment variable name; the volume projection takes all of them. You measured this rather than reading it, and you wrote the accounting table.
The propagation delay for a projected volume is bounded, not scheduled. Up to the kubelet’s sync period plus cache propagation. A fast observation is luck, not a measurement of the sync period.
Environment variables are not slow to update; they do not update. They are the process’s initial state. The only remedies are a new process: a manual rollout, or an automatic one triggered by a checksum in the Pod template.
The atomic ..data symlink is why updates are safe and why subPath updates are not. You saw both directories side by side and the difference between them explains the behaviour without any appeal to documentation.
A Secret’s base64 is not protection. You decoded one with a single command. RBAC, encryption at rest, and keeping the real credential outside the cluster are the controls; the object type is what those controls hang off.

Production notes

Map this onto a real change window.

A ConfigMap change is a deployment. Treat it in the change record the same way you treat an image bump, because for an env-var consumer it is one — the rollout is unavoidable, and pretending the change is “just config” is how a config edit ends up made outside a window with no rollback plan. Write the ConfigMap change and the rollout command as one step in the runbook, never as two.

Decide the trigger per service, in advance. Manual rollout is right for services whose config changes are rare and always accompanied by a human; the checksum annotation is right for services whose config is generated by CI or GitOps, because there the human is not in the loop and cannot be relied on. Choosing per incident, at 02:40, produces the Scenario.

Know your rollback before you patch. For a ConfigMap the rollback is kubectl rollout undo deployment/<name> for the Pod template plus a restore of the ConfigMap data itself — the undo does not revert the ConfigMap, only the template that referenced it. Capture the current data first: kubectl get configmap web-config -n prod -o yaml > web-config.pre.yaml is the cheapest insurance in this whole document.

“Hold” is a legitimate outcome. If the file surface has moved and the application has not reloaded, a rollout will fix it and will also cost you every warm cache and in-flight request in the service. Holding until the window, with the discrepancy recorded and an owner and an end time named, is frequently the better call. What is never acceptable is leaving it undecided, because the next engineer will read the ConfigMap, believe it, and act on a value that no running process has.

Secrets rotate on the file surface with no restart, and that is a feature. A credential mounted as a file can be rotated inside the sync window without touching the workload, provided the application re-reads it. A credential injected as an environment variable cannot be rotated without a restart of every consumer. That single fact should decide the consumption pattern before anything else does.

Objectives

Prerequisites

Objective

Architecture

Requirements

Scenario

Tasks

Task 1: Capture the starting state

Task 2: Write the ConfigMap and the Secret

Task 3: Write the Deployment that consumes both, four ways at once

Task 4: Read the four surfaces and account for every key

Task 5: Change both objects and time the divergence

Task 6: Look under the mount point

Task 7: Prove that base64 is not a security control

Task 8: Make the change actually reach the Pod

Validation

Expected Outcome

Troubleshooting

Cleanup

What You Learned

Production notes

Deliverables

Verification status