ConfigMap Changes That Never Reach the Pods

Scenario

Example company: the team turns off a misbehaving feature flag by editing a ConfigMap during an incident, then watches the error rate. Nothing changes.

What you see in production

The ConfigMap shows FEATURE_RECS=off, but the application keeps behaving as if it were on. Twenty minutes later someone restarts the Deployment and the problem stops. A week later, a node replacement restarts some pods, which pick up a different ConfigMap value than their siblings.

The naive code

kubectl edit configmap app-config      # change FEATURE_RECS to off and hope

Reproduce it

A pod reading the flag from an environment variable keeps the old value after the ConfigMap changes:

kubectl create namespace prod-config > /dev/null
trap 'kubectl delete namespace prod-config --wait=false > /dev/null' EXIT
kubectl -n prod-config create configmap app-config --from-literal=FEATURE_RECS=on > /dev/null
kubectl -n prod-config create deployment app --image=busybox:1.37 -- sh -c 'sleep 3600' > /dev/null
kubectl -n prod-config set env deployment/app --from=configmap/app-config > /dev/null
kubectl -n prod-config rollout status deployment/app --timeout=120s > /dev/null
flag() { kubectl -n prod-config exec deploy/app -- sh -c 'echo $FEATURE_RECS'; }
echo "running pod sees: FEATURE_RECS=$(flag)"
kubectl -n prod-config create configmap app-config --from-literal=FEATURE_RECS=off --dry-run=client -o yaml | kubectl -n prod-config apply -f - > /dev/null
echo "configmap now:    FEATURE_RECS=$(kubectl -n prod-config get configmap app-config -o jsonpath='{.data.FEATURE_RECS}')"
sleep 5
echo "running pod sees: FEATURE_RECS=$(flag) (unchanged)"
kubectl -n prod-config rollout restart deployment/app > /dev/null
kubectl -n prod-config rollout status deployment/app --timeout=120s > /dev/null
echo "after a rollout:  FEATURE_RECS=$(flag)"

running pod sees: FEATURE_RECS=
configmap now:    FEATURE_RECS=
running pod sees: FEATURE_RECS= (unchanged)
after a rollout:  FEATURE_RECS=

Root cause

Environment variables are read once, when the container starts. Mounted ConfigMap files do update in place, but only after a delay, and only if the application re-reads them. Nothing in Kubernetes restarts pods when a ConfigMap changes, so the cluster can run old and new configuration side by side, depending on which pods happen to restart.

The fix

Make a configuration change produce a rollout:

# Helm: a checksum of the rendered ConfigMap in the pod template
template:
  metadata:
    annotations:
      checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}

• Kustomize configMapGenerator adds a content hash to the ConfigMap's name, so any change creates a new ConfigMap and updates the Deployment reference, triggering a rollout (and the old one remains for rollbacks).
• Treat ConfigMaps as immutable (immutable: true) and version them.
• For flags that must change instantly, use a feature-flag service the app polls, not a ConfigMap.

Trade-offs

• Rollouts for every configuration change take time; true runtime flags belong in a flag system.
• Hash-named ConfigMaps accumulate; let your tooling prune old ones.
• Hot reload of mounted files avoids restarts but needs application support and careful validation.

Detect it in production

• Pods whose start time is older than the last change of the ConfigMap they use.
• Configuration version exposed by the app (an info endpoint or metric) compared across pods.

Key points

• ConfigMap edits don't restart pods; env vars stay at their start-time values.
• Tie config to rollouts with checksum annotations or hash-suffixed generated ConfigMaps.
• Use a feature-flag system for changes that must take effect immediately.

Further reading: ConfigMaps · Kustomize configMapGenerator