Why does changing a Deployment's selector in a chart make helm upgrade fail?

spec.selector is immutable in apps/v1 Deployments; the API server rejects the patch, so the upgrade fails and the release is marked failed.

How do you safely change a Deployment's selector labels?

Create a new Deployment (new name and selector), move traffic to it, then delete the old one; never edit the selector in place.

How can CI catch upgrades the API server will reject?

Run a server-side dry run of the rendered upgrade (helm upgrade --dry-run=server or kubectl diff/apply --dry-run=server) against a cluster that has the current release.

Upgrades That Fail on Immutable Fields — Helm, Kustomize and GitOps with Argo CD

Scenario

Example company: a team tidies up labels in its chart, renaming app: api to the recommended app.kubernetes.io/name: api everywhere, including the Deployment's selector.

What you see in production

helm upgrade fails with UPGRADE FAILED: ... spec.selector: Invalid value ... field is immutable. The release is marked failed, and the next attempts fail the same way. The on-call engineer considers deleting the Deployment, which would take the service down.

The naive code

spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: api     # was: app: api

Reproduce it

Install a release on a real cluster, then upgrade with changed selector labels:

helm upgrade against an immutable selector

kubectl delete namespace gitops-helm --ignore-not-found --wait=true > /dev/null   # leftovers from an earlier run
kubectl create namespace gitops-helm > /dev/null
trap 'helm -n gitops-helm uninstall api > /dev/null 2>&1; kubectl delete namespace gitops-helm --wait=false > /dev/null' EXIT
mkdir -p chart/templates
printf 'apiVersion: v2\nname: api\nversion: 0.1.0\n' > chart/Chart.yaml
printf 'selectorKey: app\n' > chart/values.yaml
cat > chart/templates/deploy.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: api}
spec:
  replicas: 1
  selector:
    matchLabels: {{ printf "{%s: api}" .Values.selectorKey }}
  template:
    metadata:
      labels: {{ printf "{%s: api}" .Values.selectorKey }}
    spec:
      containers: [{name: api, image: "busybox:1.37", command: ["sleep", "3600"]}]
EOF
helm -n gitops-helm install api ./chart --wait --timeout 120s > /dev/null && echo "install: deployed"
helm -n gitops-helm upgrade api ./chart --set selectorKey=app.kubernetes.io/name 2>&1 | grep '^Error:' | sed -E 's/server-side apply failed for object [^:]*: //'
echo "release status: $(helm -n gitops-helm status api -o json | python3 -c 'import json,sys; print(json.load(sys.stdin)["info"]["status"])')"
echo "pods still running the old version: $(kubectl -n gitops-helm get pods -l app=api --field-selector=status.phase=Running -o name | wc -l)"

kubectl delete namespace gitops-helm --ignore-not-found --wait=true > /dev/null   # leftovers from an earlier run
kubectl create namespace gitops-helm > /dev/null
trap 'helm -n gitops-helm uninstall api > /dev/null 2>&1; kubectl delete namespace gitops-helm --wait=false > /dev/null' EXIT
mkdir -p chart/templates
printf 'apiVersion: v2\nname: api\nversion: 0.1.0\n' > chart/Chart.yaml
printf 'selectorKey: app\n' > chart/values.yaml
cat > chart/templates/deploy.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: api}
spec:
  replicas: 1
  selector:
    matchLabels: {{ printf "{%s: api}" .Values.selectorKey }}
  template:
    metadata:
      labels: {{ printf "{%s: api}" .Values.selectorKey }}
    spec:
      containers: [{name: api, image: "busybox:1.37", command: ["sleep", "3600"]}]
EOF
helm -n gitops-helm install api ./chart --wait --timeout 120s > /dev/null && echo "install: deployed"
helm -n gitops-helm upgrade api ./chart --set selectorKey=app.kubernetes.io/name 2>&1 | grep '^Error:' | sed -E 's/server-side apply failed for object [^:]*: //'
echo "release status: $(helm -n gitops-helm status api -o json | python3 -c 'import json,sys; print(json.load(sys.stdin)["info"]["status"])')"
echo "pods still running the old version: $(kubectl -n gitops-helm get pods -l app=api --field-selector=status.phase=Running -o name | wc -l)"

Output

install: deployed
Error: UPGRADE FAILED: Deployment.apps "api" is invalid: spec.selector: Invalid value: {"matchLabels":{"app.kubernetes.io/name":"api"}}: field is immutable
release status: failed
pods still running the old version: 1

Run this on your computer — see “Get Started” for setup.

Root cause

Some fields can't change after creation: a Deployment's spec.selector, a StatefulSet's volumeClaimTemplates and serviceName, a Service's clusterIP, a Job's pod template. Helm sends the change (Helm 4 uses server-side apply), the API server rejects it, and the release is marked failed. Helm can't "fix" it without deleting and recreating the object.

The fix

Treat selector labels as permanent; add new labels to the pod template and metadata, but keep selector.matchLabels unchanged.
When a change is truly needed, migrate: create a new Deployment with a new name and selector, shift traffic (the Service selects both, or switch the Service), then delete the old one.
For StatefulSet template changes, delete the StatefulSet with --cascade=orphan (pods and volumes stay) and recreate it, carefully and rehearsed.
Run helm upgrade --dry-run=server (or a GitOps diff with server-side dry run) in CI to catch API rejections before production.

Trade-offs

Migrations by new name double resources temporarily and need traffic switching.
Orphan-and-recreate is powerful but risky; practise it in staging.

Detect it in production

CI step: server-side dry run of every upgrade against a cluster with the current release.
Alerts on Helm releases in failed state.

Key points

Deployment selectors and several other fields are immutable.
Keep selector labels stable; add new labels elsewhere.
Migrate with a new object name when you must change them, and dry-run upgrades in CI.

Further reading: Deployment selector updates · helm upgrade