Cluster Upgrades Blocked by Removed APIs

Scenario

Example company: the platform team upgrades a cluster two minor versions in one quarter. The next deploy of an older service fails, and its Helm release can't be upgraded or rolled back.

What you see in production

error: resource mapping not found for name: "checkout" ... no matches for kind "PodDisruptionBudget" in version "policy/v1beta1". Several charts in Git still use API versions removed in the new release.

The naive code

apiVersion: policy/v1beta1          # removed in Kubernetes 1.25
kind: PodDisruptionBudget

Reproduce it

Validate the same manifest against an old and a current Kubernetes version with kubeconform, as a CI check would:

cat > pdb.yaml <<'EOF'
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata: {name: checkout}
spec:
  minAvailable: 2
  selector: {matchLabels: {app: checkout}}
EOF
echo "--- CI check with kubeconform against the old and the target version:"
for v in 1.24.0 1.37.0; do
  printf '%-8s ' "$v:"
  kubeconform -kubernetes-version "$v" -summary pdb.yaml 2>&1 | tail -1 | sed 's/Summary: 1 resource found in 1 file - //'
done
echo "--- what a real 1.37 API server says (server-side dry run):"
kubectl apply --dry-run=server -f pdb.yaml 2>&1 | head -1 | sed 's/ namespace: "" from "pdb.yaml"//'
sed -i 's#policy/v1beta1#policy/v1#' pdb.yaml
echo "--- after changing to policy/v1:"
kubectl apply --dry-run=server -n default -f pdb.yaml

--- CI check with kubeconform against the old and the target version:
1.24.0:  Valid: 1, Invalid: 0, Errors: 0, Skipped: 0
1.37.0:  Valid: 0, Invalid: 0, Errors: 1, Skipped: 0
--- what a real 1.37 API server says (server-side dry run):
error: resource mapping not found for name: "checkout": no matches for kind "PodDisruptionBudget" in version "policy/v1beta1"
--- after changing to policy/v1:
poddisruptionbudget.policy/checkout created (server dry run)

Root cause

Kubernetes graduates APIs (v1beta1 to v1) and removes the old versions after a deprecation period. The API server then rejects objects in those versions. Manifests in Git, Helm charts and even stored Helm release metadata can still reference them, and nothing fails until the cluster is upgraded.

The fix

• Read the deprecated API migration guide before each upgrade; scan manifests and live clusters with tools like kubeconform (-kubernetes-version set to the target), pluto or kubent.
• Validate against the next version in CI so new code can't introduce removed APIs.
• Upgrade one minor version at a time, control plane first, then nodes; test in a staging cluster.
• Update third-party charts as part of the upgrade plan.
• For Helm releases stuck on removed APIs, the helm mapkubeapis plugin rewrites release metadata.

Trade-offs

• Frequent upgrades (three minor releases a year, about 14 months of support each) cost effort, but skipping them builds a bigger, riskier jump.
• Managed extended support buys time at a price.

Detect it in production

• API server metrics and audit logs for requests to deprecated APIs (apiserver_requested_deprecated_apis).
• CI validation against the next Kubernetes version.
• Cluster versions approaching end of support.

Key points

• Deprecated APIs are removed; manifests using them fail after upgrades.
• Validate against the target version in CI and scan live clusters before upgrading.
• Upgrade one minor version at a time, at least yearly.

Further reading: Deprecated API migration guide · Version skew policy · kubeconform