Run database migrations as a Job before the new version

Approach

Run migrations once per release as a Job (a Helm pre-upgrade hook or an Argo CD PreSync hook), with the same image as the app, and only roll out the Deployment after it succeeds.

Solution

apiVersion: batch/v1
kind: Job
metadata:
  name: migrate-1-4-2
  annotations:
    helm.sh/hook: pre-upgrade,pre-install
    helm.sh/hook-delete-policy: before-hook-creation
    argocd.argoproj.io/hook: PreSync
spec:
  backoffLimit: 1
  activeDeadlineSeconds: 900
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: migrate
          image: ghcr.io/acme/api:1.4.2
          command: ["python", "-m", "app.migrate"]
          envFrom: [{secretRef: {name: api-db}}]

Pitfalls

• Breaking migrations: old pods still running during the rollout fail; use expand-and-contract.
• Running migrations in every pod's startup, racing each other.

Variations

• Migration tools with locking (Flyway, Liquibase, Alembic with an advisory lock).
• Separate long data backfills from schema migrations.

Further reading: Helm chart hooks