Deploying Before the Database Migration Ran

Scenario

Example company: a release adds a database column and code that reads it. The migration runs as a Kubernetes Job in the same Argo CD application as the Deployment.

What you see in production

For two minutes after each such release, the new pods crash with column "discount_code" does not exist until the migration Job, started at the same time, finishes. Occasionally the Job fails and the new code keeps crashing while the old pods are already gone.

The naive code

# app/ contains job-migrate.yaml and deployment.yaml; Argo CD applies them together

Reproduce it

Apply resources by sync wave, waiting for each wave to become healthy, and stop when a wave fails:

resources = [
    {"name": "configmap/app-config", "wave": -1, "healthy": True},
    {"name": "job/migrate-0042", "wave": 0, "hook": "PreSync", "healthy": True},
    {"name": "deployment/api", "wave": 1, "healthy": True},
    {"name": "service/api", "wave": 1, "healthy": True},
]

def sync(resources, migration_ok=True):
    log = []
    ordered = sorted(resources, key=lambda r: (0 if r.get("hook") == "PreSync" else 1, r["wave"]))
    for wave in dict.fromkeys((0 if r.get("hook") == "PreSync" else 1, r["wave"]) for r in ordered):
        batch = [r for r in ordered if (0 if r.get("hook") == "PreSync" else 1, r["wave"]) == wave]
        for r in batch:
            ok = migration_ok if r["name"].startswith("job/") else r["healthy"]
            log.append(f"{r['name']}: {'healthy' if ok else 'FAILED'}")
            if not ok:
                return log + ["sync stopped: later waves not applied, old version keeps running"]
    return log

print("migration succeeds:", sync(resources))
print("migration fails:   ", sync(resources, migration_ok=False))

migration succeeds: ['job/migrate-0042: healthy', 'configmap/app-config: healthy', 'deployment/api: healthy', 'service/api: healthy']
migration fails:    ['job/migrate-0042: FAILED', 'sync stopped: later waves not applied, old version keeps running']

Root cause

Without ordering, Argo CD applies all resources in one go (with only a default ordering by kind). The Deployment's new pods start before the migration Job has finished, so code that needs the new schema runs against the old one. If the Job fails, nothing stops the rollout.

The fix

# job-migrate.yaml
metadata:
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
---
# resources that must come first in the main sync phase
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "-1"

• PreSync hooks run before the main sync; if they fail, the sync stops and the old version keeps running.
• Sync waves order resources within a phase; Argo CD waits for each wave to be healthy before the next.
• Write backward-compatible migrations (expand, then contract) so old and new code both work during the rollout.

Trade-offs

• Hooks make syncs slower and add a failure point; keep migrations small and idempotent.
• Long-running migrations need a separate process, not a sync hook with timeouts.

Detect it in production

• Errors right after deploys mentioning missing columns or tables.
• Failed hook Jobs and syncs stopped in the PreSync phase.

Key points

• Without ordering, migrations and new code start together.
• PreSync hooks and sync waves order the rollout and stop it on failure.
• Expand-and-contract migrations keep old and new versions compatible.

Further reading: Sync phases and waves · Resource hooks