The Rollback That Made It Worse

Scenario

Example team: an online marketplace deploys several times a day with a pipeline that runs database migrations and then rolls out the new pods. Release 42 renames the customers.name column to full_name and updates the code to match.

What you see in production

Release 42 has an unrelated bug, so the on-call engineer clicks rollback to release 41. Instead of recovering, every customer page now returns 500: release 41 queries customers.name, which no longer exists. During the rolling update itself, old pods also failed for a few minutes while new pods were starting. Restoring service takes a hot-fix forward under pressure.

The naive code

One migration that changes the schema in a way only the new code understands:

-- migrations/0042_rename_customer_name.sql  (runs right before the new pods start)
ALTER TABLE customers RENAME COLUMN name TO full_name;

Reproduce it

Check which code versions work with which schema states. A deployment is safe only if old and new code both work with the schema at every step, including during rollout and after rollback:

def works(code_reads, schema_columns):
    return code_reads <= schema_columns

CODE = {"v41": {"name"}, "v42": {"full_name"}, "v42-compat": {"full_name"}}

print("Naive: one migration renames the column")
for state, cols in [("before", {"name"}), ("after rename", {"full_name"})]:
    row = ", ".join(f"{v} {'ok' if works(r, cols) else 'BROKEN'}" for v, r in CODE.items() if v != "v42-compat")
    print(f"  schema {state:12}: {row}")

print("Expand / migrate / contract")
steps = [("1. expand: add full_name, write both", {"name", "full_name"}),
         ("2. backfill, deploy v42 reading full_name", {"name", "full_name"}),
         ("3. contract (later release): drop name", {"full_name"})]
for label, cols in steps:
    old_ok = works(CODE["v41"], cols)
    print(f"  {label:42} v41 {'ok' if old_ok else 'BROKEN (no rollback past here)'}, v42 ok")

Naive: one migration renames the column
  schema before      : v41 ok, v42 BROKEN
  schema after rename: v41 BROKEN, v42 ok
Expand / migrate / contract
  1. expand: add full_name, write both       v41 ok, v42 ok
  2. backfill, deploy v42 reading full_name  v41 ok, v42 ok
  3. contract (later release): drop name     v41 BROKEN (no rollback past here), v42 ok

Root cause

Code and schema are deployed by different mechanisms, at different moments, and can't be rolled back together: a migration that drops or renames something is a one-way door. During any rolling update both versions run at once, and after a rollback the old code meets the new schema. A migration is only safe if the previous code version keeps working with it.

The fix

Split breaking schema changes into backward-compatible steps across releases (the expand and contract or parallel change pattern):

1. Expand: add the new column (nullable or with a default); the code writes both columns.
2. Migrate: backfill old rows in batches; switch reads to the new column.
3. Contract: in a later release, after you no longer need to roll back past step 2, stop writing the old column and drop it.

ALTER TABLE customers ADD COLUMN full_name text;
-- app v42 writes name AND full_name, reads full_name with a fallback to name
UPDATE customers SET full_name = name WHERE full_name IS NULL AND id BETWEEN :start AND :end;  -- batched backfill job

ALTER TABLE customers DROP COLUMN name;

Also: make migrations additive by default and review destructive ones explicitly; run long backfills as background jobs, not inside the deploy; and use online schema-change tools for large tables (gh-ost, pt-online-schema-change for MySQL; PostgreSQL's CREATE INDEX CONCURRENTLY).

Trade-offs

• Three releases instead of one, and temporary dual-write code. That's the price of being able to roll back at every step.
• Feature flags help to switch reads gradually, but they don't remove the need for a compatible schema.
• Some changes (e.g. changing a primary key type) need a full parallel table and a cut-over plan.

Detect it in production

• CI check: flag DROP, RENAME and type changes in migrations and require a "contract" label plus an approval.
• Test rollback in staging: deploy N, migrate, roll back to N-1, run smoke tests.
• Alert on SQL errors like "column does not exist" right after deploys.

Key points

• Code can be rolled back; destructive migrations can't. Old and new code must both work with every schema state.
• Expand (additive), migrate (backfill, switch reads), contract (drop later).
• Test rollbacks, not just deploys.