The Migration That Took Two Days

Scenario

Example team: an API moves from an old load balancer to a new one in another account. The plan: at 10:00 Saturday, change the api.shop.example record, then delete the old stack at noon. The record's TTL is 172800 (two days), set years ago.

What you see in production

At noon the old stack is deleted and errors spike: a large share of clients still connect to the old address. Some mobile users fail until Monday. A Java service keeps calling the old IP until it is restarted, and a typo during the change briefly produced NXDOMAIN, which some resolvers kept caching for an hour after it was fixed.

The naive code

api.shop.example.   172800   IN   A   203.0.113.10      ; TTL 2 days, never revisited
; Saturday 10:00: change to 198.51.100.20, delete the old stack at 12:00

Reproduce it

Clients keep the answer they cached until their copy expires, so after a change the old address keeps getting traffic for up to the old TTL:

import random

random.seed(1)

def share_on_old(hours_after_change, ttl_seconds, clients=10_000):
    # each client cached the record at a random moment within the TTL before the change
    on_old = 0
    for _ in range(clients):
        cached_at = -random.uniform(0, ttl_seconds)
        expires = cached_at + ttl_seconds
        if expires > hours_after_change * 3600:
            on_old += 1
    return on_old / clients

for ttl in [172_800, 3_600, 60]:
    row = [f"{share_on_old(h, ttl):6.1%}" for h in [0.05, 2, 24]]
    print(f"TTL {ttl:>7}s  still on old address after 3 min / 2 h / 24 h: {' '.join(row)}")

TTL  172800s  still on old address after 3 min / 2 h / 24 h:  99.9%  95.8%  49.9%
TTL    3600s  still on old address after 3 min / 2 h / 24 h:  94.9%   0.0%   0.0%
TTL      60s  still on old address after 3 min / 2 h / 24 h:   0.0%   0.0%   0.0%

Root cause

Resolvers and clients cache a record for its TTL, and they don't hear about changes. With a two-day TTL, the change takes up to two days to reach everyone, so deleting the old target after two hours cut off most clients. Some clients add their own caching: the JVM caches lookups (30 seconds by default, but forever when a security manager is installed or networkaddress.cache.ttl=-1 is set), long-lived connection pools never re-resolve, and negative answers (NXDOMAIN) are cached too, for the zone's SOA negative-caching time.

The fix

1. Lower the TTL in advance: at least one old-TTL period before the change, set it to 60-300 seconds.
2. Shift gradually with weighted records (Route 53 weighted routing, Azure Traffic Manager weighted, Cloud DNS weighted round robin): 5%, 25%, 100%, watching errors.
3. Keep the old target serving until its traffic falls to zero (watch its request count), then delete it.
4. Fix client caching: set a finite JVM DNS TTL, and recycle pooled connections periodically (a maximum connection age) so they re-resolve.
5. Raise the TTL again afterwards (for example to 3600) to reduce query load.

# one week before:      api.shop.example.  300  IN A 203.0.113.10
# cut-over (weighted):  api  A 203.0.113.10 weight 95 ; api  A 198.51.100.20 weight 5
# java:                 -Dsun.net.inetaddr.ttl=60   (or networkaddress.cache.ttl=60 in java.security)

Trade-offs

• Low TTLs mean more DNS queries and more dependence on DNS availability; use them around changes, not permanently, unless you need fast failover.
• Weighted shifts need both stacks running and compatible during the transition.
• Some clients ignore TTLs entirely; plan a long tail and measure it rather than assuming.

Detect it in production

• Request count on the old target after the change: delete only when it reaches zero.
• Synthetic lookups from several public resolvers during the change.
• Errors from clients with long-lived pools (they keep failing until restarted).

Key points

• A DNS change takes up to the old TTL to reach everyone; lower TTLs well before migrations.
• Shift traffic with weighted records and keep the old target until its traffic is gone.
• Watch for client-side caches: JVM settings, connection pools and cached NXDOMAIN answers.