The Region Failover Nobody Had Practised

Scenario

Example company: a payments API with an SLA of 99.95%. Its primary region is us-east-1; a "DR region" us-west-2 has an Aurora global database secondary and a copy of the infrastructure code, but no running app servers. A regional service disruption hits the primary region's control plane and several services for hours.

What you see in production

The team decides to fail over after 40 minutes of debate. Then: the Terraform for us-west-2 hasn't been applied in months and fails on a renamed variable; secrets aren't replicated; the DNS record has a TTL of one hour; and the third-party payment processor only allow-lists the primary region's NAT IPs. Service is restored in the DR region five hours later, roughly when the primary recovered anyway.

The naive code

DR plan: "run terraform apply in us-west-2 and change DNS"   (never executed)
api.pay.example  A  ALIAS -> us-east-1 load balancer, TTL 3600, no health check

Reproduce it

Failover time is the sum of every step, and each untested step is slow:

steps_unpractised = [("decide to fail over", 40), ("apply DR infrastructure (fails, fix)", 95),
                     ("restore secrets and config", 50), ("promote database", 5),
                     ("partner allow-lists new IPs", 90), ("DNS caches expire (TTL 1 h)", 60)]
steps_practised = [("automated decision (health checks + runbook)", 5), ("warm standby already running", 0),
                   ("secrets replicated", 0), ("promote database (managed failover)", 2),
                   ("partners pre-approved both regions", 0), ("DNS failover, TTL 60 s", 2)]

for label, steps in [("unpractised", steps_unpractised), ("practised", steps_practised)]:
    total = sum(m for _, m in steps)
    print(f"{label}: {total} minutes")
    for name, minutes in steps:
        if minutes:
            print(f"   {minutes:3} min  {name}")

unpractised: 340 minutes
    40 min  decide to fail over
    95 min  apply DR infrastructure (fails, fix)
    50 min  restore secrets and config
     5 min  promote database
    90 min  partner allow-lists new IPs
    60 min  DNS caches expire (TTL 1 h)
practised: 9 minutes
     5 min  automated decision (health checks + runbook)
     2 min  promote database (managed failover)
     2 min  DNS failover, TTL 60 s

Root cause

The DR region was a document, not a running, tested system. Every manual step had hidden dependencies (stale code, unreplicated secrets, external allow-lists, DNS TTLs), and the decision itself had no criteria, so it took 40 minutes. For many workloads multi-region isn't even necessary; for this one it was, but it was never exercised.

The fix

1. Decide per workload (DR options) whether multi-region is needed (RTO/RPO, regulatory and SLA requirements). Most workloads are fine with multi-AZ plus cross-region backups.
2. For those that need it: warm standby or active-active, deployed continuously by the same pipeline to both regions (so the DR stack is never stale), with replicated data (Aurora Global Database, DynamoDB global tables, Cosmos DB multi-region, Spanner multi-region) and replicated secrets.
3. Automate the switch: DNS failover with health checks (Route 53 failover records, Azure Traffic Manager / Front Door, Google Cloud global load balancing) and short TTLs, or Application Recovery Controller routing controls; written criteria for when to fail over.
4. Remove external single-region assumptions: partners allow-list both regions' egress IPs; certificates and DNS records exist in both.
5. Rehearse failover and failback regularly in production-like conditions, measuring time per step.

Trade-offs

• Warm standby or active-active roughly doubles parts of the infrastructure cost and adds data-consistency questions (replication lag, conflicts).
• Automated failover risks flapping on false alarms; use health checks from several locations and require confirmation for data-store promotion.
• Failback is often harder than failover (data written in the DR region must flow back); plan it too.

Detect it in production

• Continuous deployment status per region: the DR region runs the same version.
• Replication lag metrics for global databases.
• Scheduled failover exercises with measured RTO and RPO, and findings tracked to closure.

Key points

• Multi-region is a choice per workload, not a default.
• If you need it, run it continuously: same pipeline, replicated data and secrets, automated health-checked routing.
• Rehearse failover and failback; an unpractised DR plan won't meet its RTO.