The Zone Outage We Didn't Survive

Scenario

Example team: a booking platform runs its app tier in an auto scaling group across three AZs, with RDS Multi-AZ and a load balancer in all three zones. The architecture diagram says "multi-AZ everywhere".

What you see in production

One availability zone has a power event for 90 minutes. The app still goes down for all users: the single NAT gateway lived in that zone (no outbound calls to the payment provider), the self-managed Redis on one EC2 instance was there (sessions lost), and the auto scaling group couldn't launch replacement capacity fast enough because the other zones were already at their maximum size.

The naive code

NAT gateway:     1 (zone a)                 <- every private subnet routes through it
Redis:           1 EC2 instance (zone a)    <- sessions, rate limits
ASG:             min 6, max 6 across a/b/c  <- no headroom when a zone disappears

Reproduce it

A system is only as available as its weakest serial dependency:

components = {
    "load balancer":   {"a", "b", "c"},
    "app instances":   {"a", "b", "c"},
    "database (Multi-AZ)": {"a", "b"},        # primary in a, standby in b
    "NAT gateway":     {"a"},
    "redis (1 EC2)":   {"a"},
}

def survives(lost_zone):
    return {name: bool(zones - {lost_zone}) for name, zones in components.items()}

for zone in ["a", "b"]:
    result = survives(zone)
    down = [n for n, ok in result.items() if not ok]
    print(f"lose zone {zone}: {'SERVICE DOWN, single-zone: ' + ', '.join(down) if down else 'service survives'}")

capacity_needed, per_zone_after_loss = 6, 3   # 6 instances needed; 2 zones x 3 = 6 only if max allows it
print(f"headroom: need {capacity_needed} instances in 2 zones -> max_size must allow {per_zone_after_loss} per zone")

lose zone a: SERVICE DOWN, single-zone: NAT gateway, redis (1 EC2)
lose zone b: service survives
headroom: need 6 instances in 2 zones -> max_size must allow 3 per zone

Root cause

"Multi-AZ" applied to the main tiers but not to supporting components that are easy to forget: NAT gateways, caches, self-managed brokers, bastions, DNS resolvers, license servers. And the design assumed it could launch new capacity during a zone failure, when every other customer is doing the same and control-plane actions may be impaired.

The fix

1. Inventory every dependency in the request path and check its zone footprint: one NAT gateway per zone with per-zone route tables; managed, multi-AZ caches (ElastiCache with replicas and automatic failover); brokers with replicas across zones.
2. Static stability: run enough capacity in the remaining zones to absorb the loss without launching anything (for three zones, about 150% of normal capacity spread as 50% per zone).
3. Zonal isolation and shifting: keep traffic within a zone where possible, and practise moving traffic away from an impaired zone (Route 53 Application Recovery Controller zonal shift).
4. Game days: simulate a zone loss (AWS Fault Injection Service, Azure Chaos Studio) and watch what breaks.

Trade-offs

• Static stability costs extra capacity (often 30-50% more), which is the price of not depending on launches during an event.
• A NAT gateway per zone multiplies hourly NAT costs; it's still small compared with an outage.
• Zone-local traffic patterns reduce cross-zone cost but need balanced capacity per zone.

Detect it in production

• An automated check listing resources and their zones; flag request-path components present in only one zone.
• Regular zone-failure game days with measured impact.
• Alerts on capacity per zone versus what's needed if one zone is lost.

Key points

• "Multi-AZ" is only true if every component in the request path is.
• Pre-provision enough capacity to lose a zone without launching anything (static stability).
• Practise zone failures and know how to shift traffic away.