Timeouts to One Partner API

Scenario

Example team: 300 worker pods in private subnets send order events to a payment partner's API at api.partner.example:443, which resolves to a single IP. Each event opens a new HTTPS connection and closes it. Traffic leaves through one NAT gateway per zone.

What you see in production

During peak hours about 3% of calls time out while connecting, only to that partner. Other destinations are fine, the partner says its API is healthy, and retries make it worse. The NAT gateway's ErrorPortAllocation metric, which nobody watched, is non-zero exactly during the incidents.

The naive code

import requests

def send_event(event):
    # a new TCP + TLS connection for every call; closed immediately afterwards
    return requests.post("https://api.partner.example/v1/events", json=event, timeout=10)

Reproduce it

Source NAT needs a unique (public IP, source port) for every concurrent connection to the same destination IP, port and protocol. Closed connections also hold their port for a while:

PORTS_PER_IP = 55_000            # AWS NAT gateway: simultaneous connections per unique destination, per IP
SECONDS_HELD = 0.3 + 15          # a fresh connection holds its port while open plus a cleanup period (assumed 15 s)
POOL_SIZE = 200                  # kept-alive connections shared by all workers

for calls, reuse in [(1_500, False), (4_000, False), (4_000, True)]:
    connections = POOL_SIZE if reuse else int(calls * SECONDS_HELD)
    for nat_ips in [1, 2]:
        capacity = PORTS_PER_IP * nat_ips
        state = "OK" if connections <= capacity else f"EXHAUSTED ({connections - capacity} connections fail)"
        print(f"{calls} calls/s, reuse={reuse!s:5}, {nat_ips} NAT IP(s): {connections:6} of {capacity} ports -> {state}")

1500 calls/s, reuse=False, 1 NAT IP(s):  22950 of 55000 ports -> OK
1500 calls/s, reuse=False, 2 NAT IP(s):  22950 of 110000 ports -> OK
4000 calls/s, reuse=False, 1 NAT IP(s):  61200 of 55000 ports -> EXHAUSTED (6200 connections fail)
4000 calls/s, reuse=False, 2 NAT IP(s):  61200 of 110000 ports -> OK
4000 calls/s, reuse=True , 1 NAT IP(s):    200 of 55000 ports -> OK
4000 calls/s, reuse=True , 2 NAT IP(s):    200 of 110000 ports -> OK

Root cause

The NAT gateway translates every connection to one of its public IPs and a source port. To a single destination (IP, port, protocol), one public IP offers only about 55,000 simultaneous connections on AWS. Opening a fresh connection per request multiplies the ports needed, because each closed connection keeps its translation for a cleanup period. When ports run out, new connection attempts are dropped, so clients see connect timeouts, and retries add even more connections.

The fix

1. Reuse connections. Use a session or connection pool with keep-alive, so hundreds of requests share a few connections:

import requests
from requests.adapters import HTTPAdapter

session = requests.Session()
session.mount("https://", HTTPAdapter(pool_connections=4, pool_maxsize=50))

def send_event(event):
    return session.post("https://api.partner.example/v1/events", json=event, timeout=(2, 10))

import { Agent, request } from "undici";

const partner = new Agent({ connections: 50, keepAliveTimeout: 30_000 });
export const sendEvent = (event: object) =>
  request("https://api.partner.example/v1/events", { method: "POST", body: JSON.stringify(event), dispatcher: partner });

// One shared HttpClient per application: it pools and reuses connections.
static final HttpClient CLIENT = HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(2)).build();

2. Add NAT addresses. An AWS NAT gateway can have up to 8 IP addresses (one primary plus secondary), multiplying the connections per destination. Azure NAT gateway accepts up to 16 public IPs; Google Cloud NAT lets you set ports per VM and add addresses (see Cloud NAT ports and addresses).
3. Spread or bypass: separate NAT gateways per workload, or a private connection (PrivateLink, a partner VPN) for very high volumes.
4. Retry with backoff and jitter, never in a tight loop.

Trade-offs

• Connection pools need sensible limits and idle timeouts below the NAT's own idle timeout (350 s on AWS), or long-idle connections are silently dropped.
• More NAT IPs means more addresses for the partner to allow-list.
• Long-lived connections concentrate load; make sure the client still re-resolves DNS occasionally if the partner changes IPs.

Detect it in production

• Alarm on NAT ErrorPortAllocation > 0 and track ActiveConnectionCount and ConnectionAttemptCount (Azure: SNAT connection count and failed connections; GCP: Cloud NAT dropped packets with OUT_OF_RESOURCES).
• Client-side: connect-timeout rate per destination, and new connections per second versus requests per second.

Key points

• Source NAT has a per-destination connection limit (about 55,000 per IP on AWS); fresh connections per request burn through it.
• Reuse connections first, then add NAT addresses or bypass the NAT.
• Watch the port-allocation error metric; connect timeouts to one destination are the classic symptom.