Cloud Networking in Production · Capstone Case Studies
Capstone: Postmortem — Intermittent Checkout Timeouts
Reconstruct an incident from metrics and configuration, find every contributing cause, and write the remediation plan.
Scenario
Example company: an e-commerce platform. After switching to a new payment provider on Monday, checkout shows intermittent timeouts during evening peaks: 2-6% of payment calls fail with connect timeouts, and a few hundred per million API requests return 502. Revenue impact over the week: about $180,000.
Evidence
- Checkout runs as 400 pods across three private subnets; outbound traffic goes through one NAT gateway (in zone a) for all zones, created that way to save cost.
- The new provider's API resolves to one IP; the old provider used several behind DNS round robin.
- The payment client creates a new HTTPS connection per call (
requests.postwithout a session). - NAT gateway
ErrorPortAllocationis non-zero every evening from 19:00 to 22:00. - Checkout pods run Gunicorn with default keep-alive (2 s) behind an ALB with idle timeout 60 s; the 502s predate the switch but nobody had investigated.
- When zone a had a brief networking event on Thursday, checkout failed in all zones.
Deliverables
- The root cause(s) of the payment timeouts, and of the 502s.
- Why did the switch of provider trigger it?
- Immediate and long-term fixes.
- What monitoring would have caught this before customers did?
Draft each deliverable before revealing the model answer and rubric.
Further reading: NAT gateway troubleshooting · ALB troubleshooting
Deliverables
What are the root causes?
Model answer
- Payment timeouts: NAT port exhaustion. Hundreds of pods open a new connection per payment call to a single destination IP:443, all through one NAT gateway with one IP, exceeding the ~55,000 simultaneous connections per destination at peak. New connections are dropped, so clients see connect timeouts.
- 502s: an unrelated, older problem: Gunicorn closes idle keep-alive connections after 2 s while the ALB reuses them for up to 60 s.
Rubric
Full marks: identifies per-destination NAT port exhaustion as the timeout cause (citing ErrorPortAllocation and the single destination IP) and separates the 502s as a keep-alive mismatch.
Why did changing provider trigger it?
Model answer
The old provider's traffic was spread over several IPs, so the per-destination port limit was never reached. The new provider uses one IP, concentrating every connection on one destination tuple. The connection-per-call client and single NAT were already fragile; the change removed the hidden headroom.
Rubric
Full marks: explains that port limits are per unique destination and that spreading across IPs had masked the problem.
Immediate and long-term fixes.
Model answer
Immediate: share a connection pool (requests.Session with keep-alive) in the payment client; add secondary IPs to the NAT gateway (and tell the provider the new IPs); set Gunicorn keepalive = 65.
Long term: one NAT gateway per zone (the single NAT was also a zone-wide single point of failure, as Thursday showed); retries with backoff and jitter and a circuit breaker on payments; if volume keeps growing, a private connection to the provider (PrivateLink if offered) or dedicated egress IPs per workload.
Rubric
Full marks: connection reuse first, NAT capacity second, per-zone NAT for resilience, keep-alive fix, and safe retries.
What monitoring would have caught this earlier?
Model answer
- Alarm on NAT
ErrorPortAllocation> 0 and onActiveConnectionCountapproaching the per-destination limit. - Connect-timeout rate per downstream dependency in the payment client.
- ALB 502 count compared with app 5xx count (a gap means connection-level errors).
- A pre-launch load test against the new provider at peak connection rates.
Rubric
Full marks: NAT metrics, per-dependency client metrics, LB-vs-app error comparison, and load testing provider changes.
Open in the app to mark lessons complete, take notes and track your progress.