Random 502s Behind the Load Balancer

Scenario

Example team: a Node.js API runs behind an AWS Application Load Balancer with default settings. Traffic is uneven: busy minutes followed by quiet ones.

What you see in production

A steady trickle of 502 Bad Gateway responses, a few hundred per million requests, every day. The app's logs show no errors for those requests; they never reached the handler. The ALB's HTTPCode_ELB_502_Count rises while the app's own 5xx count stays at zero.

The naive code

import express from "express";

const app = express();
app.listen(8080);   // Node's server.keepAliveTimeout defaults to 5 seconds
// ALB idle timeout: 60 seconds (default)

Reproduce it

The load balancer reuses idle connections for up to its idle timeout. If the app closes a connection first, the load balancer normally notices the close, but a request it sends in the same few milliseconds hits a closed socket:

import math

RACE_WINDOW = 0.003    # seconds: the request and the app's FIN cross on the wire
MEAN_IDLE = 8          # average seconds a backend connection sits idle between requests

def p_idle_between(a, b):                    # idle gaps follow an exponential distribution
    return math.exp(-a / MEAN_IDLE) - math.exp(-b / MEAN_IDLE)

def failed_per_million(app_keepalive, lb_idle):
    if app_keepalive >= lb_idle:
        return 0.0                           # the load balancer always closes first: no race
    return p_idle_between(app_keepalive, app_keepalive + RACE_WINDOW) * 1_000_000

for app_keepalive in [2, 5, 30, 65]:
    rate = failed_per_million(app_keepalive, lb_idle=60)
    print(f"app keep-alive {app_keepalive:2}s, ALB idle 60s -> {rate:5.0f} 502s per million requests")

app keep-alive  2s, ALB idle 60s ->   292 502s per million requests
app keep-alive  5s, ALB idle 60s ->   201 502s per million requests
app keep-alive 30s, ALB idle 60s ->     9 502s per million requests
app keep-alive 65s, ALB idle 60s ->     0 502s per million requests

Root cause

Both sides of a keep-alive connection have an idle timeout. The ALB keeps idle backend connections for up to 60 seconds and assumes they are usable; Node.js closes them after 5 seconds. When the app's FIN and the ALB's next request cross on the wire, the request arrives at a closed socket, and the ALB returns 502. The same race happens with any proxy pair: nginx to Gunicorn, Envoy to a JVM, an API gateway to a VPC service.

The fix

The backend's keep-alive timeout must be longer than the proxy's idle timeout, so the proxy always closes first:

const server = app.listen(8080);
server.keepAliveTimeout = 65_000;   // longer than the ALB's 60 s idle timeout
server.headersTimeout = 66_000;     // must be greater than keepAliveTimeout

# gunicorn.conf.py
keepalive = 65            # seconds; the default (2) is far below the ALB's 60
# uvicorn: --timeout-keep-alive 65

keepalive_timeout 65s;

Then check the other timeouts in the chain: client to load balancer, the load balancer's idle timeout, and the app's request timeout (a request longer than the idle timeout is cut off with 504).

Trade-offs

• Longer keep-alive holds more idle connections and file descriptors on the backend; that is cheap compared with failed requests.
• Lowering the load balancer's idle timeout instead also works, but affects slow clients and long requests.
• Retries on 502 hide the symptom but can duplicate non-idempotent requests.

Detect it in production

• Compare load balancer 502 counts with application 5xx counts: 502s with no matching app errors point at connection handling.
• ALB access logs: 502s with target_processing_time = -1 (the target never answered) and elb_status_code=502.
• A small, constant 502 rate that doesn't follow deploys or load is a strong hint.

Key points

• Proxy and backend each time out idle connections; the backend must wait longer than the proxy.
• ALB idle timeout is 60 s by default; Node.js closes after 5 s and Gunicorn after 2 s unless configured.
• 502s with no application errors are usually connection-level, not code bugs.