Load Balancers

What a load balancer does

A load balancer spreads incoming connections across several servers (targets), removes unhealthy ones, and gives clients a single stable address. It also terminates TLS in most setups.

Algorithms

• Round robin: each target in turn.
• Least outstanding requests / least connections: the least busy target, which is better when request durations vary.
• Hashing on client IP or a key: the same client goes to the same target (sticky).

import itertools

targets = ["app-1", "app-2", "app-3"]
rr = itertools.cycle(targets)
print("round robin:      ", [next(rr) for _ in range(6)])

active = {"app-1": 12, "app-2": 3, "app-3": 7}
picks = []
for _ in range(4):
    target = min(active, key=active.get)
    active[target] += 1
    picks.append(target)
print("least connections:", picks)

round robin:       ['app-1', 'app-2', 'app-3', 'app-1', 'app-2', 'app-3']
least connections: ['app-2', 'app-2', 'app-2', 'app-2']

Health checks

The load balancer calls a health endpoint (for example GET /healthz) on every target and stops sending traffic to targets that fail. Keep health checks cheap and about this instance: if a health check also tests the shared database, a database blip marks every target unhealthy at once.

Timeouts and 502s

A load balancer keeps idle connections to your app open for reuse, up to its idle timeout (60 seconds by default on an AWS Application Load Balancer). If your app closes idle connections sooner (Node.js's default keep-alive timeout is 5 seconds), the load balancer sometimes sends a request on a connection the app just closed and returns 502. Set the app's keep-alive timeout longer than the load balancer's idle timeout.