The Health Check That Took Everything Down

Scenario

Example team: to make health checks "meaningful", an API's /health endpoint queries the database, calls the payment provider and checks Redis. The load balancer and Kubernetes liveness probes both use it.

What you see in production

The payment provider has a 40-second slowdown. Every instance's /health times out at once. Kubernetes restarts all pods (liveness failures), the load balancer removes every target, and the whole API is down for 6 minutes, including the 90% of endpoints that don't use payments. Restarted pods hammer the database with cold caches and connection storms as they come back.

The naive code

@app.get("/health")
def health():
    db.execute("SELECT 1")
    payments.ping()             # third-party dependency
    redis.ping()
    return {"ok": True}
# used for: ALB target health check AND Kubernetes livenessProbe

Reproduce it

A dependency failure is shared by every instance, so a health check that includes it fails everywhere at the same moment:

instances = [f"api-{i}" for i in range(1, 9)]
dependency_ok = False                     # the payment provider is slow
instance_ok = {name: name != "api-3" for name in instances}   # api-3 really is broken (out of memory)

def deep_check(name):
    return instance_ok[name] and dependency_ok

def shallow_check(name):
    return instance_ok[name]

for label, check in [("deep", deep_check), ("shallow", shallow_check)]:
    healthy = [n for n in instances if check(n)]
    print(f"{label:7} check: {len(healthy)}/{len(instances)} healthy -> ", end="")
    if not healthy:
        print("no targets: outage (or the load balancer fails open to all targets)")
    else:
        print(f"api-3 removed, others keep serving; payment calls degrade gracefully")

deep    check: 0/8 healthy -> no targets: outage (or the load balancer fails open to all targets)
shallow check: 7/8 healthy -> api-3 removed, others keep serving; payment calls degrade gracefully

Root cause

A health check should answer "should this instance get traffic?" Including shared dependencies turns it into "is the whole system healthy?", whose answer is the same for every instance. A partial outage of one dependency becomes a full outage, and liveness probes multiply it by restarting healthy processes. (An AWS ALB fails open when every target is unhealthy and routes to all of them anyway, which can hide or worsen the problem depending on the cause.)

The fix

Separate the checks by purpose:

@app.get("/livez")
def livez():
    return {"ok": True}                       # the event loop answers: the process is alive

@app.get("/readyz")
def readyz():
    if not app.state.warmed_up or app.state.db_pool.exhausted():
        return JSONResponse({"ready": False}, status_code=503)
    return {"ready": True}

Handle dependency failures in the request path instead: timeouts, circuit breakers and graceful degradation ("payments temporarily unavailable" while browsing still works).

Trade-offs

• Shallow checks can keep an instance in rotation while one of its dependencies is broken for it alone (for example a bad local network path); cover that with request error-rate alarms and outlier detection.
• Liveness probes that are too aggressive restart slow-but-healthy processes; give them generous thresholds.

Detect it in production

• Alert when the healthy-host count drops by more than a few instances at once: that's almost always a shared cause.
• Track health-check latency; checks that call dependencies slow down before they fail.
• Review health endpoints in code review: any network call in a liveness probe is a red flag.

Key points

• Health checks decide traffic per instance; shared dependencies make every instance fail together.
• Liveness: process only. Readiness/LB: local ability to serve. Dependencies: monitoring and circuit breakers.
• An ALB fails open when all targets are unhealthy; don't design around it, avoid the situation.