How should you design health checks for services behind a load balancer?

Summary

Make load balancer and readiness checks answer 'can this instance serve now?' using only local state, keep liveness checks trivial, and monitor shared dependencies separately. Health checks that call the database or third-party APIs fail on every instance at once during a dependency blip, removing all targets.

Longer Answer

Use separate endpoints (/livez, /readyz). Handle dependency failures in the request path with timeouts, circuit breakers and graceful degradation. Remember an AWS ALB fails open when all targets are unhealthy, and alert on sudden drops in healthy host count.

Key Points

• LB/readiness: local ability to serve.
• Liveness: process only.
• No shared dependencies in checks.
• Degrade gracefully instead.

Deep dive: Cloud Networking in Production: The Health Check That Took Everything Down

What interviewers look for

Avoiding correlated failures and distinguishing probe types.