Cloud Computing Tutorial · Compute · Lesson 14 of 24
Load Balancing and Scaling Patterns
Put load balancers in front of fleets, scale stateless tiers horizontally, and protect stateful tiers.
A typical web architecture
users -> CDN -> load balancer (multi-AZ) -> stateless app tier (auto scaling, multi-AZ)
-> cache -> managed database (Multi-AZ, read replicas)
-> queue -> workers (auto scaling on queue depth)
- Stateless app tiers scale horizontally: sessions and files live in shared stores, so any instance can serve any request.
- Databases scale differently: bigger instances, read replicas, caching, then partitioning.
- Queues absorb spikes: accept work fast, process it at a steady rate.
import math
def workers_needed(queue_depth, msgs_per_worker_per_min, target_minutes):
return max(1, math.ceil(queue_depth / (msgs_per_worker_per_min * target_minutes)))
for depth in [500, 12_000, 90_000]:
n = workers_needed(depth, msgs_per_worker_per_min=60, target_minutes=5)
print(f"queue depth {depth:6,}: {n:3} workers to drain in about 5 minutes")
Output
queue depth 500: 2 workers to drain in about 5 minutes queue depth 12,000: 40 workers to drain in about 5 minutes queue depth 90,000: 300 workers to drain in about 5 minutes
Scaling signals
| Tier | Scale on |
|---|---|
| Web/API | Requests per target, CPU, latency |
| Workers | Queue depth or age of oldest message |
| Kubernetes pods | HPA on CPU or custom metrics; KEDA for event sources |
Load balancers themselves are managed and scale automatically (ALB, Azure Application Gateway/Front Door, Google Cloud Load Balancing), but sudden huge spikes may need pre-warming conversations with the provider.
Test yourself
What makes an app tier easy to scale horizontally?
Keeping it stateless, with sessions and files in shared storesStoring sessions in local memoryUsing bigger instancesPinning users to one server
Show answer
Answer: Keeping it stateless, with sessions and files in shared stores
If any instance can serve any request, adding instances adds capacity.