Scaling Stopped at the Quota

Scenario

Example team: a ticketing company prepares for a concert sale expected to bring 20 times normal traffic. Load tests ran in a separate, bigger test account. Production autoscaling is configured with high maximums.

What you see in production

At 10:00 the sale opens. Auto scaling tries to launch 300 more instances and fails with VcpuLimitExceeded; the Lambda functions that process orders hit the account's concurrency quota and are throttled. Requests queue, time out, and the sale is moved to the next day.

The naive code

resource "aws_autoscaling_group" "web" {
  min_size = 10
  max_size = 400            # but the account's vCPU quota only allows about 150 of these instances
}

Reproduce it

Autoscaling limits mean nothing if the account quota is lower:

VCPUS_PER_INSTANCE = 8
QUOTAS = {"on-demand vCPUs": 1_280, "lambda concurrency": 1_000}

def plan(peak_rps):
    instances = peak_rps // 200                     # each instance handles about 200 req/s
    lambda_concurrency = peak_rps * 0.4 * 0.5      # 40% of requests call an order function taking 0.5 s
    needs = {"on-demand vCPUs": instances * VCPUS_PER_INSTANCE, "lambda concurrency": int(lambda_concurrency)}
    for name, need in needs.items():
        quota = QUOTAS[name]
        status = "OK" if need <= quota * 0.8 else "NEAR LIMIT" if need <= quota else "EXCEEDED"
        print(f"  {name:18} need {need:6,} of quota {quota:6,} -> {status}")

for rps in [3_000, 60_000]:
    print(f"peak {rps:,} req/s")
    plan(rps)

peak 3,000 req/s
  on-demand vCPUs    need    120 of quota  1,280 -> OK
  lambda concurrency need    600 of quota  1,000 -> OK
peak 60,000 req/s
  on-demand vCPUs    need  2,400 of quota  1,280 -> EXCEEDED
  lambda concurrency need 12,000 of quota  1,000 -> EXCEEDED

Root cause

Cloud accounts have quotas (service limits) per region: vCPUs per instance family, Lambda concurrency, load balancers, IP addresses, API request rates. Many are adjustable but require a request that can take hours or days; some are fixed. The test account had different quotas than production, so the test proved nothing about production limits.

The fix

1. Inventory the quotas your architecture depends on and compare them with planned peak usage (with headroom).
2. Request increases early (weeks before launches) through Service Quotas; Azure and Google Cloud have quota requests too.
3. Alarm on usage versus quota at 70-80% (Service Quotas integrates with CloudWatch usage metrics).
4. Load test in an account with production quotas, or verify production quotas explicitly.
5. Protect critical functions with reserved concurrency so non-critical ones can't consume the whole pool, and throttle or queue non-critical traffic.

Trade-offs

• Raising quotas increases the blast radius of runaway code or compromised credentials; pair it with budgets and alerts.
• Reserved concurrency guarantees capacity for one function by taking it from others.
• Some limits are hard (per-partition throughput, API rate limits): design around them with queues and sharding.

Detect it in production

• Usage-versus-quota alarms for the top quotas per region and account.
• Errors such as VcpuLimitExceeded, TooManyRequestsException, ThrottlingException and QuotaExceeded in logs and metrics.
• Launch checklists that include quota review.

Key points

• Quotas are per account and region; autoscaling maximums don't override them.
• Review and raise quotas well before peaks; alarm at 70-80% of quota.
• Test with production quotas and protect critical paths with reserved capacity.