AI Engineering Tutorial · Tools and Agents · Lesson 15 of 25
Retries, Timeouts and Rate Limits
LLM APIs fail sometimes: rate limits, overload and timeouts. Retry the right errors with backoff, and set timeouts.
Which errors to retry
| Error | Retry? |
|---|---|
| 429 rate limited | Yes, after the retry-after time or with backoff |
| 500, 529 overloaded / server errors | Yes, with backoff |
| Timeout | Yes, a limited number of times |
| 400 invalid request, 401 bad key | No: fix the request or the key |
Use exponential backoff with jitter (wait 1 s, 2 s, 4 s... plus randomness) and a maximum number of attempts. Official SDKs retry some errors automatically; check their defaults.
import random
def call_with_retries(attempt_results, max_attempts=4, base=1.0, rng=random.Random(4)):
waited = 0.0
for attempt, status in enumerate(attempt_results[:max_attempts], 1):
if status == 200:
return f"success on attempt {attempt} after waiting {waited:.1f}s"
if status in (400, 401):
return f"error {status}: not retryable"
delay = base * 2 ** (attempt - 1) * (0.5 + rng.random() / 2) # backoff with jitter
waited += delay
print(f"attempt {attempt}: {status}, waiting {delay:.1f}s")
return "gave up"
print(call_with_retries([529, 429, 200]))
print(call_with_retries([401]))
Output
attempt 1: 529, waiting 0.6s attempt 2: 429, waiting 1.1s success on attempt 3 after waiting 1.7s error 401: not retryable
See Errors and Rate limits.
Test yourself
Should you retry a 401 error?
Yes, immediatelyYes, with backoffNo, the API key is invalid; retrying won't helpOnly at night
Show answer
Answer: No, the API key is invalid; retrying won't help
Authentication errors need a fix, not a retry.