The Backfill That Got Us Blocked

Scenario

Example deployment: an FDE connects a customer-support platform to a customer's CRM. The integration must load five years of history (2.3 million records) and then keep up with changes. On Friday evening the FDE starts the backfill with 64 parallel workers to finish over the weekend.

What you see in production

Within minutes the CRM returns 429 Too Many Requests; the workers retry immediately, and at 21:40 the CRM's security system locks the integration user for suspected abuse. The customer's own sales team, sharing the same API quota, can't sync their mobile app all weekend. On Monday the customer's CRM admin asks for the project to be paused.

The naive code

from concurrent.futures import ThreadPoolExecutor

def fetch_page(n):
    while True:
        r = session.get(f"{CRM}/records", params={"page": n, "per_page": 100})
        if r.status_code == 200:
            return r.json()
        # 429? try again right away

with ThreadPoolExecutor(64) as pool:
    pool.map(fetch_page, range(23_000))

Reproduce it

A token bucket models the API's limit; compare an unthrottled backfill with a paced one:

LIMIT_PER_SEC = 20           # the customer's API allows 20 requests per second for this integration
LOCK_AFTER_429S = 500        # their abuse detection locks the user after this many rejections

def run(request_rate, seconds=120):
    tokens, ok, rejected = LIMIT_PER_SEC, 0, 0
    for _ in range(seconds):
        tokens = min(LIMIT_PER_SEC, tokens + LIMIT_PER_SEC)
        for _ in range(request_rate):
            if tokens >= 1:
                tokens -= 1
                ok += 1
            else:
                rejected += 1
        if rejected >= LOCK_AFTER_429S:
            return ok, rejected, "ACCOUNT LOCKED"
    return ok, rejected, "fine"

for name, rate in [("64 workers, instant retries", 300), ("paced at 80% of the limit", 16)]:
    ok, rejected, state = run(rate)
    print(f"{name:28} ok {ok:5}  429s {rejected:6}  -> {state}")
print(f"2.3M records / 100 per page at 16 req/s = {2_300_000 / 100 / 16 / 60:.0f} minutes")

64 workers, instant retries  ok    40  429s    560  -> ACCOUNT LOCKED
paced at 80% of the limit    ok  1920  429s      0  -> fine
2.3M records / 100 per page at 16 req/s = 24 minutes

Root cause

The backfill treated the customer's production API as if it were the integration's own: unlimited parallelism, immediate retries on 429, no knowledge of the quota, and no coordination with the API owner. Retrying instantly turned a rate limit into an attack pattern. The integration also shared a quota with the customer's own users.

The fix

1. Agree the backfill with the API owner: window, rate, and a dedicated integration user or quota if possible.
2. Throttle to a target rate (for example 70-80% of the limit) with a client-side rate limiter, and adapt to rate-limit headers.
3. Back off on 429/503: honour Retry-After; otherwise exponential backoff with jitter, capped; stop and alert after repeated failures.
4. Checkpoint progress (last cursor or page) so the job resumes instead of restarting.
5. Prefer bulk APIs or exports for history, then switch to incremental sync.

import random, time

def get_with_backoff(url, params, max_attempts=6):
    for attempt in range(max_attempts):
        limiter.acquire()                                   # client-side token bucket at ~80% of the quota
        r = session.get(url, params=params, timeout=30)
        if r.status_code not in (429, 503):
            return r
        retry_after = r.headers.get("Retry-After")
        time.sleep(float(retry_after) if retry_after else random.uniform(0, min(60, 2 ** attempt)))
    raise RuntimeError("rate limited repeatedly; pausing backfill for review")

Trade-offs

• A paced backfill takes longer; set expectations with the customer instead of racing the limit.
• Dedicated quotas or bulk exports need the customer's admins, which is why to ask early.
• Adaptive throttling based on headers is more efficient but depends on the API exposing them consistently.

Detect it in production

• Track 429/503 rates per customer integration and alert well before lockout thresholds.
• Backfill progress dashboards (records per minute, ETA, checkpoint position).
• Ask the customer to share their API usage dashboards during the backfill.

Key points

• A customer's API is shared production infrastructure; pace yourself.
• Honour Retry-After, back off with jitter, cap retries, and checkpoint progress.
• Agree large backfills in advance and prefer bulk exports for history.