Design a Notification System

The question

"Design a notification service other teams can call to send push notifications, emails and SMS to users."

Clarify the requirements

• Functional: an API to send a notification to a user on one or more channels (push, email, SMS); templates; user preferences (opt-outs, quiet hours); scheduled sends; delivery status.
• Scale: 50 million notifications/day on normal days, bursts of 10 million in minutes (a marketing campaign or breaking news).
• Non-functional: transactional messages (password reset, 2FA codes) arrive within seconds; no duplicates; nothing lost; marketing never delays transactional messages.
• Out of scope: building the email/SMS delivery itself (we use providers such as an email API and an SMS gateway; push goes through Apple's push notification service and Google's Firebase Cloud Messaging).

Estimate the scale

• Average: 50M/day ≈ 600/s; campaign burst: 10M in 10 minutes ≈ 17,000/s.
• Providers have their own limits (e.g. a few thousand SMS/s per account), so the system must queue and pace rather than send everything at once.
• Storage: 50M × ~1 KB status records ≈ 50 GB/day; keep 30–90 days.

API and data model

POST /notifications
{ "idempotency_key": "order-123-shipped", "user_id": 42, "template": "order_shipped",
  "data": {"order": "123"}, "channels": ["push", "email"], "priority": "transactional" }
-> 202 { "notification_id": "..." }

GET /notifications/{id}   -> per-channel status: queued | sent | delivered | failed

notifications: id | idempotency_key (unique) | user_id | template | data | priority | created_at
deliveries:    notification_id | channel | status | attempts | provider_id | last_error
preferences:   user_id | channel | category | enabled | quiet_hours
devices:       user_id | push_token | platform | last_seen

High-level design

callers ─► notification API ─► store request (dedupe on idempotency key)
                │
                ├─► check preferences, render template
                │
                └─► queues by channel and priority:
                      push.high  push.low  email.high  email.low  sms.high
                          │         │          │          │          │
                       push workers     email workers         sms workers   (rate-limited per provider)
                          │                 │                     │
                      APNs / FCM       email provider          SMS gateway
                          └──────── status callbacks ──► deliveries table

Separate queues per channel (a slow SMS provider doesn't block push) and per priority (a campaign never delays a 2FA code).

Deep dives

1. Retries without duplicates. Workers retry failed sends with exponential backoff and jitter, up to a limit, then move the message to a dead-letter queue for inspection. Because a worker can crash after sending but before recording success, the provider call should carry an idempotency key where supported, and our own record is keyed on (notification_id, channel).

import random

random.seed(7)        # deterministic for the demo


def backoff(attempt, base=1.0, cap=300.0):
    '''Full jitter: a random wait up to base * 2^attempt, capped.'''
    return random.uniform(0, min(cap, base * 2 ** attempt))


def send_with_retries(provider_results, max_attempts=5):
    '''provider_results: what the provider returns on each try.'''
    waited = 0.0
    for attempt, result in enumerate(provider_results[:max_attempts]):
        if result == "ok":
            return f"sent on attempt {attempt + 1} after waiting {waited:.1f}s"
        if result == "invalid_token":
            return "permanent failure: remove the device token, don't retry"
        waited += backoff(attempt)          # temporary failure: wait and retry
    return f"gave up after {max_attempts} attempts -> dead-letter queue"


print(send_with_retries(["ok"]))
print(send_with_retries(["timeout", "503", "ok"]))
print(send_with_retries(["invalid_token"]))
print(send_with_retries(["503"] * 10))

sent on attempt 1 after waiting 0.0s
sent on attempt 3 after waiting 0.6s
permanent failure: remove the device token, don't retry
gave up after 5 attempts -> dead-letter queue

The jitter matters: if a provider has a 1-minute outage, thousands of workers retrying at exactly 1, 2 and 4 seconds would hit it in synchronized waves.

2. Preferences and quiet hours. Check opt-outs before queueing (and again just before sending, for scheduled messages). Transactional messages bypass marketing opt-outs; quiet hours delay non-urgent pushes until morning in the user's time zone.

3. Pacing campaigns. A scheduler releases a 10-million-user campaign into the low-priority queues in batches; per-provider rate limiters in the workers keep within provider limits.

4. Device tokens. Push tokens go stale when apps are uninstalled. Providers answer "invalid token"; delete those tokens immediately so you stop paying for failed sends.

Trade-offs

• At-least-once + dedup is realistic; exactly-once delivery across external providers isn't.
• One queue vs many: more queues isolate channels and priorities at the cost of operational complexity.
• Rendering before vs after queueing: rendering early keeps workers simple; rendering late picks up template fixes and fresh data.
• Fallbacks: if push fails for an urgent message, fall back to SMS, which costs money; make it opt-in per notification type.

What interviewers look for

• Asynchronous processing with queues separated by channel and priority.
• Idempotency keys at the API and per delivery.
• Retries with backoff and jitter, dead-letter queues.
• Respecting user preferences and provider rate limits.

Key points

• Accept fast (202), queue per channel and priority, send with workers.
• Idempotency key on every request; record per (notification, channel).
• Exponential backoff with jitter; permanent errors aren't retried.
• Transactional traffic must never wait behind marketing.