How should retries and dead-letter queues work?

Summary

Retry transient failures with exponential backoff and jitter up to a limit; after max attempts, move the message to a dead-letter queue (DLQ) for inspection and replay instead of blocking the queue or losing it. Poison messages (that always fail) must not block other messages.

Longer Answer

Distinguish transient (timeouts, 503) from permanent errors (validation), sending permanent ones straight to the DLQ. Alert on DLQ growth and provide tooling to replay after fixes.

Key Points

• Backoff + jitter
• Max attempts → DLQ
• Don't block on poison messages
• Alert and replay tooling