Cloud Computing Tutorial · Integration and Operations · Lesson 19 of 24
Queues, Topics and Events
Decouple services with queues (SQS, Service Bus, Pub/Sub), publish/subscribe topics and event buses, and handle retries safely.
Building blocks
| Pattern | AWS | Azure | Google Cloud |
|---|---|---|---|
| Queue (one consumer group works through messages) | SQS | Service Bus queues, Storage queues | Pub/Sub (pull subscription) |
| Publish/subscribe fan-out | SNS | Service Bus topics, Event Grid | Pub/Sub |
| Event bus with routing rules | EventBridge | Event Grid | Eventarc |
| Ordered, replayable stream | Kinesis, MSK (Kafka) | Event Hubs | Pub/Sub, Managed Service for Apache Kafka |
At-least-once delivery
Most cloud queues deliver each message at least once. A consumer receives a message, which becomes invisible for the visibility timeout (30 seconds by default on SQS); if the consumer doesn't delete it in time, it becomes visible again and another consumer processes it. So:
- Set the visibility timeout longer than the maximum processing time.
- Make consumers idempotent (processing a message twice has the same effect as once).
- Send messages that keep failing to a dead-letter queue after a few attempts, and alert on it.
def deliveries(visibility_timeout_s, processing_s):
# A consumer receives the message at t=0. Each time the visibility timeout ends before processing
# finishes, the message becomes visible again and another consumer receives it.
times = [0]
while times[-1] + visibility_timeout_s < processing_s:
times.append(times[-1] + visibility_timeout_s)
return times
for vt in [30, 60]:
d = deliveries(vt, processing_s=45)
print(f"visibility {vt}s, processing 45s: delivered {len(d)}x at t={d} -> {'DUPLICATE' if len(d) > 1 else 'once'}")
Output
visibility 30s, processing 45s: delivered 2x at t=[0, 30] -> DUPLICATE visibility 60s, processing 45s: delivered 1x at t=[0] -> once
Test yourself
Consumers take up to 2 minutes per message but the SQS visibility timeout is 30 s. What happens?
Messages reappear and are processed more than onceMessages are lostSQS slows down consumersNothing
Show answer
Answer: Messages reappear and are processed more than once
Set the timeout above the processing time and make consumers idempotent.