Kafka in Production · Capstone
Capstone: An Event-Driven Order Pipeline
Design Kafka topics, keys, producers, consumers, failure handling and operations for an e-commerce order pipeline.
Scenario
An e-commerce company is moving from a monolith to services. When an order is placed, payment, inventory, shipping, email and analytics services must react. Peak traffic is 3,000 orders per minute during sales. The finance team needs an exact record of every payment state change.
Requirements
- Every order event is processed by every interested service, in order per order.
- No order is charged twice or shipped twice.
- A slow or failing service must not block the others.
- Events can be replayed to rebuild a service's state or to debug.
Constraints
- Kafka (3 brokers), PostgreSQL per service, Python and Java services on Kubernetes.
Deliverables
- Topics and keys.
- Producing events reliably from the order service.
- Consumers and failure handling.
- Exactly-once concerns.
- Operations.
Hints
- Writing to the database and to Kafka are two separate systems.
- Think about which identifier keeps related events in order.
Draft each deliverable below before revealing the model answer and rubric.
Further reading: Kafka documentation · Transactional outbox pattern
Deliverables
Which topics and keys do you use?
Model answer
orders(OrderPlaced, OrderCancelled),payments(PaymentAuthorised, PaymentFailed, Refunded),shipments; keyed byorder_idso each order's events stay ordered.- Partitions sized for peak plus growth (for example 24-48), replication factor 3,
min.insync.replicas=2; retention 7-30 days, or longer (or compaction for state topics) where replay is needed. - Versioned schemas in a registry with backward compatibility.
Rubric
Look for entity-based keys for ordering, partition counts sized ahead, replication settings and schema governance.
How does the order service produce events reliably?
Model answer
- Transactional outbox: write the order and an outbox row in one database transaction; a relay (Debezium CDC or a poller) publishes outbox rows to Kafka with an idempotent producer (
acks=all). - Event IDs in each message for deduplication downstream; ordering per order through the key.
Rubric
Strong answers avoid dual writes with an outbox and use idempotent, durable producers.
How do consumers handle failures without blocking others?
Model answer
- One consumer group per service, so services progress independently.
- Commit after processing (at-least-once); bounded retries with backoff for transient errors, retry topics for longer delays, dead-letter topic with error headers for poison messages; alert on DLQ size.
- Processing kept within
max.poll.interval.ms; slow external calls handled asynchronously with bounded concurrency per partition.
Rubric
Look for independent groups, commit-after-processing, retries plus DLQ, and attention to poll intervals.
How do you avoid double charging and double shipping?
Model answer
- Idempotent consumers: a processed-events table with a unique event ID, written in the same database transaction as the effect; payment provider calls with idempotency keys.
- Kafka transactions (read_committed) for Kafka-to-Kafka steps such as enrichment services.
Rubric
Strong answers combine durable deduplication, provider idempotency keys and Kafka transactions where they apply.
How do you operate the pipeline?
Model answer
- Dashboards and alerts: consumer lag per group and partition, DLQ sizes, producer error rates, under-replicated partitions, broker disk.
- Replay runbook: reset a group's offsets to a timestamp (
kafka-consumer-groups --reset-offsets --to-datetime) for a new or rebuilt service. - Capacity: load-test sales peaks; schema compatibility checks in CI; access control per topic.
Rubric
Look for lag-based alerting, a replay procedure, capacity testing and governance.
Open in the app to mark lessons complete, take notes and track your progress.