Free course · 10 lessons · Advanced
Kafka in Production
Build and run event-driven systems on Apache Kafka 4: topics, partitions, keys and offsets, reliable producers with acks and idempotence, consumer groups and rebalancing, offset commits and delivery semantics, retries and dead-letter topics, transactions and exactly-once processing, partitioning and hot partitions, schema evolution, and operations: consumer lag, retention and compaction. Every example was run against a real Kafka 4.4 broker.
1. Kafka Fundamentals for Production
Topics, partitions and offsets; reliable producers; consumer groups and rebalancing; and offset commits and delivery semantics.
- Topics, Partitions and Offsets — Create topics with partitions, see how message keys decide partitions and ordering, and read messages with their partition and offset.
- Reliable Producers: acks, Idempotence and Delivery Reports — Configure producers so acknowledged messages aren't lost or duplicated, check delivery reports, and flush before shutdown.
- Consumer Groups and Rebalancing — Scale consumption with consumer groups, see how partitions are assigned and rebalanced, and keep processing within poll intervals.
- Offset Commits and Delivery Semantics — Commit offsets after processing for at-least-once delivery, see what happens when a consumer crashes, and make processing idempotent.
2. Kafka Patterns and Operations
Retries and dead-letter topics, transactions and exactly-once processing, keys and partitioning, schemas and evolution, and operating Kafka: lag, retention and compaction.
- Retries and Dead-Letter Topics — Handle failing messages without blocking a partition: retry transient errors, and move poison messages to a dead-letter topic with error details.
- Transactions and Exactly-Once Processing — Read, transform and write with a transactional producer so output messages and consumer offsets commit atomically, and read only committed data.
- Keys, Partitioning and Hot Partitions — Choose message keys for ordering and balance, spot hot partitions, and understand why adding partitions breaks key-to-partition mapping.
- Message Schemas and Safe Evolution — Treat message formats as contracts: version them, evolve them compatibly, and validate messages before producing them.
- Operating Kafka: Lag, Retention and Compaction — Monitor consumer lag, configure retention and log compaction per topic, and set replication for durability.
3. Capstone
Design an event-driven system on Kafka.
- Capstone: An Event-Driven Order Pipeline — Design Kafka topics, keys, producers, consumers, failure handling and operations for an e-commerce order pipeline.