Topics, Partitions and Offsets

What you'll build

An orders topic with three partitions, messages produced with customer IDs as keys, and a consumer that prints each message with its partition and offset, showing how keys keep each customer's events in order. Every example in this course was run against Apache Kafka 4.4 (KRaft mode, no ZooKeeper); the output shown is what it printed.

The idea

• A topic is an append-only log split into partitions. Each message in a partition has an increasing offset.
• Messages with the same key go to the same partition, so they're ordered relative to each other. There's no ordering across partitions.
• Partitions are the unit of parallelism: a consumer group can have at most one active consumer per partition.
• Messages stay for the retention period (or size), whether or not they've been read; consumers track their own position.

The code

# the Kafka CLI tools, run inside the broker container
kafka() { docker exec -i rc-course-kafka /opt/kafka/bin/kafka-"$1".sh --bootstrap-server localhost:19092 "${@:2}"; }

kafka topics --create --topic orders --partitions 3 --replication-factor 1 > /dev/null
kafka topics --describe --topic orders | grep -o 'PartitionCount: [0-9]*'
printf 'customer-1:order 1 created\ncustomer-2:order 2 created\ncustomer-1:order 1 paid\ncustomer-3:order 3 created\ncustomer-1:order 1 shipped\n' |
  kafka console-producer --topic orders --reader-property parse.key=true --reader-property key.separator=:
kafka console-consumer --topic orders --from-beginning --max-messages 5 --timeout-ms 10000 \
  --formatter-property print.key=true --formatter-property print.partition=true --formatter-property print.offset=true 2>/dev/null |
  sed -E 's/\t/ | /g' | sort -t'|' -k1,1 -k2,2
kafka topics --delete --topic orders

PartitionCount: 3
Partition:0 | Offset:0 | customer-1 | order 1 created
Partition:0 | Offset:1 | customer-1 | order 1 paid
Partition:0 | Offset:2 | customer-1 | order 1 shipped
Partition:1 | Offset:0 | customer-3 | order 3 created
Partition:2 | Offset:0 | customer-2 | order 2 created

All of customer-1's events landed in one partition, in order (offsets 0, 1, 2). The other customers' events are in other partitions; their order relative to customer-1's isn't defined.

Common mistakes

• Expecting global ordering across a topic; order is only per partition, so key by the entity whose events must stay ordered.
• Too few partitions, limiting consumer parallelism; or thousands of partitions per small topic, adding overhead.
• Using Kafka as a database without thinking about retention; old data disappears after the retention period unless compaction is used.

Key points

• Topics are partitioned logs; offsets identify messages within a partition.
• The key decides the partition and therefore ordering.
• Partition count caps consumer parallelism.

Further reading: Kafka introduction · Kafka design: the log