Operating Kafka: Lag, Retention and Compaction

What you'll build

The everyday operational views: consumer group lag after a consumer falls behind, per-topic retention settings, and a compacted topic that keeps the latest value per key (like a changelog of customer profiles).

The idea

• Consumer lag = latest offset minus committed offset, per partition. Growing lag means consumers can't keep up; alert on it.
• Retention (retention.ms, retention.bytes) deletes old segments; set it per topic according to replay needs.
• Compaction (cleanup.policy=compact) keeps at least the latest message per key; a message with a null value (a tombstone) deletes the key.
• Durability: replication factor 3 with min.insync.replicas=2 and producers using acks=all.

The code

# the Kafka CLI tools, run inside the broker container
kafka() { docker exec -i rc-course-kafka /opt/kafka/bin/kafka-"$1".sh --bootstrap-server localhost:19092 "${@:2}"; }

kafka topics --create --topic audit-log --partitions 2 --replication-factor 1 --config retention.ms=604800000 > /dev/null
seq 1 50 | kafka console-producer --topic audit-log
kafka console-consumer --topic audit-log --group reporting --from-beginning --max-messages 20 --timeout-ms 10000 > /dev/null 2>&1
kafka consumer-groups --describe --group reporting 2>/dev/null |
  awk 'NR > 1 && $2 == "audit-log" { lag += $6; print "partition", $3, "lag", $6 } END { print "total lag:", lag }' | sort

kafka topics --create --topic customer-profiles --partitions 1 --replication-factor 1 \
  --config cleanup.policy=compact --config min.cleanable.dirty.ratio=0.01 --config segment.ms=100 > /dev/null
kafka configs --describe --topic customer-profiles --entity-type topics 2>/dev/null | tr ',' '\n' | grep -E 'cleanup.policy|segment.ms' | sed 's/^ *//'
kafka configs --describe --topic audit-log --entity-type topics 2>/dev/null | tr ',' '\n' | grep -o 'retention.ms=[0-9]*'
kafka topics --delete --topic audit-log
kafka topics --delete --topic customer-profiles
kafka consumer-groups --delete --group reporting > /dev/null 2>&1
echo "cleaned up"

partition 0 lag 30
partition 1 lag 0
total lag: 30
cleaned up

The reporting group read 20 of 50 messages, so 30 are lagging. In production, export lag per group and partition to your monitoring system (for example with a Kafka exporter or your platform's metrics) and alert on sustained growth.

Common mistakes

• Monitoring broker health but not consumer lag, the first sign of a stuck or slow consumer.
• One retention setting for every topic; audit logs and click streams need different durations.
• Using compaction for topics without keys, or expecting compaction to happen immediately.
• Replication factor 1 (or min.insync.replicas=1) in production, risking data loss on a single broker failure.

Key points

• Watch consumer lag per group and partition.
• Set retention and cleanup policy per topic.
• Use replication factor 3, min.insync.replicas 2 and acks=all for important data.

Further reading: Kafka operations · Log compaction · Topic configs