7. Kafka Consumers Explained

Consumer groups let multiple consumers share the work of reading a topic. Kafka assigns partitions so that each partition is processed by only one consumer inside the group. That creates parallelism without duplicate work inside the same group.

If a topic has three partitions and a consumer group has three active consumers, each consumer can read one partition. If the group has more consumers than partitions, the extra consumers do not receive any partitions and will sit idle. That is why partition count matters so much when planning scale.

Consumer groups are also what make Kafka flexible for different use cases. One group can support shipping, another can support analytics, and another can support search indexing. Each group reads the same events independently and maintains its own offsets.

Offsets are Kafka’s way of tracking how far a consumer has read in a partition. When a consumer processes records, it commits the offset so Kafka knows where to continue later. This is what makes consumer restarts possible without starting from zero.

Offsets can be committed automatically or manually. Automatic commits are simple, but manual commits give more control because you can wait until processing is fully complete before advancing the offset. For important workflows, manual commit is usually the safer approach.

Kafka stores committed offsets internally, so there is no need for a separate database just to remember progress. That makes recovery easier after a failure or restart. If the consumer comes back later, it can continue from its last committed position.

Not every consumer needs to read data the same way. Some want the full history, some want only new messages, and some want to replay a specific range after fixing a bug. Kafka supports all of those styles.

A consumer can start from the earliest retained record if it needs historical data. It can also start from the latest record if it only cares about new events going forward. For example, analytics might read from the beginning, while a live dashboard may only want fresh order updates.

This flexibility is one of Kafka’s strongest beginner-level ideas once offsets make sense. The same topic can support both live processing and historical replay. That gives the bookstore system a lot of room to grow without changing how data is produced.