3. How Apache Kafka Works

Kafka moves data from producer to broker, where it is written into a partition log, and then later a consumer reads that same data from the partition. The broker acts as a durable middle layer the whole time. That gap between writing and reading is one of Kafka’s most useful ideas.

Because messages stay in Kafka for a retention period, consumers can be slow or even offline without losing data. They only fall behind temporarily. In the bookstore example, shipping and analytics can read the same order event later, each at its own pace.

Publishing a message starts when the producer serializes the data into bytes and chooses a partition. It then sends the message to the broker responsible for that partition and waits for acknowledgment, depending on the durability settings. The broker writes the record to its local log and may also wait for replicas before confirming success.

That process is what gives Kafka its reliability. For example, a bookstore order should not be treated as complete until the broker has safely accepted it. If a broker fails right after the write, replication helps keep the record from disappearing.

Kafka consumers pull data instead of having it pushed to them. They poll for records, process the batch they receive, and then commit their offset after the work is done. That gives each consumer control over its own speed.

If a consumer crashes, it can restart from the last committed offset instead of starting over. That makes Kafka much more resilient to restarts, deployments, and network issues. It also makes it easy to add more consumers later without changing the producer side.