2. Apache Kafka Architecture Explained

Kafka brokers are the servers that do the actual work of handling reads and writes. Each broker owns a set of partitions, stores their data on disk, and serves requests for those partitions. Brokers also coordinate with one another so the cluster stays balanced and healthy.

If one broker fails, the others take over leadership for the affected partitions. That is what keeps Kafka available even when individual machines have problems. In the bookstore setup, one broker might handle partition 0 of the orders topic while another handles partition 1, which spreads load and storage across the cluster.

Topics are Kafka’s way of grouping related events, and every topic is split into one or more partitions. A partition is an ordered, append-only log, which is what gives Kafka its ability to scale reads and writes efficiently. Records always stay in order within a partition.

Across different partitions, though, Kafka does not promise global ordering. That matters when the sequence of events for one customer or one order must be preserved. In the bookstore, using customer ID as the key helps keep all of that customer’s events in the same partition.

Producers are the applications that write records into Kafka topics, while consumers are the applications that read them. A producer decides which partition a record should go to, and a consumer keeps track of its offset so it can continue from the right place after a restart. That makes both sides independent and easy to scale.

The important part is that producers and consumers never talk to each other directly. Everything passes through Kafka brokers, which means either side can be restarted or replaced without breaking the other. In the bookstore flow, checkout acts as the producer and shipping acts as the consumer.

Kafka used to rely on ZooKeeper to manage cluster metadata such as live brokers, partitions, and leaders. Today, Kafka uses KRaft, where controllers inside Kafka manage that metadata directly without an external coordination service. That simplifies the system and makes startup and metadata handling faster.

KRaft also reduces the number of moving parts you have to install and maintain. For new Kafka users, the key thing to remember is that KRaft is the current direction, while ZooKeeper is mostly a legacy context. You may still see ZooKeeper mentioned in older articles or older clusters, but it is no longer the modern default.