Kafka Partitions and Replication

Kafka topics are divided into partitions, and those partitions are where Kafka actually spreads work across brokers and consumers. In practice, partitions are important because they let Kafka scale horizontally while still preserving the order of records inside each partition. If you need events for the same customer or device to stay in sequence, sending them to the same partition is the usual approach.

The number of partitions is one of the most important early design decisions, because changing it later is not always simple operationally. A common starting point is to plan for more partitions than the number of consumers you expect to run at the same time. In many real systems, a range of 12 to 48 partitions per topic is often workable, but the right number depends on throughput, consumer parallelism, and broker capacity.

1. Why Partitions Matter

Partitions are the main unit of parallelism in Kafka. More partitions usually mean more opportunity for concurrent reads and writes, but they also add operational overhead. A topic with too few partitions can become a bottleneck, while too many partitions can increase coordination and memory pressure on brokers.

A useful example is an order-processing system. If all events for one order ID must be processed in order, hashing the order ID to a partition keeps that sequence intact. At the same time, different order IDs can still be processed in parallel across multiple partitions.

1. Partitioning Strategies

Kafka chooses a partition differently depending on whether you send a key with the message. If you provide a key, Kafka hashes it and maps that key to a partition, which keeps all records for the same key together. If you do not provide a key, Kafka can use sticky partitioning to improve batching and throughput, but you do not get ordering guarantees across those records.

The key choice matters a lot for both balance and correctness. High-cardinality keys such as customer ID, order ID, or device ID usually spread load well. Low-cardinality keys such as country code or date can create hot partitions, where one partition gets overloaded while others stay underused.

In most applications, the best key is the one that matches the unit of ordering you actually care about. If the business rule says “keep each user’s activity in order,” then user ID is a strong partition key. If the rule is weaker, you may choose a different key to improve throughput.

Kafka uses replication to protect data from broker failures. Each partition is copied to multiple brokers, and one replica becomes the leader that serves reads and writes. The other replicas are followers that copy data from the leader in the background.

If the leader broker fails, Kafka can promote another replica to take over. The replicas that are fully caught up form the ISR, or In-Sync Replica set. This set is important because it tells Kafka which replicas are safe to promote without losing confirmed data.

For stronger durability, producers often use acks=all together with min.insync.replicas=2. That means Kafka confirms a write only after enough in-sync replicas have stored it. This setting reduces the chance of data loss if one broker goes down.