5.Kafka Performance Optimization

Kafka performance is usually a balance between throughput, latency, and durability. The best settings depend on whether the system is optimized for fast ingestion, low-latency delivery, or strong safety guarantees. In practice, tuning works best when you look at the producer, consumer, and broker together.

Producer performance is heavily influenced by batching, compression, and acknowledgment settings. linger.ms and batch.size control how long the producer waits to gather records before sending them. A small linger window, such as 5 to 20 milliseconds, and a batch size in the range of 512 KB to 1 MB can improve throughput by reducing network overhead.

Compression can also make a big difference. Snappy is often a balanced default, LZ4 is usually very fast, and Zstandard often gives the best compression ratio for large-volume topics. The right choice depends on whether your workload is more constrained by network bandwidth or CPU.

The acks setting determines how much durability the producer waits for. acks=0 is fastest but offers no durability guarantee, acks=1 waits for the leader only, and acks=all waits for all in-sync replicas. For most production systems, acks=all with min.insync.replicas=2 is a safer choice.

Consumer throughput depends on both fetch settings and processing speed. fetch.min.bytes tells the broker to wait for more data before replying, which improves efficiency. fetch.max.wait.ms limits that wait so latency stays under control.

The real bottleneck is often the work done after records are fetched. Slow database calls or blocking API calls can make a consumer fall behind even if Kafka itself is healthy. Using max.poll.records to batch downstream writes can improve efficiency, and disabling auto-commit is usually safer for stateful processing.

In a well-designed consumer, record processing should be as predictable as possible. If each message triggers a slow external call, the consumer will often spend more time waiting than working. In that case, buffering, batching, or asynchronous handling can improve throughput more than Kafka tuning alone.

Broker performance depends on thread counts, disk layout, cache behavior, and JVM tuning. num.network.threads and num.io.threads may need to be increased for clusters that handle high request volume or disk-heavy workloads. Kafka usually depends on the OS page cache, so forcing synchronous flushes improves durability but usually reduces throughput.

Disk layout is also important. Multiple log directories on separate physical disks can improve parallel I/O, and NVMe SSDs are much better than spinning disks for high-throughput clusters. For JVM settings, G1GC with a moderate heap and short pause targets is a common starting point, and GC pauses should be watched together with consumer lag.

Broker tuning should always be guided by measurements rather than guesswork. If consumers are lagging, the cause may be broker saturation, slow disks, or just poor application code. Looking at CPU, disk latency, GC pauses, and lag together usually gives the clearest picture.