4. Kafka Storage and Retention

Kafka stores data as an append-only log. Records are written sequentially to the end of each partition, which is much faster than random writes. This storage model is one of the main reasons Kafka can support very high throughput.

Each partition is broken into log segments on disk. When a segment reaches its configured size, Kafka closes it and starts a new one. Index files help Kafka find offsets quickly without scanning the entire log from the beginning.

The append-only model is efficient because modern disks and operating systems handle sequential writes well. Kafka also relies heavily on the operating system page cache, which keeps frequently accessed data in memory. As a result, many reads do not need to hit disk directly.

This architecture makes Kafka feel more like a durable distributed log than a traditional message queue. It is a good fit for event streaming because the same records can be consumed multiple times by different applications. That reuse is part of what makes Kafka so useful for modern data pipelines.

Kafka can remove old data using time-based retention, size-based retention, or both. Time-based retention deletes records older than a configured age, while size-based retention removes the oldest segments once storage grows beyond a limit. Kafka applies whichever retention rule triggers first.

Retention settings can be changed without restarting brokers. Longer retention is useful for replay, reprocessing, and debugging. Shorter retention saves disk space, but it also limits how far back consumers can recover if they fall behind.

If a consumer falls behind past the earliest retained offset, it can no longer resume from that point. That is why retention planning should match the recovery needs of the application. A topic used for audit or replay usually needs a much longer retention period than a transient telemetry stream.

Log compaction keeps the latest record for each key and removes older versions of the same key. Unlike time-based retention, the goal here is not to preserve every event. Instead, Kafka preserves the newest known value for each key so systems can recover current state.

This is especially useful for changelog topics and state recovery in Kafka Streams. It is also common in CDC pipelines where the latest database row state matters more than the full update history. Compaction runs in the background, so older versions may remain for a while before being cleaned up.

A good mental model is that retention answers “how long should data stay?”, while compaction answers “which versions should stay?”. In many systems, the two are used for different topics because they solve different problems. Event topics often use retention, while state topics often use compaction.