Kafka Message Delivery Guarantees

Kafka delivery semantics describe how many times a record may be processed by a consumer. The three common models are at most once, at least once, and exactly once. Each one trades off simplicity, performance, and safety in a different way.

In real systems, the choice depends on how expensive duplication is versus how expensive loss is. For logging and telemetry, duplicates may be acceptable. For payments or inventory, stronger guarantees are usually needed.

At most once delivery means a message is processed zero or one time, but never more than once. This usually happens when the consumer commits the offset before it finishes processing. If the application crashes after the commit but before the work is complete, the message will not be read again.

This model is often used when speed matters more than completeness. It is also commonly seen when auto-commit is enabled, because offsets may be committed before the application has actually finished processing the records. That makes the model simple, but it also means some messages can be skipped during failures.

At most once is acceptable only when small data loss is tolerable. For example, a non-critical metrics pipeline may prefer this approach to avoid extra processing overhead. For business-critical workflows, it is usually too risky.

At least once delivery means every message is delivered, but some messages may be processed more than once. In this model, the consumer processes the message first and commits the offset afterward. If a crash happens before the commit, Kafka will send the same message again.

This is the most common pattern in production because it avoids data loss. The trade-off is that downstream systems must handle duplicates safely. Idempotent writes, upserts, and merge-based updates are common techniques for making repeated processing harmless.

A good example is an order status update. If the same event arrives twice, an idempotent database update should still leave the order in the correct final state. That is why at least once delivery is often paired with idempotent downstream logic.

Exactly once delivery means each message is processed one time without loss or duplication. Kafka supports this through idempotent producers and transactional producers. Idempotent producers prevent duplicate writes caused by retry attempts, while transactional producers can atomically commit multiple writes and offsets together.

Consumers using read_committed only see records from committed transactions. This gives stronger consistency inside Kafka itself, especially when a pipeline writes to one topic and then updates another. However, true end-to-end exactly once still depends on external systems also supporting idempotent or transactional behavior.

Exactly once is very useful in stream processing and financial workflows. Still, it adds complexity, so it should be used when the business value of correctness is high enough to justify the operational cost.