6. Kafka Producers Explained

A producer is the application that sends data into Kafka. In the bookstore example, the checkout service becomes the producer every time a customer places an order. The producer builds the record, decides where it should go, and sends it to Kafka through the broker.

A Kafka record usually includes a key, a value, and optional headers. The key helps Kafka choose a partition, and the value holds the actual business data. After the message is sent, the producer can get confirmation later through a callback instead of waiting in a blocking way.

With Python and confluent-kafka, the flow is straightforward: create a producer, call produce(), and then call flush() before the script exits. That last step matters because data may still be buffered in memory if the program ends too early. The simple rule is connect, send, and flush.

Producer settings control the balance between speed, safety, and duplicate protection. Three of the most important are acks, retries, and enable.idempotence. These settings help Kafka behave well when a broker is slow or the network is unstable.

acks defines how many acknowledgments are needed before Kafka treats a message as safely written. acks=0 is the fastest but least safe, acks=1 waits for the leader only, and acks=all waits for all in-sync replicas. For important business events like orders, acks=all is usually the safest choice.

retries tells Kafka how many times to resend a failed message, and enable.idempotence=True helps prevent duplicates during those retries. That is especially helpful when delivery must be reliable without writing the same record twice. For order events, that combination is a strong starting point.

Kafka only stores bytes, so application data has to be turned into bytes before it can be sent. That process is called serialization. On the consumer side, those bytes are turned back into useful data again through deserialization.

For beginners, JSON is the easiest format to start with. It is readable, widely supported, and simple to create from Python dictionaries. Avro and Protobuf are better when teams need stronger schemas and stricter contracts across services.

In the bookstore example, the producer can convert an order dictionary into JSON before sending it. The consumer then converts that JSON back into a Python dictionary with json.loads(). That is one of the most common Kafka patterns you will see in real projects.