0

Folks, I am trying to understand Idempotent Producer. I am using Kafka 2.13

  1. I read https://www.cloudkarafka.com/blog/apache-kafka-idempotent-producer-avoiding-message-duplication.html

  2. I went thru Kafka Spring boot course that said the following should be set up for Idempotent producer

     configProps.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
 configProps.put(ProducerConfig.ACKS_CONFIG, "all");
 configProps.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 5);
 configProps.put(ProducerConfig.RETRIES_CONFIG, Integer.MAX_VALUE);

  3. I also came to know that upon settings enable.idempotence=true rest of the other above configs are set automatically

  4. With #2 I was not able to produce unique messages in Spring boot

  5. Hence I wanted to try in console producer. I read How to enable idempotence in kafka-console-producer? and tried the below.

enter image description here

  1. As you see in the screenshot I am able to send two Apple - 01 and O - 01 messages. I was expecting the Producer to send these messages once.

Kindly let me know how can I make the producer remove duplicates? Any leads are much appreciated!

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Hari Rao
  • 2,990
  • 3
  • 21
  • 34
  • 1
    These are actually different messages for kafka only the content is the same. The feature is not intended to deduplicate messages for you. https://stackoverflow.com/questions/58894281/difference-between-idempotence-and-exactly-once-in-kafka-stream – Andreas Feb 16 '22 at 19:59
  • Please show your Java code that tried to produce two messages – OneCricketeer Feb 16 '22 at 20:27
  • @Andreas I read the post in the link. Thank you. I understand that if the ack is not delivered to the producer, producer is going to retry by sending the same message. Now the Broker is going to be clever and is not going to commit the offset rather will be sending the ack again. This way broker processes the message only once. – Hari Rao Feb 17 '22 at 16:12

0 Answers0