0

I'm trying to create a priority queue using Kafka Consumer, I read about Bucket pattern,and how to use it in one Topic, distributing partitions between the Buckets.

But what I need is to do here, is using different topics high, medium, low ensure I will only consume from High as long as there are message in the topic to be consumed

Kafka Consumer provide a way to pass a list of topics to the consumers but it seems he is doing round-robin balancing between topics

Here a code example that I did to prove it

https://github.com/politrons/reactive/blob/master/kafka/src/test/java/com/politrons/kafka/KafkaOrdering.java

What I would like to know is, if exist any mechanism in Kafka Consumer to tell him, when he is subscribing to several topics, that he have to focus on one, and no read anything from another topic, if this first one is still getting events.

Regards.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
paul
  • 12,873
  • 23
  • 91
  • 153
  • See if you can get multiple topic names from [`ConsumerPartitionAssignor`](https://kafka.apache.org/30/javadoc/org/apache/kafka/clients/consumer/ConsumerPartitionAssignor.html) – OneCricketeer Jan 20 '22 at 18:05
  • Hi, sorry but I don’t understand your answer – paul Jan 20 '22 at 18:24
  • This interface is responsible for assigning partitions to consumers for its subscribed topics. Yes, the default is round-robin, but you can override that by using this interface and [`partition.assignment.strategy`](https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html#consumerconfigs_partition.assignment.strategy). Now, of course, this doesn't "stop" the "low/medium" topic consumer when a "high" event is detected (you need to know topic name information for this), so I don't really think one consumer with three topics is correct approach here – OneCricketeer Jan 20 '22 at 18:26
  • What would you recommend? Have topics I’m afraid is mandatory – paul Jan 20 '22 at 19:08
  • I'm not saying you need less topics. I'm saying you need multiple consumers (at least 3 threads, and even more for all partitions). You then need external communication / RPC between all the threads to pause/stop "lower consumers" when the "higher priority" consumer starts reading data... Out of the box, Kafka has nothing like this, and I don't know any framework that does – OneCricketeer Jan 20 '22 at 20:46
  • Ok, that’s exactly the design that I had in mind. Thanks! – paul Jan 20 '22 at 20:48
  • But, if you did want to build it into the Consumer API, I still think digging into interfaces like `ConsumerPartitionAssignor` + `ConsumerRebalanceListener` + `ConsumerInterceptor` might be useful – OneCricketeer Jan 20 '22 at 20:52
  • Have you taken a look at this one: https://stackoverflow.com/a/66013251/4602706 – Marco Vargas Feb 04 '22 at 20:25

0 Answers0