How to assign priorities to KAFKA TOPICS such that all consumers in all consumer groups first read from higher priority topics and if there are no messages in higher priority topics then it should shift to lower priority topics. Also how to make it preemptive such that whenever new messages arrives in Higher priority topic, consumer should shift again to them.
Asked
Active
Viewed 568 times
0
-
1There's no such setting for the consumer api – OneCricketeer May 30 '21 at 13:32
-
Isn't it there any implementation for this case? I think priority queue should be the solution for the case, but I am unable to code through this and getting wrong outputs. – Krati Jain May 31 '21 at 02:55
-
You're welcome to edit your question to include what you've tried – OneCricketeer May 31 '21 at 14:14
-
Please take a look at this one, it may be what you are looking for: https://stackoverflow.com/a/66013251/4602706 – Marco Vargas Feb 04 '22 at 20:28
1 Answers
1
You should implement that through code, one way is setting one topic with different partitions , and implement The PartitionAssignor interface , sending "high" priority key to majority of partitions , "meduim" to less and "low" to lesser, like 10,5,3 , the consumer that way will avoid starvation situations and your "high" messages would have higher processing attention
Ran Lupovich
- 1,655
- 1
- 6
- 13
-
I had implemented that case (bucket priority pattern) but I' m stuck with this case specifically where I need the consumers to switch between topics if there are no messages in higher priority ones. And should switch again in case of any incoming messages. – Krati Jain May 31 '21 at 02:51
-
This is why using one topic with partitions using consumer group should give you better implementation solution – Ran Lupovich May 31 '21 at 03:31