Affects Version/s: 1.1.0
Fix Version/s: None
One of the sources is creating a long lag, and hence the data from other sources are not getting processed until the lag is cleared. Is there a way to make a queue take priority?
The customer would like to have Kafka use some type of priority Queue so it would avoid the FIFO nature of Kafka's topics. This is regarding the ATLAS_HOOK in which the customer want other resources besides Hive h
ook to have a faster processing.
Basically the idea is to have some resource to publish messages in different levels of priority.
Atlas would have to consumer these messages accordingly to the Queue priority.
Topics or prioritizing messages for the ATLAS_HOOK topic.
note: Hive is the bigger publisher, They have a LAG of about a week on the amount of messages produced by Hive. Currently the LAG produce by Hive impacts the other services publishers;
"Single-Event Processing" - book:Kafka: The Definitive Guide: Real-Time Data And Stream Processing At Scale