[ATLAS-3169] Create priority Queue for Atlas messages coming from 2 different sources;

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Duplicate
• Affects Version/s: 1.1.0
• Component/s: atlas-intg
• Assignee: Unassigned
• Reporter: Ernani Pereira de Mattos Junior
• Created: 24/Apr/19 20:58
• Updated: 07/Jan/20 09:28
• Resolved: 07/Jan/20 09:28

Description

One of the sources is creating a long lag, and hence the data from other sources are not getting processed until the lag is cleared. Is there a way to make a queue take priority?
==

The customer would like to have Kafka use some type of priority Queue so it would avoid the FIFO nature of Kafka's topics. This is regarding the ATLAS_HOOK in which the customer want other resources besides Hive h
ook to have a faster processing.

Basically the idea is to have some resource to publish messages in different levels of priority.

1) high_priority_queue
2) low_priority_queue

Atlas would have to consumer these messages accordingly to the Queue priority.

Business Case:

Topics or prioritizing messages for the ATLAS_HOOK topic.

HIVECLUSTER_ATLAS – publish --> KAFKA-ATLAS_HOOK – subscribed --> ATLAS-1.1
OTHER_Services– publish --> KAFKA-ATLAS_HOOK – subscribed --> ATLAS-1.1

note:  Hive is the bigger publisher, They have a LAG of about a week on the amount of messages produced by Hive. Currently the LAG produce by Hive impacts the other services publishers;

Some References:
https://stackoverflow.com/a/30686523
https://cwiki.apache.org/confluence/display/KAFKA/KIP-349%3A+Priorities+for+Source+Topics - KAFKA-6690
"Single-Event Processing" - book:Kafka: The Definitive Guide: Real-Time Data And Stream Processing At Scale

Issue Links

• is duplicated by: ATLAS-3313 Enhance Atlas server to process notifications from multiple Kafka topics