Uploaded image for project: 'Atlas'
  1. Atlas
  2. ATLAS-3169

Create priority Queue for Atlas messages coming from 2 different sources;

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Critical
    • Resolution: Duplicate
    • Affects Version/s: 1.1.0
    • Fix Version/s: None
    • Component/s: atlas-intg
    • Labels:
      None

      Description

      One of the sources is creating a long lag, and hence the data from other sources are not getting processed until the lag is cleared. Is there a way to make a queue take priority?
      ==

      The customer would like to have Kafka use some type of priority Queue so it would avoid the FIFO nature of Kafka's topics. This is regarding the ATLAS_HOOK in which the customer want other resources besides Hive h
      ook to have a faster processing.

      Basically the idea is to have some resource to publish messages in different levels of priority.

      1) high_priority_queue
      2) low_priority_queue

      Atlas would have to consumer these messages accordingly to the Queue priority.

      Business Case:

      Topics or prioritizing messages for the ATLAS_HOOK topic.

      HIVECLUSTER_ATLAS – publish --> KAFKA-ATLAS_HOOK – subscribed --> ATLAS-1.1
      OTHER_Services– publish --> KAFKA-ATLAS_HOOK – subscribed --> ATLAS-1.1

      note:  Hive is the bigger publisher, They have a LAG of about a week on the amount of messages produced by Hive. Currently the LAG produce by Hive impacts the other services publishers;

      Some References:
      https://stackoverflow.com/a/30686523
      https://cwiki.apache.org/confluence/display/KAFKA/KIP-349%3A+Priorities+for+Source+Topics - KAFKA-6690
      "Single-Event Processing" - book:Kafka: The Definitive Guide: Real-Time Data And Stream Processing At Scale

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                emattosHWX Ernani Pereira de Mattos Junior
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: