Uploaded image for project: 'Atlas'
  1. Atlas
  2. ATLAS-3169

Create priority Queue for Atlas messages coming from 2 different sources;

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Critical
    • Resolution: Duplicate
    • 1.1.0
    • None
    • atlas-intg
    • None

    Description

      One of the sources is creating a long lag, and hence the data from other sources are not getting processed until the lag is cleared. Is there a way to make a queue take priority?
      ==

      The customer would like to have Kafka use some type of priority Queue so it would avoid the FIFO nature of Kafka's topics. This is regarding the ATLAS_HOOK in which the customer want other resources besides Hive h
      ook to have a faster processing.

      Basically the idea is to have some resource to publish messages in different levels of priority.

      1) high_priority_queue
      2) low_priority_queue

      Atlas would have to consumer these messages accordingly to the Queue priority.

      Business Case:

      Topics or prioritizing messages for the ATLAS_HOOK topic.

      HIVECLUSTER_ATLAS – publish --> KAFKA-ATLAS_HOOK – subscribed --> ATLAS-1.1
      OTHER_Services– publish --> KAFKA-ATLAS_HOOK – subscribed --> ATLAS-1.1

      note:  Hive is the bigger publisher, They have a LAG of about a week on the amount of messages produced by Hive. Currently the LAG produce by Hive impacts the other services publishers;

      Some References:
      https://stackoverflow.com/a/30686523
      https://cwiki.apache.org/confluence/display/KAFKA/KIP-349%3A+Priorities+for+Source+Topics - KAFKA-6690
      "Single-Event Processing" - book:Kafka: The Definitive Guide: Real-Time Data And Stream Processing At Scale

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              emattosHWX Ernani Pereira de Mattos Junior
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: