Uploaded image for project: 'Atlas'
  1. Atlas
  2. ATLAS-3305

Unable to scale atlas kafka consumers

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.1.0, 2.0.0
    • Fix Version/s: None
    • Component/s: atlas-core, atlas-intg
    • Labels:

      Description

      We wanted to scale kafka consumers for atlas, as we are getting many lineage messages and processing them just with one consumer is not enough. 

       

      There is parameter atlas.notification.hook.numthreads to scale consumers in  

      NotificationHookConsumer.

      But the method:

       

      notificationInterface.createConsumers(NotificationType.HOOK, numThreads)

       

      is always returning one element list, which effectively always starts one consumer

      List<NotificationConsumer<T>> consumers = Collections.singletonList(kafkaConsumer);

       

      Log incorrectly says that nuber of consumers has been created:

      LOG.info("<== KafkaNotification.createConsumers(notificationType={}, numConsumers={}, autoCommitEnabled={})", notificationType, numConsumers, autoCommitEnabled)

        Attachments

        1. multiple_consumers_perf.png
          61 kB
          Adam Rempter
        2. ATLAS-3305_multiple_kafka_consumers.patch
          7 kB
          Adam Rempter

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              arempter Adam Rempter
            • Votes:
              3 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 10m
                10m