Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-8122

Change kafka metadata obtaining due to KAFKA-5697

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 1.17.0, 1.18.0, 1.19.0
    • 1.20.0
    • None
    • None

    Description

      Starting from kafka 2.0 was changed behavior of the KafkaConsumer#poll . In Drill was used one hack, where poll(0) is used only to update consumer metadata. But after the next change: https://github.com/apache/kafka/pull/4855 this hack is not working now as poll() is not blocked more until metadata update.

      Unfortunately, Kafka doesn't have public method to only update assignment metadata in consumer and it's on the discussion now whether to add it or not: KIP-505 .  Once it will be introduced we can use it but until that it needs to implement a workaround.

      Code to change:

      https://github.com/apache/drill/blob/15b2f52260e4f0026f2dfafa23c5d32e0fb66502/contrib/storage-kafka/src/main/java/org/apache/drill/exec/store/kafka/KafkaGroupScan.java#L185
       
       
       

       

      Attachments

        Activity

          People

            mrymar Maksym Rymar
            mrymar Maksym Rymar
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: