Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25151

Apply Apache Commons Pool to KafkaDataConsumer

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.0
    • 3.0.0
    • Structured Streaming
    • None

    Description

      KafkaDataConsumer contains its own logic for caching InternalKafkaConsumer which looks like can be simplified via applying Apache Commons Pool. Benefits of applying Apache Commons Pool are following:

      • We can get rid of synchronization of KafkaDataConsumer object while acquiring and returning InternalKafkaConsumer.
      • We can extract the feature of object pool to outside of the class, so that the behaviors of the pool can be tested easily. Now it doesn't have detailed tests and only covers reported issues.
      • We can get various statistics for the object pool, and also be able to enable JMX for the pool.

      Attachments

        Issue Links

          Activity

            People

              kabhwan Jungtaek Lim
              kabhwan Jungtaek Lim
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: