Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-2295

Dynamically loaded classes (encoders, etc.) may not be found by Kafka Producer

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.9.0.0
    • Component/s: producer
    • Labels:
      None

      Description

      Kafka Producer (via CoreUtils.createObject) effectively uses Class.forName to load encoder classes. Class.forName is by design finds classes only in the defining classloader of the enclosing class (which is often the bootstrap class loader). It does not use the current thread context class loader. This can lead to problems in environments where classes are dynamically loaded and therefore may not be present in the bootstrap classloader.

      This leads to ClassNotFound Exceptions in environments like Spark where classes are loaded dynamically using custom classloaders. Issues like this have reported. E.g. -
      https://www.mail-archive.com/user@spark.apache.org/msg30951.html

      Other references regarding this issue with Class.forName
      http://stackoverflow.com/questions/21749741/though-my-class-was-loaded-class-forname-throws-classnotfoundexception

      This is a problem we have faced repeatedly in Apache Spark and we solved it by explicitly specifying the class loader to use. See
      https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/util/Utils.scala#L178

        Attachments

        1. KAFKA-2295_2015-07-06_11:32:58.patch
          7 kB
          Manikumar
        2. KAFKA-2295_2015-08-20_17:44:56.patch
          6 kB
          Manikumar
        3. KAFKA-2295.patch
          4 kB
          Manikumar

          Issue Links

            Activity

              People

              • Assignee:
                omkreddy Manikumar
                Reporter:
                tdas Tathagata Das
                Reviewer:
                Guozhang Wang
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: