Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-27781

Tried to access method org.apache.avro.specific.SpecificData.<init>()V

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.4.3
    • Fix Version/s: 2.4.4
    • Component/s: Spark Core
    • Labels:
      None

      Description

      It appears that there is a conflict in avro dependency versions at runtime when using Spark 2.4.3 and Scala 2.12 (spark-2.4.3-bin-without-hadoop-scala-2.12 binary distribution) and Hadoop 2.7.7.

       

      Specifically, the Spark 2.4.3 binary distribution for Hadoop 2.7.x includes avro-1.8.2.jar

      $ find spark-2.4.3-bin-hadoop2.7 *.jar | grep avro

      jars/avro-1.8.2.jar

      jars/avro-mapred-1.8.2-hadoop2.jar

      jars/avro-ipc-1.8.2.jar

       

      Whereas the Spark 2.4.3 binary distribution for Scala 2.12 without Hadoop does not

      $ find spark-2.4.3-bin-without-hadoop-scala-2.12 *.jar | grep avro

      jars/avro-mapred-1.8.2-hadoop2.jar

       

      Including Hadoop 2.7.7 onto the classpath brings in avro-1.7.4.jar, which conflicts at runtime

      $ find hadoop-2.7.7 -name *.jar | grep avro

      share/hadoop/mapreduce/lib/avro-1.7.4.jar

      share/hadoop/kms/tomcat/webapps/kms/WEB-INF/lib/avro-1.7.4.jar

      share/hadoop/tools/lib/avro-1.7.4.jar

      share/hadoop/common/lib/avro-1.7.4.jar

      hadoop/httpfs/tomcat/webapps/webhdfs/WEB-INF/lib/avro-1.7.4.jar

       

      Issue filed downstream in

      https://github.com/bigdatagenomics/adam/issues/2151

       

      Attached a smaller reproducing test case.

        Attachments

        1. reproduce.sh
          1 kB
          Michael Heuer

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              heuermh Michael Heuer
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: