Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-3217

Shaded Guava jar doesn't play well with Maven build when SPARK_PREPEND_CLASSES is set

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotVotersStop watchingWatchersCreate sub-taskConvert to sub-taskLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.2.0
    • 1.2.0
    • Build
    • None

    Description

      PR #1813 shaded Guava jar file and moved Guava classes to package org.spark-project.guava when Spark is built by Maven. But if developers set the environment variable SPARK_PREPEND_CLASSES to true, commands like bin/spark-shell throws ClassNotFoundException:

      # Set the env var
      $ export SPARK_PREPEND_CLASSES=true
      
      # Build Spark with Maven
      $ mvn clean package -Phive,hadoop-2.3 -Dhadoop.version=2.3.0 -DskipTests
      ...
      
      # Then spark-shell complains
      $ ./bin/spark-shell
      Spark assembly has been built with Hive, including Datanucleus jars on classpath
      Exception in thread "main" java.lang.NoClassDefFoundError: com/google/common/util/concurrent/ThreadFactoryBuilder
              at org.apache.spark.util.Utils$.<init>(Utils.scala:636)
              at org.apache.spark.util.Utils$.<clinit>(Utils.scala)
              at org.apache.spark.repl.SparkILoop.<init>(SparkILoop.scala:134)
              at org.apache.spark.repl.SparkILoop.<init>(SparkILoop.scala:65)
              at org.apache.spark.repl.Main$.main(Main.scala:30)
              at org.apache.spark.repl.Main.main(Main.scala)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:606)
              at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:317)
              at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:73)
              at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
      Caused by: java.lang.ClassNotFoundException: com.google.common.util.concurrent.ThreadFactoryBuilder
              at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
              at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
              at java.security.AccessController.doPrivileged(Native Method)
              at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
              at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
              ... 13 more
      
      # Check the assembly jar file
      $ jar tf assembly/target/scala-2.10/spark-assembly-1.1.0-SNAPSHOT-hadoop2.3.0.jar | grep -i ThreadFactoryBuilder
      org/spark-project/guava/common/util/concurrent/ThreadFactoryBuilder$1.class
      org/spark-project/guava/common/util/concurrent/ThreadFactoryBuilder.class
      

      SBT build is fine since we don't shade Guava with SBT right now.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            vanzin Marcelo Masiero Vanzin Assign to me
            lian cheng Cheng Lian
            Votes:
            0 Vote for this issue
            Watchers:
            7 Stop watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Issue deployment