Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-3217

Shaded Guava jar doesn't play well with Maven build when SPARK_PREPEND_CLASSES is set

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.2.0
    • Fix Version/s: 1.2.0
    • Component/s: Build
    • Labels:
      None
    • Target Version/s:

      Description

      PR #1813 shaded Guava jar file and moved Guava classes to package org.spark-project.guava when Spark is built by Maven. But if developers set the environment variable SPARK_PREPEND_CLASSES to true, commands like bin/spark-shell throws ClassNotFoundException:

      # Set the env var
      $ export SPARK_PREPEND_CLASSES=true
      
      # Build Spark with Maven
      $ mvn clean package -Phive,hadoop-2.3 -Dhadoop.version=2.3.0 -DskipTests
      ...
      
      # Then spark-shell complains
      $ ./bin/spark-shell
      Spark assembly has been built with Hive, including Datanucleus jars on classpath
      Exception in thread "main" java.lang.NoClassDefFoundError: com/google/common/util/concurrent/ThreadFactoryBuilder
              at org.apache.spark.util.Utils$.<init>(Utils.scala:636)
              at org.apache.spark.util.Utils$.<clinit>(Utils.scala)
              at org.apache.spark.repl.SparkILoop.<init>(SparkILoop.scala:134)
              at org.apache.spark.repl.SparkILoop.<init>(SparkILoop.scala:65)
              at org.apache.spark.repl.Main$.main(Main.scala:30)
              at org.apache.spark.repl.Main.main(Main.scala)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:606)
              at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:317)
              at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:73)
              at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
      Caused by: java.lang.ClassNotFoundException: com.google.common.util.concurrent.ThreadFactoryBuilder
              at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
              at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
              at java.security.AccessController.doPrivileged(Native Method)
              at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
              at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
              ... 13 more
      
      # Check the assembly jar file
      $ jar tf assembly/target/scala-2.10/spark-assembly-1.1.0-SNAPSHOT-hadoop2.3.0.jar | grep -i ThreadFactoryBuilder
      org/spark-project/guava/common/util/concurrent/ThreadFactoryBuilder$1.class
      org/spark-project/guava/common/util/concurrent/ThreadFactoryBuilder.class
      

      SBT build is fine since we don't shade Guava with SBT right now.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                vanzin Marcelo Masiero Vanzin
                Reporter:
                lian cheng Cheng Lian
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: