Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-8410

Hive VersionsSuite RuntimeException

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.3.1, 1.4.0
    • Fix Version/s: 1.4.1, 1.5.0
    • Component/s: SQL
    • Labels:
      None
    • Environment:

      IBM Power system - P7
      running Ubuntu 14.04LE

      Description

      While testing Spark Project Hive, there are RuntimeExceptions as follows,

      VersionsSuite:

      • success sanity check *** FAILED ***
        java.lang.RuntimeException: download failed: org.jboss.netty#netty;3.2.2.Final!netty.jar(bundle), download failed: org.codehaus.groovy#groovy-all;2.1.6!groovy-all.jar, download failed: asm#asm;3.2!asm.jar
        at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:978)
        at org.apache.spark.sql.hive.client.IsolatedClientLoader$$anonfun$3.apply(IsolatedClientLoader.scala:62)
        at org.apache.spark.sql.hive.client.IsolatedClientLoader$$anonfun$3.apply(IsolatedClientLoader.scala:62)
        at org.apache.spark.sql.catalyst.util.package$.quietly(package.scala:38)
        at org.apache.spark.sql.hive.client.IsolatedClientLoader$.org$apache$spark$sql$hive$client$IsolatedClientLoader$$downloadVersion(IsolatedClientLoader.scala:61)
        at org.apache.spark.sql.hive.client.IsolatedClientLoader$$anonfun$1.apply(IsolatedClientLoader.scala:44)
        at org.apache.spark.sql.hive.client.IsolatedClientLoader$$anonfun$1.apply(IsolatedClientLoader.scala:44)
        at scala.collection.mutable.MapLike$class.getOrElseUpdate(MapLike.scala:189)
        at scala.collection.mutable.AbstractMap.getOrElseUpdate(Map.scala:91)
        at org.apache.spark.sql.hive.client.IsolatedClientLoader$.forVersion(IsolatedClientLoader.scala:44)
        ...

      The tests are executed with the following set of options,

      build/mvn --pl sql/hive --fail-never -Pyarn -Phadoop-2.4 -Dhadoop.version=2.6.0 test

      Adding the following dependencies in the "spark/sql/hive/pom.xml" file solves this issue,

      < <dependency>
      < <groupId>org.jboss.netty</groupId>
      < <artifactId>netty</artifactId>
      < <version>3.2.2.Final</version>
      < <scope>test</scope>
      < </dependency>
      < <dependency>
      < <groupId>org.codehaus.groovy</groupId>
      < <artifactId>groovy-all</artifactId>
      < <version>2.1.6</version>
      < <scope>test</scope>
      < </dependency>
      <
      < <dependency>
      < <groupId>asm</groupId>
      < <artifactId>asm</artifactId>
      < <version>3.2</version>
      < <scope>test</scope>
      < </dependency>
      <

      The question is, Is this the correct way to fix this runtimeException ?
      If yes, Can a pull request fix this issue permanently ?
      If not, suggestions please.

      Updates:
      The above mentioned quick fix is not working with the latest 1.4 because of
      this pull commits :
      SPARK-8095 Resolve dependencies of --packages in local ivy cache #6788
      https://github.com/apache/spark/pull/6788

      Due to this above commit, now the lookup directories during testing phase
      has changed as follows,
      :: problems summary ::
      :::: WARNINGS
      [NOT FOUND ] org.jboss.netty#netty;3.2.2.Final!netty.jar(bundle) (2ms)

      ==== local-m2-cache: tried

      file:/home/joe/sparkibmsoe/spark/sql/hive/dummy/.m2/repository/org/jboss/netty/netty/3.2.2.Final/netty-3.2.2.Final.jar

      [NOT FOUND ] org.codehaus.groovy#groovy-all;2.1.6!groovy-all.jar (0ms)

      ==== local-m2-cache: tried

      file:/home/joe/sparkibmsoe/spark/sql/hive/dummy/.m2/repository/org/codehaus/groovy/groovy-all/2.1.6/groovy-all-2.1.6.jar

      [NOT FOUND ] asm#asm;3.2!asm.jar (0ms)

      ==== local-m2-cache: tried

      file:/home/joe/sparkibmsoe/spark/sql/hive/dummy/.m2/repository/asm/asm/3.2/asm-3.2.jar

      ::::::::::::::::::::::::::::::::::::::::::::::

      :: FAILED DOWNLOADS ::

      :: ^ see resolution messages for details ^ ::

      ::::::::::::::::::::::::::::::::::::::::::::::

      :: org.jboss.netty#netty;3.2.2.Final!netty.jar(bundle)

      :: org.codehaus.groovy#groovy-all;2.1.6!groovy-all.jar

      :: asm#asm;3.2!asm.jar

      ::::::::::::::::::::::::::::::::::::::::::::::

        Attachments

          Activity

            People

            • Assignee:
              brkyvz Burak Yavuz
              Reporter:
              josam Josiah Samuel Sathiadass
            • Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: