Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25075 Build and test Spark against Scala 2.13
  3. SPARK-34098

HadoopVersionInfoSuite faild when maven test in Scala 2.13

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Minor
    • Resolution: Not A Problem
    • 3.2.0
    • None
    • SQL
    • None

    Description

       

      mvn clean test -Phadoop-3.2 -Phive-2.3 -Phadoop-cloud -Pmesos -Pyarn -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive -pl sql/hive -Pscala-2.13  -DwildcardSuites=org.apache.spark.sql.hive.client.HadoopVersionInfoSuite -Dtest=none 
      

      Independent test HadoopVersionInfoSuite all case passed, but execute

       

      mvn clean test -Phadoop-3.2 -Phive-2.3 -Phadoop-cloud -Pmesos -Pyarn -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive -pl sql/hive -Pscala-2.13 
      

       

      HadoopVersionInfoSuite failed as follow:

      HadoopVersionInfoSuite: 22:32:30.310 WARN org.apache.spark.sql.hive.client.IsolatedClientLoader: Failed to resolve Hadoop artifacts for the version 2.7.4. We will change the hadoop version from 2.7.4 to 2.7.4 and try again. It is recommended to set jars used by Hive metastore client through spark.sql.hive.metastore.jars in the production environment. - SPARK-32256: Hadoop VersionInfo should be preloaded *** FAILED *** java.lang.RuntimeException: [unresolved dependency: org.apache.hive#hive-metastore;2.0.1: not found, unresolved dependency: org.apache.hive#hive-exec;2.0.1: not found, unresolved dependency: org.apache.hive#hive-common;2.0.1: not found, unresolved dependency: org.apache.hive#hive-serde;2.0.1: not found, unresolved dependency: com.google.guava#guava;14.0.1: not found, unresolved dependency: org.apache.hadoop#hadoop-client;2.7.4: not found] at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1423) at org.apache.spark.sql.hive.client.IsolatedClientLoader$.$anonfun$downloadVersion$2(IsolatedClientLoader.scala:122) at org.apache.spark.sql.catalyst.util.package$.quietly(package.scala:42) at org.apache.spark.sql.hive.client.IsolatedClientLoader$.downloadVersion(IsolatedClientLoader.scala:122) at org.apache.spark.sql.hive.client.IsolatedClientLoader$.liftedTree1$1(IsolatedClientLoader.scala:75) at org.apache.spark.sql.hive.client.IsolatedClientLoader$.forVersion(IsolatedClientLoader.scala:63) at org.apache.spark.sql.hive.client.HadoopVersionInfoSuite.$anonfun$new$1(HadoopVersionInfoSuite.scala:46) at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85) at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83) at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
      

      need some investigate.

       

       

       

      Attachments

        Activity

          People

            Unassigned Unassigned
            LuciferYang Yang Jie
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: