[SPARK-34098] HadoopVersionInfoSuite faild when maven test in Scala 2.13

• Type: Sub-task
• Status: Closed
• Priority: Minor
• Resolution: Not A Problem
• Affects Version/s: 3.2.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Yang Jie
• Created: 13/Jan/21 12:08
• Updated: 25/Feb/21 23:46
• Resolved: 25/Feb/21 23:45

Description

 

mvn clean test -Phadoop-3.2 -Phive-2.3 -Phadoop-cloud -Pmesos -Pyarn -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive -pl sql/hive -Pscala-2.13  -DwildcardSuites=org.apache.spark.sql.hive.client.HadoopVersionInfoSuite -Dtest=none 

Independent test HadoopVersionInfoSuite all case passed, but execute

 

mvn clean test -Phadoop-3.2 -Phive-2.3 -Phadoop-cloud -Pmesos -Pyarn -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive -pl sql/hive -Pscala-2.13 

 

HadoopVersionInfoSuite failed as follow:

HadoopVersionInfoSuite: 22:32:30.310 WARN org.apache.spark.sql.hive.client.IsolatedClientLoader: Failed to resolve Hadoop artifacts for the version 2.7.4. We will change the hadoop version from 2.7.4 to 2.7.4 and try again. It is recommended to set jars used by Hive metastore client through spark.sql.hive.metastore.jars in the production environment. - SPARK-32256: Hadoop VersionInfo should be preloaded *** FAILED *** java.lang.RuntimeException: [unresolved dependency: org.apache.hive#hive-metastore;2.0.1: not found, unresolved dependency: org.apache.hive#hive-exec;2.0.1: not found, unresolved dependency: org.apache.hive#hive-common;2.0.1: not found, unresolved dependency: org.apache.hive#hive-serde;2.0.1: not found, unresolved dependency: com.google.guava#guava;14.0.1: not found, unresolved dependency: org.apache.hadoop#hadoop-client;2.7.4: not found] at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1423) at org.apache.spark.sql.hive.client.IsolatedClientLoader$.$anonfun$downloadVersion$2(IsolatedClientLoader.scala:122) at org.apache.spark.sql.catalyst.util.package$.quietly(package.scala:42) at org.apache.spark.sql.hive.client.IsolatedClientLoader$.downloadVersion(IsolatedClientLoader.scala:122) at org.apache.spark.sql.hive.client.IsolatedClientLoader$.liftedTree1$1(IsolatedClientLoader.scala:75) at org.apache.spark.sql.hive.client.IsolatedClientLoader$.forVersion(IsolatedClientLoader.scala:63) at org.apache.spark.sql.hive.client.HadoopVersionInfoSuite.$anonfun$new$1(HadoopVersionInfoSuite.scala:46) at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85) at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83) at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)

need some investigate.