Description
Right now, we hardcoded hadoop 2.4 to download when we need to download hive and hadoop jars for metadata hive. However, SPARK-9596 makes isoalted client loader share hadoop classes. So, if there is any compatibility issues between the shared hadoop and hadoop 2.4, VersionsSuite will failed. Actually, in spark 1.6's hadoop 2.2 tests, VersionsSuite always fail (https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Dashboard%20(Patrick)/job/Spark-1.6-SBT/AMPLAB_JENKINS_BUILD_PROFILE=hadoop2.2,label=spark-test/40/testReport/junit/org.apache.spark.sql.hive.client/). Because HftpFileSystem has been moved to a new location in Hadoop 2.3+. From Hadoop 2.4, we get org.apache.hadoop.hdfs.web.HftpFileSystem. But, when we look up the class (by looking at hadoop 2.2's jars), it is not in this location.
Attachments
Issue Links
- relates to
-
SPARK-9596 Avoid reloading Hadoop classes like UserGroupInformation
-
- Resolved
-
- links to
