Details
Description
Spark Dataframes fail to run on Hadoop 3.0.x, because hive.jar's shimloader considers 3.x to be an unknown Hadoop version.
Hive itself will have to fix this; as Spark uses its own hive 1.2.x JAR, it will need to be updated to match.
Attachments
Issue Links
- blocks
-
SPARK-23534 Spark run on Hadoop 3.0.0
- Resolved
- depends upon
-
SPARK-23807 Add Hadoop 3 profile with relevant POM fix ups
- Resolved
-
HIVE-16081 allow "0.23" shim creation for Hadoop 3
- Resolved
- is broken by
-
HIVE-15326 Hive shims report Unrecognized Hadoop major version number: 3.0.0-alpha2-SNAPSHOT
- Closed
- relates to
-
HIVE-15016 Run tests with Hadoop 3.0.0-beta1
- Closed
-
HADOOP-13852 hadoop build to allow hadoop version property to be explicitly set
- Resolved