[SPARK-18673] Dataframes doesn't work on Hadoop 3.x; Hive rejects Hadoop version

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1.0
• Fix Version/s: 3.0.0
• Component/s: SQL

Environment

Spark built with -Dhadoop.version=3.0.0-alpha2-SNAPSHOT

• Assignee: Yuming Wang
• Reporter: Steve Loughran
• Created: 01/Dec/16 11:08
• Updated: 12/Dec/22 18:11
• Resolved: 23/May/19 23:03

Description

Spark Dataframes fail to run on Hadoop 3.0.x, because hive.jar's shimloader considers 3.x to be an unknown Hadoop version.

Hive itself will have to fix this; as Spark uses its own hive 1.2.x JAR, it will need to be updated to match.

Issue Links

• blocks: SPARK-23534 Spark run on Hadoop 3.0.0
• depends upon: SPARK-23807 Add Hadoop 3 profile with relevant POM fix ups
• depends upon: HIVE-16081 allow "0.23" shim creation for Hadoop 3
• is broken by: HIVE-15326 Hive shims report Unrecognized Hadoop major version number: 3.0.0-alpha2-SNAPSHOT
• relates to: HIVE-15016 Run tests with Hadoop 3.0.0-beta1
• relates to: HADOOP-13852 hadoop build to allow hadoop version property to be explicitly set