[SPARK-23710] Upgrade the built-in Hive to 2.3.5 for hadoop-3.2

• Type: Umbrella
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 3.0.0
• Component/s: SQL
• Target Version/s: 3.0.0
• Assignee: Yuming Wang
• Reporter: Yuming Wang
• Created: 16/Mar/18 16:07
• Updated: 12/Dec/22 18:10
• Resolved: 08/Feb/20 05:24

Description

Spark fail to run on Hadoop 3.x, because Hive's ShimLoader considers Hadoop 3.x to be an unknown Hadoop version. see SPARK-18673 and HIVE-16081 for more details. So we need to upgrade the built-in Hive for Hadoop-3.x. This is an umbrella JIRA to track this upgrade.

 

Upgrade Plan:

1. SPARK-27054 Remove the Calcite dependency. This can avoid some jar conflicts.
2. SPARK-23749 Replace built-in Hive API (isSub/toKryo) and remove OrcProto.Type usage
3. SPARK-27158, SPARK-27130 Update dev/* to support dynamic change profiles when testing
4. Fix ORC dependency conflict to makes it test passed on Hive 1.2.1 and compile passed on Hive 2.3.4
5. Add an empty hive-thriftserverV2 module. then we could test all test cases in next step
6. Make Hadoop-3.1 with Hive 2.3.4 test passed
7. Adapted hive-thriftserverV2 from hive-thriftserver with Hive 2.3.4's TCLIService.thrift

 

I have completed the initial work and plan to finish this upgrade step by step.
 

 

Issue Links

• blocks: SPARK-27361 YARN support for GPU-aware scheduling
• is related to: SPARK-31113 Support DDL "SHOW VIEWS"
• is related to: SPARK-12014 Spark SQL query containing semicolon is broken in Beeline (related to HIVE-11100)
• is related to: SPARK-24766 CreateHiveTableAsSelect and InsertIntoHiveDir won't generate decimal column stats in parquet
• relates to: SPARK-27500 Add tests for built-in Hive 2.3
• supercedes: SPARK-27377 Upgrade YARN to 3.1.2+ to support GPU
• supercedes: SPARK-24472 Orc RecordReaderFactory throws IndexOutOfBoundsException
• supercedes: SPARK-28748 0 as decimal (n , n) in Hive tables shows as NULL in Spark
• links to: GitHub Pull Request #23552
• links to: GitHub Pull Request #23553
• links to: GitHub Pull Request #23788

Sub-Tasks

• Remove Calcite dependency: Resolved
• Replace built-in Hive API (isSub/toKryo) and remove OrcProto.Type usage: Resolved
• Automatically select profile when executing sbt-checkstyle: Resolved
• dev/mima and dev/scalastyle support dynamic profiles: Resolved
• Reduce the code duplicate when upgrading built-in Hive: Resolved
• Move the conflict source code of the sql/core module to sql/core/v1.2.1: Resolved
• Dealing with TimeVars removed in Hive 2.x: Resolved
• Upgrade hadoop-3's built-in Hive maven dependencies to 2.3.4: Resolved
• Upgrade hadoop-3 to 3.2.0: Resolved
• Exclude javax.ws.rs:jsr311-api from hadoop-client: Resolved
• Fix testing issues with yarn module in Hadoop-3: Resolved
• Avoid using hard-coded jar names in Hive tests: Resolved
• Exclude commons-httpclient when interacting with different versions of the HiveMetastoreClient: Resolved
• Fix hadoop-3.2 test issue(except the hive-thriftserver module): Resolved
• Move incompatible code from the hive-thriftserver module to sql/hive-thriftserver/v1.2.1: Resolved
• Upgrade commons-logging to 1.1.3: Resolved
• parquet-hadoop-bundle is incorrect in dev/deps/spark-deps-hadoop-3.2: Closed
• Exclude com.zaxxer:HikariCP-java7 from hadoop-yarn-server-web-proxy: Resolved
• Beeline should show database in the prompt: Resolved
• Upgrade to Hive 2.3.5 for Hive Metastore Client and Hadoop-3.2 profile: Resolved
• Upgrade the built-in Hive to 2.3.5 for hadoop-3.2: Resolved
• hadoop-3.2 support hive-thriftserver: Resolved
• Dataframes doesn't work on Hadoop 3.x; Hive rejects Hadoop version: Resolved
• Update building-spark.md: Resolved
• Thriftserver throws java.math.BigDecimal incompatible with org.apache.hadoop.hive.common.type.HiveDecimal: Resolved
• Update LICENSE and NOTICE for Hive 2.3: Resolved
• --jar argument with spark-sql failed to load the jars to driver classpath: Resolved
• drop database throws Exception: Resolved
• orc library is incorrect in dev/deps/spark-deps-hadoop-3.2: Closed
• dev/deps/spark-deps-hadoop-3.2 orc jar is incorrect: Resolved
• Hive 2.3 profile should still use orc-nohive: Closed
• Migration guide for Hive 2.3: Resolved