[SPARK-30784] Hive 2.3 profile should still use orc-nohive

• Type: Sub-task
• Status: Closed
• Priority: Critical
• Resolution: Not A Bug
• Affects Version/s: 3.0.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Yin Huai
• Created: 10/Feb/20 22:25
• Updated: 10/Mar/20 08:50
• Resolved: 04/Mar/20 19:26

Description

Originally reported at https://github.com/apache/spark/pull/26619#issuecomment-583802901

 

Right now, Hive 2.3 profile pulls in regular orc, which depends on hive-storage-api. However, hive-storage-api and hive-common have the following common class files

 

org/apache/hadoop/hive/common/ValidReadTxnList.class
org/apache/hadoop/hive/common/ValidTxnList.class
org/apache/hadoop/hive/common/ValidTxnList$RangeResponse.class

For example, https://github.com/apache/hive/blob/rel/storage-release-2.6.0/storage-api/src/java/org/apache/hadoop/hive/common/ValidReadTxnList.java (pulled in by orc 1.5.8) and https://github.com/apache/hive/blob/rel/release-2.3.6/common/src/java/org/apache/hadoop/hive/common/ValidReadTxnList.java (from hive-common 2.3.6) both are in the classpath and they are different. Having both versions in the classpath can cause unexpected behavior due to classloading order. We should still use orc-nohive, which has hive-storage-api shaded.

Issue Links

• is caused by: SPARK-27176 Upgrade hadoop-3's built-in Hive maven dependencies to 2.3.4
• links to: GitHub Pull Request #27536