[HIVE-23014] ORC reading performance

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.3.6
• Component/s: ORC
• Assignee: Unassigned
• Reporter: Peter Toth
• Created: 12/Mar/20 06:51
• Updated: 12/Mar/20 15:27

Description

Spark 3 adds support for using Hive 2.3.6 besides the old Hive 1.2.1 version. Some of the ORC reading benchmark shows that there is a huge performance difference in ORC reading between the 2 versions. I measured that org.apache.hadoop.hive.ql.io.orc.ReaderImpl in hive-exec-2.3.6-core.jar is ~3-5 times slower than in hive-exec-1.2.1.spark2.jar.

I'm not sure if more recent Hive versions still suffer from this performance regression.

Please see some details here: SPARK-30565