[SPARK-21422] Depend on Apache ORC 1.4.0

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.3.0
• Component/s: Build
• Assignee: Dongjoon Hyun
• Reporter: Dongjoon Hyun
• Created: 14/Jul/17 21:38
• Updated: 16/Aug/17 06:08
• Resolved: 16/Aug/17 06:08

Description

Like Parquet, this issue aims to depend on the latest Apache ORC 1.4 for Apache Spark 2.3. There are key benefits for now.

• Stability: Apache ORC 1.4.0 has many fixes and we can depend on ORC community more.
• Maintainability: Reduce the Hive dependency and can remove old legacy code later.

Later, we can get the following two key benefits by adding new ORCFileFormat in SPARK-20728, too.

• Usability: User can use ORC data sources without hive module, i.e, -Phive.
• Speed: Use both Spark ColumnarBatch and ORC RowBatch together. This is faster than the current implementation in Spark.

Issue Links

• blocks: SPARK-20901 Feature parity for ORC with Parquet
• blocks: SPARK-20682 Add new ORCFileFormat based on Apache ORC
• blocks: SPARK-20728 Make ORCFileFormat configurable between sql/hive and sql/core
• links to: [Github] Pull Request #18640 (dongjoon-hyun)