[SPARK-20728] Make ORCFileFormat configurable between sql/hive and sql/core

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.3.0
• Component/s: SQL
• Assignee: Dongjoon Hyun
• Reporter: Dongjoon Hyun
• Created: 12/May/17 22:39
• Updated: 17/Jan/18 06:28
• Resolved: 05/Dec/17 12:48

Description

SPARK-20682 is trying to improve Apache Spark to have a new ORCFileFormat based on Apache ORC for many reasons.

This issue depends on SPARK-20682 and aims to provide a configuration to choose the default ORCFileFormat from legacy `sql/hive` module or new `sql/core` module.

For example, this configuration will affects the following operations.

spark.read.orc(...)

CREATE TABLE t
USING ORC
...

Issue Links

• blocks: SPARK-20901 Feature parity for ORC with Parquet
• is blocked by: SPARK-21422 Depend on Apache ORC 1.4.0
• is blocked by: SPARK-20682 Add new ORCFileFormat based on Apache ORC
• supercedes: SPARK-19109 ORC metadata section can sometimes exceed protobuf message size limit
• links to: [Github] Pull Request #17980 (dongjoon-hyun)
• links to: [Github] Pull Request #19871 (dongjoon-hyun)
• links to: [Github] Pull Request #19903 (dongjoon-hyun)