[SPARK-6795] Avoid reading Parquet footers on driver side when an global arbitrative schema is available

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.0.2, 1.1.1, 1.2.1, 1.3.1
• Fix Version/s: 1.5.0
• Component/s: SQL
• Target Version/s: 1.5.0
• Assignee: Cheng Lian
• Reporter: Cheng Lian
• Created: 09/Apr/15 03:13
• Updated: 13/Aug/15 08:40
• Resolved: 06/Aug/15 14:47

Description

With the help of Parquet MR PR #91 which will be included in the official release of Parquet MR 1.6.0, now it's possible to avoid reading footers on the driver side completely when an global arbitrative schema is available.

Currently, the global schema can be either Hive metastore schema or specified via data sources DDL. All tasks should verify Parquet data files and reconcile possible schema conflicts locally against this global schema.

However, when no global schema is available and schema merging is enabled, we still need to read schemas from all data files to infer a valid global schema.

Issue Links

• incorporates: SPARK-7340 Use latest parquet release 1.6.0 in spark
• relates to: SPARK-8125 Accelerate ParquetRelation2 metadata discovery
• links to: [WIP][SQL][SPARK-6632]: Read schema from each input split in the ReadSupport hook, reconciling with the metastore schema at that time #5298