[SPARK-6774] Implement Parquet complex types backwards-compatiblity rules

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.0.0, 1.1.1, 1.2.1, 1.3.0
• Component/s: SQL
• Target Version/s: 1.6.0
• Assignee: Unassigned
• Reporter: Cheng Lian
• Shepherd: Cheng Lian
• Created: 08/Apr/15 12:32
• Updated: 08/Oct/15 23:20
• Resolved: 08/Oct/15 23:20

Description

Parquet format PR #17 standardized representation of Parquet complex types and listed backwards-compatibility rules. Spark SQL should implement these compatibility rules to improve interoperatability.

Before, Spark SQL is only compatible with parquet-avro, parquet-hive, and Impala. And it's done in an error prone ad-hoc way, because Parquet format spec didn't explicitly specify complex type structures at the time Spark SQL Parquet support was firstly authored. After fixing this issue, we are expected to be compatible with most (if not all) systems that generated Parquet data in a systematic way by conforming to Parquet format spec and implementing all backwards-compatibility rules.

Issue Links

• is duplicated by: SPARK-5508 Arrays and Maps stored with Hive Parquet Serde may not be able to read by the Parquet support in the Data Souce API
• is duplicated by: SPARK-8811 Read array struct data from parquet error

Sub-Tasks

• Simplify CatalystConverter class hierarchy and pass in Parquet schema: Resolved
• Implement backwards-compatibility rules in Catalyst converters (which convert Parquet record to rows): Resolved
• Implement backwards-compatibility rules in Parquet schema converters: Resolved
• Write Parquet LISTs and MAPs conforming to Parquet format spec: Resolved
• A direct write API for testing Parquet compatibility: Resolved