[PARQUET-221] For array type, inconsistent names are used as the array element name.

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.6.0
• Component/s: parquet-mr
• Assignee: Unassigned
• Reporter: Yin Huai
• Created: 19/Mar/15 04:57
• Updated: 19/Mar/15 18:22

Description

When creating a convert for an array, Parquet Avro uses "array" as the field name name (see here) , but Parquet Hive SerDe uses "array_element" as the field name see here. In Spark SQL, our native Parquet support is following Parquet Avro's convention, for data generated by Parquet Hive SerDe, the array value cannot be correctly read and null will be returned.

Issue Links

• is related to: SPARK-5508 Arrays and Maps stored with Hive Parquet Serde may not be able to read by the Parquet support in the Data Souce API