[SPARK-13127] Upgrade Parquet to 1.9 (Fixes parquet sorting)

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.0.0, 2.0.1
• Component/s: SQL
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Justin Pihony
• Created: 02/Feb/16 03:57
• Updated: 08/Oct/19 05:42
• Resolved: 08/Oct/19 05:42

Description

Currently, when you write a sorted DataFrame to Parquet, then reading the data back out is not sorted by default. This is due to a bug in Parquet that was fixed in 1.9.

There is a workaround to read the file back in using a file glob (filepath/*).

Issue Links

• is blocked by: PARQUET-1027 release Parquet-mr 1.9.1
• is duplicated by: SPARK-18860 Update Parquet to 1.9.0
• is duplicated by: SPARK-18140 Parquet NPE / Update to 1.9
• is duplicated by: SPARK-20406 Upgrade parquet to 1.9
• is related to: SPARK-10109 NPE when saving Parquet To HDFS
• is related to: SPARK-10294 When Parquet writer's close method throws an exception, we will call close again and trigger a NPE
• links to: [Github] Pull Request #16281 (dongjoon-hyun)