[SPARK-23972] Upgrade to Parquet 1.10

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 2.4.0
• Component/s: Spark Core
• Assignee: Ryan Blue
• Reporter: Henry Robinson
• Created: 12/Apr/18 23:14
• Updated: 20/Sep/18 17:51
• Resolved: 09/May/18 04:29

Description

Now that Parquet 1.10 is out, we should consider moving to it, if for no other reason than to address SPARK-23852. 

There are some non-trivial changes to address. In particular, the ValuesReader interface has changed from passing data as byte[] to a ByteBufferInputStream abstraction that can hide multiple {{ByteBuffer}}s under the hood.

Issue Links

• links to: [Github] Pull Request #21070 (rdblue)

Sub-Tasks

• TPC-DS Performance Tests for Parquet 1.10.0 Upgrade: Resolved
• Micro-benchmark of Parquet Filter Pushdown: Resolved