[SPARK-20406] Upgrade parquet to 1.9

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.1.0
• Component/s: Spark Core
• Labels: maven
• Assignee: Unassigned
• Reporter: Kaspar Sakmann
• Created: 20/Apr/17 09:22
• Updated: 20/Apr/17 09:58
• Resolved: 20/Apr/17 09:58

Description

Please upgrade parquet to 1.9.
with 500G memory 48 CPUs and a dataset about 1G in size
using Parquet 1.8 causes warnings:

"Total allocation exceeds 95.00% (2,147,483,647 bytes) of heap memory
Scaling row group sizes to 34.04% for 47 writers"

The issue appears to have been fixed with parquet 1.9, see
https://issues.apache.org/jira/browse/PARQUET-340

Issue Links

• duplicates: SPARK-13127 Upgrade Parquet to 1.9 (Fixes parquet sorting)