[SPARK-28376] Support to write sorted parquet files in each row group

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.4.3
• Component/s: Input/Output, Spark Core
• Assignee: Unassigned
• Reporter: t oo
• Created: 12/Jul/19 23:09
• Updated: 12/Dec/22 18:10
• Resolved: 15/Jul/19 05:11

Description

this is for the ability to writeee parquet with sorteed values in each rowgroup

 

see https://stackoverflow.com/questions/52159938/cant-write-ordered-data-to-parquet-in-spark

https://www.slideshare.net/RyanBlue3/parquet-performance-tuning-the-missing-guide (slidee 26-27)

 

Issue Links

• is related to: SPARK-15420 Repartition and sort before Parquet writes