[PARQUET-1685] Truncate the stored min and max for String statistics to reduce the footer size

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.10.1
• Fix Version/s: 1.11.0
• Component/s: parquet-mr
• Labels: pull-request-available
• Assignee: Xinli Shang
• Reporter: Xinli Shang
• Created: 26/Oct/19 01:53
• Updated: 23/Jun/24 03:31
• Resolved: 13/Nov/19 09:15

Description

Iceberg has a cool feature that truncates the stored min, max statistics to minimize the metadata size. We can borrow to truncate them in Parquet also to reduce the size of the footer, or even the page header. Here is the code in IceBerg https://github.com/apache/incubator-iceberg/blob/master/api/src/main/java/org/apache/iceberg/util/UnicodeUtil.java. 

 

 

 

 

 

 

Issue Links

• links to: GitHub Pull Request #696
• links to: PARQUET-1685:Truncate Min/Max for Statistics