[IMPALA-3909] Parquet file writer should populate the min/max statistics per block per column to be used by the reader

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 2.8.0
• Fix Version/s: Impala 2.9.0
• Component/s: Backend
• Labels: performance
• Epic Link: Improve the reliability and effectiveness of ETL
• Target Version: Impala 2.9.0
• Assignee: Lars Volker
• Reporter: Mostafa Mokhtar
• Created: 25/Jul/16 23:14
• Updated: 02/Jun/17 22:53
• Resolved: 03/Feb/17 01:43

Description

The Parquet file writer should populate the min/max indexes while writing data such that it can be used by the reader.
Today data written by Hive populates those statistics but not Impala.

Issue Links

• is blocked by: IMPALA-4163 Introduce SORTBY plan hint for insert statements
• is related to: IMPALA-2328 Parquet scan should use min/max statistics to skip blocks based on predicate
• relates to: IMPALA-4887 Broken local filesystem TestHdfsParquetTableStatsWriter