[SPARK-25419] Parquet predicate pushdown improvement

• Type: Umbrella
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 2.4.0
• Component/s: SQL
• Assignee: Yuming Wang
• Reporter: Yuming Wang
• Created: 13/Sep/18 07:40
• Updated: 18/Sep/18 22:09
• Resolved: 13/Sep/18 07:48

Description

Parquet predicate pushdown support: ByteType, ShortType, DecimalType, DateType, TimestampType. 

Sub-Tasks

• Support DATE predict push down in parquet: Resolved
• Refactor ParquetFilters: Resolved
• Support ByteType and ShortType pushdown to parquet: Resolved
• Support DecimalType push down to the parquet data sources: Resolved
• Timestamp support pushdown to parquet data source: Resolved
• StringStartsWith support push down: Resolved
• Convert IN predicate to equivalent Parquet filter: Resolved
• ByteArrayDecimalType support push down to parquet data sources: Resolved
• Case-insensitve field resolution for filter pushdown when reading Parquet: Resolved
• Improve in filter pushdown for ParquetFilters: Resolved