[SPARK-33537] Hive Metastore filter pushdown improvement

• Type: Umbrella
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.0
• Component/s: SQL
• Assignee: Yuming Wang
• Reporter: Yuming Wang
• Created: 24/Nov/20 16:11
• Updated: 25/Feb/21 12:49

Description

This umbrella ticket to track Hive Metastore filter pushdown improvement. It includes:
1. Date type push down
2. Like push down
3. InSet pushdown improvement
and other fixes.

Sub-Tasks

• Hive partition pruning support Contains, StartsWith and EndsWith predicate: Resolved
• Avoid Hive metastore stack overflow when InSet predicate have many values: Resolved
• RuntimeException when querying a view on a partitioned parquet table: Resolved
• Hive partition pruning support date type: Resolved
• Refactor HivePartitionFilteringSuite: Resolved
• Partition predicate pushdown into Hive metastore support not-equals: Resolved
• Partition predicate pushdown into Hive metastore support cast string type to date type: Resolved
• Check isNull and isNotNull in tests: Resolved
• Directly push IN predicates to the Hive Metastore: Open
• Support multiple types of function partition pruning on hive metastore: In Progress
• Hive Metastore support filter by not-in: Resolved
• Use expressions to filter Hive partitions at client side: Resolved