[SPARK-28983] Can not pruning hive partitions by Or predicate

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: 2.4.4
• Component/s: SQL
• Assignee: Unassigned
• Reporter: huajian lan
• Created: 05/Sep/19 04:25
• Updated: 05/Sep/19 12:43
• Resolved: 05/Sep/19 09:44

Description

Assume the hive partition table has p1/p2 partition column and c1 non-partition column,

the below where clause can not prunning partitions in sparksql, but we can prunning by "p1 = 'a' or p2 = 'b'" in principal.

where
(p1 = 'a' and c1 = 1)
or
(p2 = 'b' and c1 = 2)

 

The problem is in HiveTableScans#apply(plan: LogicalPlan),  when the predicate has any child that contains non-partition column, it will make a outer FilterExec and empty predicate in HiveTableScanExec.

 

Issue Links

• duplicates: SPARK-28169 Spark can’t push down partition predicate for OR expression
• links to: GitHub Pull Request #25688