[SPARK-11150] Dynamic partition pruning

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.5.1, 1.6.0, 2.0.0, 2.1.2, 2.2.1, 2.3.0
• Fix Version/s: 3.0.0
• Component/s: SQL
• Labels: pull-request-available release-notes
• Assignee: Wei Xue
• Reporter: Younes
• Created: 16/Oct/15 17:15
• Updated: 16/Jun/24 02:51
• Resolved: 04/Sep/19 20:14

Description

Implements dynamic partition pruning by adding a dynamic-partition-pruning filter if there is a partitioned table and a filter on the dimension table. The filter is then planned using a heuristic approach:

1. As a broadcast relation if it is a broadcast hash join. The broadcast relation will then be transformed into a reused broadcast exchange by the ReuseExchange rule; or
2. As a subquery duplicate if the estimated benefit of partition table scan being saved is greater than the estimated cost of the extra scan of the duplicated subquery; otherwise
3. As a bypassed condition (true).

 Below shows a basic example of DPP.

Issue Links

• duplicates: SPARK-28888 Implement Dynamic Partition Pruning
• is related to: SPARK-30186 support Dynamic Partition Pruning in Adaptive Execution
• is required by: SPARK-28959 Support `EqualNullSafe` as join condition in Dynamic Partition Pruning
• links to: [Github] Pull Request #28872 (AngersZhuuuu)
• links to: [Github] Pull Request #28872 (AngersZhuuuu)
• links to: [Github] Pull Request #34768 (weixiuli)
• links to: [Github] Pull Request #34768 (weixiuli)
• links to: GitHub Pull Request #25600
• links to: GitHub Pull Request #26328
• links to: GitHub Pull Request #27581