Implements dynamic partition pruning by adding a dynamic-partition-pruning filter if there is a partitioned table and a filter on the dimension table. The filter is then planned using a heuristic approach:
- As a broadcast relation if it is a broadcast hash join. The broadcast relation will then be transformed into a reused broadcast exchange by the ReuseExchange rule; or
- As a subquery duplicate if the estimated benefit of partition table scan being saved is greater than the estimated cost of the extra scan of the duplicated subquery; otherwise
- As a bypassed condition (true).
Below shows a basic example of DPP.
- duplicates
-
SPARK-28888 Implement Dynamic Partition Pruning
-
- Resolved
-
- is related to
-
SPARK-30186 support Dynamic Partition Pruning in Adaptive Execution
-
- Open
-
- is required by
-
SPARK-28959 Support `EqualNullSafe` as join condition in Dynamic Partition Pruning
-
- Open
-
- links to
(3 links to)