[SPARK-30186] support Dynamic Partition Pruning in Adaptive Execution

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 3.2.0
• Component/s: SQL
• Assignee: Ke Jia
• Reporter: Xiaoju Wu
• Created: 09/Dec/19 11:52
• Updated: 23/Jul/21 13:53
• Resolved: 23/Jul/21 13:53

Description

Currently Adaptive Execution cannot work if Dynamic Partition Pruning is applied.

private def supportAdaptive(plan: SparkPlan): Boolean =

{ // TODO migrate dynamic-partition-pruning onto adaptive execution. sanityCheck(plan) && !plan.logicalLink.exists(_.isStreaming) && *!plan.expressions.exists(_.find(_.isInstanceOf[DynamicPruningSubquery]).isDefined)* && plan.children.forall(supportAdaptive) }

It means we cannot benefit the performance from both AE and DPP.

This ticket is target to make DPP + AE works together.

Issue Links

• relates to: SPARK-11150 Dynamic partition pruning
• links to: GitHub Pull Request #26814