Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-30186

support Dynamic Partition Pruning in Adaptive Execution

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.1.0
    • Fix Version/s: None
    • Component/s: SQL
    • Labels:
      None

      Description

      Currently Adaptive Execution cannot work if Dynamic Partition Pruning is applied.

      private def supportAdaptive(plan: SparkPlan): Boolean =

      { // TODO migrate dynamic-partition-pruning onto adaptive execution. sanityCheck(plan) && !plan.logicalLink.exists(_.isStreaming) && *!plan.expressions.exists(_.find(_.isInstanceOf[DynamicPruningSubquery]).isDefined)* && plan.children.forall(supportAdaptive) }

      It means we cannot benefit the performance from both AE and DPP.

      This ticket is target to make DPP + AE works together.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                xiaojuwu Xiaoju Wu
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated: