Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33828 SQL Adaptive Query Execution QA
  3. SPARK-36032

Use inputPlan instead of currentPhysicalPlan to initialize logical link

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.2.0, 3.3.0
    • 3.2.0
    • SQL
    • None

    Description

      At initialPlan we may remove some Spark Plan with queryStagePreparationRules, if removed Spark Plan is top level node, then we will lose the linked logical node.

      Since we support AQE side broadcast join config. It's more common that a join is SMJ at normal planner and changed to BHJ after AQE reOptimize. However, RemoveRedundantSorts is applied before reOptimize at initialPlan, then a local sort might be removed incorrectly if a join is SMJ at first but changed to BHJ during reOptimize.

      Attachments

        Activity

          People

            ulysses XiDuo You
            ulysses XiDuo You
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: