Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-31412 New Adaptive Query Execution in Spark SQL
  3. SPARK-30524

Disable OptimizeSkewJoin rule if introducing additional shuffle.

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 3.0.0
    • Component/s: SQL
    • Labels:
      None

      Description

      The OptimizeSkewedJoin will break the outputPartitioning of origin SMJ. And it may introduce additional shuffle after apply the OptimizeSkewedJoin. This PR will disable "OptimizeSkewedJoin" rule if introducing additional shuffle.

        Attachments

          Activity

            People

            • Assignee:
              Jk_Self Ke Jia
              Reporter:
              Jk_Self Ke Jia
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: