Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-40105

Improve repartition in ReplaceCTERefWithRepartition

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 3.4.0
    • 3.4.0
    • SQL
    • None

    Description

      If cte can not inlined, the ReplaceCTERefWithRepartition will add repartition to force a shuffle so that the reference can reuse shuffle exchange.
      The added repartition should be optimized by AQE for better performance.

      If the user has specified a rebalance, the ReplaceCTERefWithRepartition should skip add repartition.

      Attachments

        Activity

          People

            ulysses XiDuo You
            ulysses XiDuo You
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: