Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-28356

Do not reduce the number of partitions for repartition in adaptive execution

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 3.0.0
    • 3.0.0
    • SQL
    • None

    Description

      Adaptive execution reduces the number of post-shuffle partitions at runtime, even for shuffles caused by repartition. However, the user likely wants to get the desired number of partition when he calls repartition even in adaptive execution. It will be good to support not changing the number of post-shuffle partition for repartition in adaptive execution.

      Attachments

        Activity

          People

            carsonwang Carson Wang
            carsonwang Carson Wang
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: