Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33235 Push-based Shuffle Improvement Tasks
  3. SPARK-33665

Enable ShuffleBlockPusher to stop pushing blocks for a particular shuffle partition

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.1.0
    • None
    • Shuffle
    • None

    Description

      ShuffleBlockPusher, which was introduced in SPARK-32917 stops pushing shuffle blocks for the entire shuffle when it receives "TOO Late" exception. However, with the change https://github.com/apache/spark/pull/30433, there is also a need to stop pushing shuffle blocks for a particular reduce partition. Refer https://github.com/apache/spark/pull/30433#discussion_r533694433

       

      Attachments

        Activity

          People

            Unassigned Unassigned
            csingh Chandni Singh
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated: