Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-41469

Task rerun on decommissioned executor can be avoided if shuffle data has migrated

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.1.3, 3.2.2, 3.3.1
    • 3.4.0
    • Spark Core

    Description

      Currently, we will always rerun a finished shuffle map task if it once runs the lost executor. However, in the case of the executor loss is caused by decommission, the shuffle data might be migrated so that task doesn't need to rerun.

      Attachments

        Issue Links

          Activity

            People

              Ngone51 wuyi
              Ngone51 wuyi
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: