Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-41469

Task rerun on decommissioned executor can be avoided if shuffle data has migrated

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.1.3, 3.2.2, 3.3.1
    • 3.4.0
    • Spark Core

    Description

      Currently, we will always rerun a finished shuffle map task if it once runs the lost executor. However, in the case of the executor loss is caused by decommission, the shuffle data might be migrated so that task doesn't need to rerun.

      Attachments

        Activity

          People

            Ngone51 wuyi
            Ngone51 wuyi
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: