Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-11163

Remove unnecessary addPendingTask calls in TaskSetManager.executorLost

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.6.0
    • Component/s: Scheduler, Spark Core
    • Labels:
      None

      Description

      The proposed commit removes unnecessary calls to addPendingTask in
      TaskSetManager.executorLost. These calls are unnecessary: for
      tasks that are still pending and haven't been launched, they're
      still in all of the correct pending lists, so calling addPendingTask
      has no effect. For tasks that are currently running (which may still be
      in the pending lists, depending on how they were scheduled), we call
      addPendingTask in handleFailedTask, so the calls at the beginning
      of executorLost are redundant.

      I think these calls are left over from when we re-computed the locality
      levels in addPendingTask; now that we call recomputeLocality separately,
      I don't think these are necessary.

        Attachments

          Activity

            People

            • Assignee:
              kayousterhout Kay Ousterhout
              Reporter:
              kayousterhout Kay Ousterhout
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: