Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-16106

TaskSchedulerImpl does not correctly handle new executors on existing hosts

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Trivial
    • Resolution: Fixed
    • Affects Version/s: 2.0.0
    • Fix Version/s: 2.1.0
    • Component/s: Scheduler
    • Labels:
      None

      Description

      The TaskSchedulerImpl updates the set of executors and hosts in each call to resourceOffers. During this call, it also tracks whether there are any new executors observed in newExecAvail:

            executorIdToHost(o.executorId) = o.host
            executorIdToTaskCount.getOrElseUpdate(o.executorId, 0)
            if (!executorsByHost.contains(o.host)) {
              executorsByHost(o.host) = new HashSet[String]()
              executorAdded(o.executorId, o.host)
              newExecAvail = true
            }
      

      However, this only detects when a new host is added, not when an additional executor is added to an existing host (a relatively common event in dynamic allocation).

      The end result is that task locality and failedEpochs is not updated correctly for new executors.

        Attachments

          Activity

            People

            • Assignee:
              irashid Imran Rashid
              Reporter:
              irashid Imran Rashid
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: