[SPARK-16106] TaskSchedulerImpl does not correctly handle new executors on existing hosts

• Type: Bug
• Status: Resolved
• Priority: Trivial
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.1.0
• Component/s: Scheduler, Spark Core
• Assignee: Imran Rashid
• Reporter: Imran Rashid
• Created: 21/Jun/16 15:35
• Updated: 17/May/20 17:48
• Resolved: 27/Jun/16 21:41

Description

The TaskSchedulerImpl updates the set of executors and hosts in each call to resourceOffers. During this call, it also tracks whether there are any new executors observed in newExecAvail:

      executorIdToHost(o.executorId) = o.host
      executorIdToTaskCount.getOrElseUpdate(o.executorId, 0)
      if (!executorsByHost.contains(o.host)) {
        executorsByHost(o.host) = new HashSet[String]()
        executorAdded(o.executorId, o.host)
        newExecAvail = true
      }

However, this only detects when a new host is added, not when an additional executor is added to an existing host (a relatively common event in dynamic allocation).

The end result is that task locality and failedEpochs is not updated correctly for new executors.

Issue Links

• links to: [Github] Pull Request #13826 (squito)