[SPARK-21834] Incorrect executor request in case of dynamic allocation

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.1.2, 2.2.1, 2.3.0
• Component/s: Scheduler, Spark Core
• Assignee: Sital Kedia
• Reporter: Sital Kedia
• Created: 25/Aug/17 01:01
• Updated: 17/May/20 17:48
• Resolved: 30/Aug/17 21:20

Description

killExecutor api currently does not allow killing an executor without updating the total number of executors needed. In case of dynamic allocation is turned on and the allocator tries to kill an executor, the scheduler reduces the total number of executors needed ( see https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala#L635) which is incorrect because the allocator already takes care of setting the required number of executors itself.

Issue Links

• is duplicated by: SPARK-22598 ExecutorAllocationManager does not requests new executors when executor fail and target has not change
• relates to: SPARK-23365 DynamicAllocation with failure in straggler task can lead to a hung spark job
• links to: [Github] Pull Request #19048 (sitalkedia)
• links to: [Github] Pull Request #19081 (sitalkedia)