[SPARK-26588] Idle executor should properly be killed when no job is submitted

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.4.0
• Component/s: Scheduler, Spark Core
• Assignee: Unassigned
• Reporter: Qingxin Wu
• Created: 10/Jan/19 14:09
• Updated: 12/Feb/19 22:57
• Resolved: 12/Feb/19 22:57

Description

I enable dynamic allocation feature with spark-shell and do not submit any task. After spark.dynamicAllocation.executorIdleTimeout seconds(default 60s), there is still one active executor, which is abnormal. All idle executors are timeout and should be removed.(default spark.dynamicAllocation.minExecutors=0). The spark-shell command show below:

spark-shell --master=yarn --conf spark.ui.port=8040 --conf spark.dynamicAllocation.enabled=true --conf spark.dynamicAllocation.maxExecutors=8 --conf spark.dynamicAllocation.initialExecutors=4 --conf spark.shuffle.service.enabled=true

Issue Links

• duplicates: SPARK-26758 Idle Executors are not getting killed after spark.dynamicAllocation.executorIdleTimeout value
• links to: GitHub Pull Request #23509