[SPARK-26758] Idle Executors are not getting killed after spark.dynamicAllocation.executorIdleTimeout value

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 2.3.4, 2.4.1, 3.0.0
• Component/s: Spark Core, YARN

Environment

Spark Version:2.4

• Assignee: Sandeep Katta
• Reporter: ABHISHEK KUMAR GUPTA
• Created: 29/Jan/19 05:51
• Updated: 12/Dec/22 18:10
• Resolved: 05/Feb/19 04:18

Description

Steps:

1. Submit Spark shell with below initial Executor 3, minimum Executor=0 and executorIdleTimeout=60s

bin/spark-shell --master yarn --conf spark.dynamicAllocation.enabled=true \
  --conf spark.dynamicAllocation.initialExecutors=3 \
  --conf spark.dynamicAllocation.minExecutors=0 \
  --conf spark.dynamicAllocation.executorIdleTimeout=60s

2. Launch Spark UI and check under Executor Tab

Observation:

Initial 3 Executors assigned. After 60s( executorIdleTimeout) , number of active executor remains same.

Expected:

Apart from AM container, all other executors should be dead.

 

 

Issue Links

• is duplicated by: SPARK-26588 Idle executor should properly be killed when no job is submitted
• links to: GitHub Pull Request #23697