[SPARK-38019] ExecutorMonitor.timedOutExecutors should be deterministic

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0, 3.2.1, 3.3.0
• Fix Version/s: 3.3.0, 3.2.2
• Component/s: Spark Core
• Assignee: Dongjoon Hyun
• Reporter: Dongjoon Hyun
• Created: 25/Jan/22 08:13
• Updated: 16/Dec/22 17:30
• Resolved: 25/Jan/22 11:01

Description

Since the AS-IS timedOutExecutors returns the result indeterministic, it kills the executors in a random order at Dynamic Allocation setting.

spark/core/src/main/scala/org/apache/spark/scheduler/dynalloc/ExecutorMonitor.scala

 private val executors = new ConcurrentHashMap[String, Tracker]() 
...
 timedOutExecs = executors.asScala 

This random behavior not only makes the users confusing but also causes a K8s decommission tests flaky like the following case in Java 17 on Apple Silicon environment. The K8s test expects the decommission of executor 1 while the executor 2 is chosen at this time.

22/01/25 06:11:16 DEBUG ExecutorMonitor: Executors 1,2 do not have active shuffle data after job 0 finished.
22/01/25 06:11:16 DEBUG ExecutorAllocationManager: max needed for rpId: 0 numpending: 0, tasksperexecutor: 1
22/01/25 06:11:16 DEBUG ExecutorAllocationManager: No change in number of executors
22/01/25 06:11:16 DEBUG ExecutorAllocationManager: Request to remove executorIds: (2,0), (1,0)
22/01/25 06:11:16 DEBUG ExecutorAllocationManager: Not removing idle executor 1 because there are only 1 executor(s) left (minimum number of executor limit 1)
22/01/25 06:11:16 INFO KubernetesClusterSchedulerBackend: Decommission executors: 2

Issue Links

• links to: [Github] Pull Request #35315 (dongjoon-hyun)
• links to: [Github] Pull Request #35315 (dongjoon-hyun)