[SPARK-37049] executorIdleTimeout is not working for pending pods on K8s

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 3.1.3, 3.2.1, 3.3.0
• Component/s: Kubernetes, Spark Core
• Assignee: Weiwei Yang
• Reporter: Weiwei Yang
• Created: 18/Oct/21 23:08
• Updated: 26/Oct/21 08:06
• Resolved: 20/Oct/21 05:52

Description

SPARK-33099 added the support to respect "spark.dynamicAllocation.executorIdleTimeout" in ExecutorPodsAllocator. However, when it checks if a pending executor pod is timed out, it checks against the pod's "startTime". A pending pod "startTime" is empty, and this causes the function "isExecutorIdleTimedOut()" always return true for pending pods.

This caused the issue, pending pods are deleted immediately when a stage is finished and several new pods got recreated again in the next stage.

Issue Links

• links to: [Github] Pull Request #34319 (yangwwei)
• links to: [Github] Pull Request #34319 (yangwwei)