[SPARK-33231] Make podCreationTimeout configurable

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0, 3.0.1, 3.1.0
• Fix Version/s: 3.1.0
• Component/s: Kubernetes
• Assignee: Holden Karau
• Reporter: Holden Karau
• Created: 23/Oct/20 18:36
• Updated: 27/Oct/20 18:54
• Resolved: 27/Oct/20 18:54

Description

Execution Monitor & Pod Allocator have differing views of the world which can lead to pod trashing.

The executor monitor can be notified of an executor coming up before a snapshot is delivered to the PodAllocator. This can cause the executor monitor to believe it needs to delete a pod, and the pod allocator to believe that it needs to create a new pod. This happens if the podCreationTimeout is too low for the cluster. Currently podCreationTimeout can only be configured by increasing the batch delay but that has additional consequences leading to slower spin up.

Issue Links

• links to: [Github] Pull Request #30155 (holdenk)
• links to: [Github] Pull Request #30155 (holdenk)