[SPARK-34334] ExecutorPodsAllocator fails to identify some excess requests during downscaling

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.0.0, 3.0.1, 3.1.0, 3.1.1, 3.1.2, 3.2.0
• Fix Version/s: 3.1.1, 3.2.0
• Component/s: Kubernetes
• Assignee: Attila Zsolt Piros
• Reporter: Attila Zsolt Piros
• Created: 02/Feb/21 20:18
• Updated: 22/Feb/21 06:58
• Resolved: 09/Feb/21 18:11

Description

During downscaling there are two kinds of POD requests which can lead to POD deletion as it is identified as excess requests by the dynamic allocation:

• timed out newly created POD requests (which even haven't reached the Pod Pending state, yet)
• timed out pending pod requests

The current implementation fails to delete a timed out pending pod requests when there are not enough timed out newly created POD requests to delete but there are some non-timed out ones.

Issue Links

• links to: [Github] Pull Request #31445 (attilapiros)