[SPARK-36052] Introduce pending pod limit for Spark on K8s

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.3, 3.1.2, 3.2.0, 3.3.0
• Fix Version/s: 3.2.0, 3.3.0
• Component/s: Kubernetes
• Labels: releasenotes
• Assignee: Attila Zsolt Piros
• Reporter: Attila Zsolt Piros
• Created: 08/Jul/21 11:40
• Updated: 28/Jan/22 01:07
• Resolved: 11/Aug/21 03:16

Description

Introduce a new configuration to limit the number of pending PODs for Spark on K8S as the K8S scheduler could be overloaded with requests which slows down the resource allocations (especially in case of dynamic allocation).

Issue Links

• is related to: SPARK-36060 Support backing off dynamic allocation increases if resources are "stuck"
• links to: [Github] Pull Request #33492 (attilapiros)
• links to: [Github] Pull Request #33755 (dongjoon-hyun)
• links to: [Github] Pull Request #33755 (dongjoon-hyun)