[SPARK-28487] K8S pod allocator behaves poorly with dynamic allocation

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: Kubernetes, Spark Core
• Assignee: Marcelo Masiero Vanzin
• Reporter: Marcelo Masiero Vanzin
• Created: 23/Jul/19 21:27
• Updated: 17/May/20 18:25
• Resolved: 14/Aug/19 00:30

Description

The current pod allocator in Spark is tuned towards the behavior without dynamic allocation; it needs some enhancements so that dynamic allocation behaves better on K8S.

I'll be submitting an updated and enhanced version of a patch we've used internally.

Issue Links

• is related to: SPARK-27963 Allow dynamic allocation without an external shuffle service
• relates to: SPARK-24432 Add support for dynamic resource allocation
• links to: GitHub Pull Request #25236