[SPARK-24432] Add support for dynamic resource allocation

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.0
• Component/s: Kubernetes, Spark Core
• Assignee: Unassigned
• Reporter: Yinan Li
• Created: 30/May/18 19:02
• Updated: 02/Nov/20 19:17

Description

This is an umbrella ticket for work on adding support for dynamic resource allocation into the Kubernetes mode. This requires a Kubernetes-specific external shuffle service. The feature is available in our fork at github.com/apache-spark-on-k8s/spark.

Issue Links

• is blocked by: SPARK-25299 Use remote storage for persisting shuffle data
• is duplicated by: SPARK-25796 Enable external shuffle service for kubernetes mode.
• is related to: SPARK-27963 Allow dynamic allocation without an external shuffle service
• is related to: SPARK-28487 K8S pod allocator behaves poorly with dynamic allocation
• links to: GitHub Pull Request #22722
• links to: GitHub Pull Request #24083