[SPARK-32642] Add support for ESS in Spark sidecar

• Type: Improvement
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.0
• Component/s: Kubernetes
• Assignee: Holden Karau
• Reporter: Holden Karau
• Created: 17/Aug/20 22:21
• Updated: 21/Aug/20 21:30

Description

For Spark on Kubernetes there are multiple ways to achieve dynamic scaling. Unlike in YARN mode ESS on Spark on K8s as a sidecar would not help with dynamic scaling. Instead this helps with the situations where the executor JVM may be overloaded or crash. If we do this as a sidecar we can share the same mounts between the executor and the external shuffle storage.

Issue Links

• links to: [Github] Pull Request #29511 (holdenk)
• links to: [Github] Pull Request #29511 (holdenk)