-
Type:
Improvement
-
Status: In Progress
-
Priority:
Major
-
Resolution: Unresolved
-
Affects Version/s: 3.1.0
-
Fix Version/s: None
-
Component/s: Kubernetes
-
Labels:None
For Spark on Kubernetes there are multiple ways to achieve dynamic scaling. Unlike in YARN mode ESS on Spark on K8s as a sidecar would not help with dynamic scaling. Instead this helps with the situations where the executor JVM may be overloaded or crash. If we do this as a sidecar we can share the same mounts between the executor and the external shuffle storage.