[SPARK-23980] Resilient Spark driver on Kubernetes

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.3.0
• Component/s: Spark Core
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Sebastian Toader
• Created: 13/Apr/18 13:51
• Updated: 25/May/21 01:51
• Resolved: 25/May/21 01:43

Description

The current implementation of `Spark driver` on Kubernetes is not resilient to node failures as it’s implemented as a `Pod`. In case of a node failure Kubernetes terminates the pods that were running on that node. Kubernetes doesn't reschedule these pods to any of the other nodes of the cluster.

If the `driver` is implemented as Kubernetes Job than it will be rescheduled to other node.

When the driver is terminated its executors (that may run on other nodes) are terminated by Kubernetes with some delay by Kubernetes Garbage collection.

This can lead to concurrency issues where the re-spawned `driver` was trying to create new executors with same name as the executors being in the middle of being cleaned up by Kubernetes garbage collection.

To solve this issue the executor name must be made unique for each `driver` instance.

The PR linked to this lira is an implementation of the above that creates spark driver as a Job and ensures that executor pod names are unique per driver instance.

Issue Links

• is duplicated by: SPARK-26441 Add kind configuration of driver pod
• links to: [Github] Pull Request #21067 (baluchicken)