[SPARK-24248] [K8S] Use the Kubernetes cluster as the backing store for the state of pods

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.4.0
• Component/s: Kubernetes, Spark Core
• Assignee: Unassigned
• Reporter: Matt Cheah
• Created: 10/May/18 22:10
• Updated: 23/Jun/23 18:08
• Resolved: 14/Jun/18 23:16

Description

We have a number of places in KubernetesClusterSchedulerBackend right now that maintains the state of pods in memory. However, the Kubernetes API can always give us the most up to date and correct view of what our executors are doing. We should consider moving away from in-memory state as much as can in favor of using the Kubernetes cluster as the source of truth for pod status. Maintaining less state in memory makes it so that there's a lower chance that we accidentally miss updating one of these data structures and breaking the lifecycle of executors.

Issue Links

• causes: SPARK-44158 Remove unused `spark.kubernetes.executor.lostCheck.maxAttempts`
• links to: [Github] Pull Request #21366 (mccheah)