[SPARK-26742] Bump Kubernetes Client Version to 4.1.2

• Type: Dependency upgrade
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0, 3.0.0
• Fix Version/s: 2.4.1, 3.0.0
• Component/s: Kubernetes, Spark Core
• Labels: easyfix
• Assignee: Jiaxin Shan
• Reporter: Steve Davids
• Created: 26/Jan/19 21:00
• Updated: 17/May/20 18:24
• Resolved: 13/Mar/19 22:05

Description

Spark 2.x is using Kubernetes Client 3.x which is pretty old, the master branch has 4.0, the client should be upgraded to 4.1.1 to have the broadest Kubernetes compatibility support for newer clusters: https://github.com/fabric8io/kubernetes-client#compatibility-matrix

Issue Links

• is duplicated by: SPARK-25590 kubernetes-model-2.0.0.jar masks default Spark logging config
• is related to: SPARK-27812 kubernetes client import non-daemon thread which block jvm exit.
• relates to: SPARK-26603 Update minikube backend in K8s integration tests
• links to: GitHub Pull Request #23814
• links to: GitHub Pull Request #23993
• links to: GitHub Pull Request #24002