[SPARK-28925] Update Kubernetes-client to 4.4.2 to be compatible with Kubernetes 1.13 and 1.14

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: 2.3.3, 2.4.3
• Component/s: Kubernetes, Spark Core
• Assignee: Unassigned
• Reporter: Eric
• Created: 30/Aug/19 06:57
• Updated: 17/May/20 18:26
• Resolved: 01/Sep/19 15:02

Description

Hello,

If you use Spark with Kubernetes 1.13 or 1.14 you will see this error:

{"time": "2019-08-28T09:56:11.866Z", "lvl":"INFO", "logger": "org.apache.spark.internal.Logging", "thread":"kubernetes-executor-snapshots-subscribers-0","msg":"Going to request 1 executors from Kubernetes."}
{"time": "2019-08-28T09:56:12.028Z", "lvl":"WARN", "logger": "io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$2", "thread":"OkHttp https://kubernetes.default.svc/...","msg":"Exec Failure: HTTP 403, Status: 403 - "}
java.net.ProtocolException: Expected HTTP 101 response but was '403 Forbidden'

Apparently the bug is fixed here: https://github.com/fabric8io/kubernetes-client/pull/1669

We have currently compiled Spark source code with Kubernetes-client 4.4.2 and it's working great on our cluster. We are using Kubernetes 1.13.10.

 

Could it be possible to update that dependency version?

 

Thanks!

Issue Links

• duplicates: SPARK-28921 Spark jobs failing on latest versions of Kubernetes (1.15.3, 1.14.6, 1,13.10, 1.12.10, 1.11.10)