If you use Spark with Kubernetes 1.13 or 1.14 you will see this error:
Apparently the bug is fixed here: https://github.com/fabric8io/kubernetes-client/pull/1669
We have currently compiled Spark source code with Kubernetes-client 4.4.2 and it's working great on our cluster. We are using Kubernetes 1.13.10.
Could it be possible to update that dependency version?