Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Incomplete
-
2.3.1
-
None
Description
Currently the Kubernetes Master internal URL is hardcoded in the Constants.scala file (https://github.com/apache/spark/blob/85fe1297e35bcff9cf86bd53fee615e140ee5bfb/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Constants.scala#L75)
In some cases these URL should be changed e.g. if the Certificate is valid for another Hostname.
Is it possible to make this URL a property like: spark.kubernetes.authenticate.driver.hostname?
Kubernetes The Hard Way maintained by Kelsey Hightower for example uses kubernetes.default as hostname, this will produce again a SSLPeerUnverifiedException.
Here is the use of the Hardcoded Host: https://github.com/apache/spark/blob/85fe1297e35bcff9cf86bd53fee615e140ee5bfb/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterManager.scala#L52 maybe this could be changed like the KUBERNETES_NAMESPACE property in Line 53.