[SPARK-36833] Can't use SSL with spark on kubernetes on service level

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 3.0.0
• Component/s: Kubernetes, Security
• Assignee: Unassigned
• Reporter: zoli
• Created: 23/Sep/21 12:17
• Updated: 08/Dec/22 12:43

Description

Currently it seems impossible to create the correct cert for driver's pod because of the random naming of the service.

I would like to use ssl on spark Ui which will be accessed by other pods using the driver's service.

"spark.ssl.enabled"=true
"spark.ssl.keyStore"=my-spark.jks
"spark.ssl.keyStorePassword"=mypassword
..etc..

At this point we already have to know the domain for the cert.

Which we don't because it will be generated at time when the driver pod generated.

my-application-75f3654hj76gb67n-driver
my-application-75f3654hj76gb67n-driver-svc

So SSL handshake will fail with :

" SSL: no alternative certificate subject name matches target host name my-application-75f3654hj76gb67n-driver-svc

I tried to mod the pod name with:

 spark.kubernetes.driver.pod.name

but it only affects the pod name and not the service name

If it is neither a bug nor a missed feature then please guide me how to use SSL when hitting the driver's service (or how to define fixed name service like for pods).

----------------------------------------------------------------------------------------------------------------------------------------------------------------

I found a partial solution using wildcards for domain inside the cert, but because it only works on subdomain level I have to refer the service with :
<POD_NAME>-*-driver-svc.<NS>.svc as alternatedomain inside the cert
and using it with the namespace , svc added just to conform the wildcard's rule subdomain restriction