[SPARK-33332] Errors in running spark job on K8 with RPC Authentication Secret File properties

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.0
• Component/s: Kubernetes
• Assignee: Unassigned
• Reporter: Surbhi Aggarwal
• Created: 03/Nov/20 23:20
• Updated: 17/Jun/22 14:08

Description

I am running a spark job on Kubernetes with RPC authentication feature. First i am mounting a K8 secret to both driver and executor to a path and providing the same path for Auth configuration. Below is the spark-conf i am passing related to RPC Auth.

--conf spark.authenticate=true
--conf spark.kubernetes.driver.secrets.spark-secret-sa=/tmp/secrets
--conf spark.kubernetes.executor.secrets.spark-secret-sa=/tmp/secrets
--conf spark.authenticate.secret.driver.file=/tmp/secrets/token
--conf spark.authenticate.secret.executor.file=/tmp/secrets/token

 

Mounting of secret is successfully done on both driver and executor pods and i can see the token file written to the containers. Driver is able to load the secret as well from the file but executor pods are failing to load the secret from file, and fails with the below exception:

java.lang.IllegalArgumentException: A secret key must be specified via the spark.authenticate.secret config
 at org.apache.spark.SecurityManager.$anonfun$getSecretKey$6(SecurityManager.scala:298)
 at scala.Option.getOrElse(Option.scala:189)
 at org.apache.spark.SecurityManager.getSecretKey(SecurityManager.scala:297)

 

The value of the spark.authenticate.secret.executor.file property is being passed in SPARK_JAVA_OPS

SPARK_JAVA_OPT_8: -Dspark.authenticate.secret.driver.file=/tmp/secrets/token
SPARK_JAVA_OPT_9: -Dspark.authenticate.secret.executor.file=/tmp/secrets/token

I am suspecting that values from java options are not being read into the spark conf and ultimately leading to failure.

 

I am not sure if its a bug or a mis-configuration on my end. Amy help is greatly appreciated.