[SPARK-33063] Improve error message for insufficient K8s volume confs

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 3.1.0
• Component/s: Kubernetes
• Assignee: German Schiavon Matteo
• Reporter: German Schiavon Matteo
• Created: 04/Oct/20 08:56
• Updated: 05/Oct/20 16:05
• Resolved: 05/Oct/20 16:05

Description

Providing error handling when creating k8s volumes and clearer error messages.

For example, when creating a hostPath volume, if you don't specify 

hostPath.volumeName.options.path

 it fails with a 

 key not found error

which is clear that you are missing a key, but I couldn't find anywhere in the docs that says you need to specify it.

To reproduce the issue, you have to do the spark-submit command like this for example:

./bin/spark-submit \
--master k8s://https://127.0.0.1:32768 \
--deploy-mode cluster \
--name spark-app\
--class class \
--conf spark.kubernetes.driver.volumes.hostPath.spark.mount.path=/tmp/jars/ \
--conf spark.kubernetes.executor.volumes.hostPath.spark.mount.path=/tmp/jars/ \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
--conf spark.kubernetes.container.image=spark:latest \
 local:///opt/spark/examples/jars/app.jar

 

Issue Links

• links to: [Github] Pull Request #29941 (Gschiavon)