[SPARK-43329] driver and executors shared same Kubernetes PVC in Spark 3.4+

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.1
• Component/s: Kubernetes
• Assignee: Qian Sun
• Reporter: comet
• Created: 01/May/23 03:25
• Updated: 07/May/23 01:48
• Resolved: 07/May/23 01:48

Description

I able to shared same PVC for spark 3.3. but on Spark 3.4 onward. i get below error.  I would like all the executors and driver to mount the same PVC. Is this a bug ? I don't want to use SPARK_EXECUTOR_ID or OnDemand because otherwise each of the executors will use an unique and separate PVC.
 
Error message is "should contain OnDemand or SPARK_EXECUTOR_ID when requiring multiple executors"
 
below is how I enabled it pvc in spark 3.3 and it works, but does not work in Spark 3.4

spark.kubernetes.driver.volumes.persistentVolumeClaim.rwxpvc.options.claimName=rwxpvc 
--conf spark.kubernetes.driver.volumes.persistentVolumeClaim.rwxpvc.mount.path=/opt/spark/work-dir 
--conf spark.kubernetes.executor.volumes.persistentVolumeClaim.rwxpvc.options.claimName=rwxpvc 
--conf spark.kubernetes.executor.volumes.persistentVolumeClaim.rwxpvc.mount.path=/opt/spark/work-dir 
 

 
 
 

Issue Links

• is caused by: SPARK-39006 Show a directional error message for PVC Dynamic Allocation Failure