I able to shared same PVC for spark 3.3. but on Spark 3.4 onward. i get below error. I would like all the executors and driver to mount the same PVC. Is this a bug ? I don't want to use SPARK_EXECUTOR_ID or OnDemand because otherwise each of the executors will use an unique and separate PVC.
Error message is "should contain OnDemand or SPARK_EXECUTOR_ID when requiring multiple executors"
below is how I enabled it pvc in spark 3.3 and it works, but does not work in Spark 3.4