Details
Major Description
driver.extraJavaOptions and executor.extraJavaOptions are not being picked up . Even though I see the parameters are being passed fine in the spark launch command I do not see these parameters are being picked up for some unknown reason. My source code throws an error stating the java params are empty
This is my spark submit command:
output=`spark-submit \
--class com.demo.myApp.App \
--conf 'spark.executor.extraJavaOptions=-Dapp.env=dev -Dapp.country=US -Dapp.banner=ABC -Doracle.net.tns_admin=/work/artifacts/oracle/current -Djava.security.egd=file:/dev/./urandom' \
--conf 'spark.driver.extraJavaOptions=-Dapp.env=dev -Dapp.country=US -Dapp.banner=ABC -Doracle.net.tns_admin=/work/artifacts/oracle/current -Djava.security.egd=file:/dev/./urandom' \
--executor-memory "$EXECUTOR_MEMORY" \
--executor-cores "$EXECUTOR_CORES" \
--total-executor-cores "$TOTAL_CORES" \
--driver-memory "$DRIVER_MEMORY" \
--deploy-mode cluster \
/home/spark/asm//current/myapp-*.jar 2>&1 &`
Is there any other way I can access the java params with out using extraJavaOptions.