Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-26606

parameters passed in extraJavaOptions are not being picked up

    XMLWordPrintableJSON

    Details

      Description

      driver.extraJavaOptions and executor.extraJavaOptions are not being picked up . Even though I see the parameters are being passed fine in the spark launch command I do not see these parameters are being picked up for some unknown reason. My source code throws an error stating the java params are empty

       

      This is my spark submit command: 

          output=`spark-submit \
      --class com.demo.myApp.App \
      --conf 'spark.executor.extraJavaOptions=-Dapp.env=dev -Dapp.country=US -Dapp.banner=ABC -Doracle.net.tns_admin=/work/artifacts/oracle/current -Djava.security.egd=file:/dev/./urandom' \
      --conf 'spark.driver.extraJavaOptions=-Dapp.env=dev -Dapp.country=US -Dapp.banner=ABC -Doracle.net.tns_admin=/work/artifacts/oracle/current -Djava.security.egd=file:/dev/./urandom' \
      --executor-memory "$EXECUTOR_MEMORY" \
      --executor-cores "$EXECUTOR_CORES" \
      --total-executor-cores "$TOTAL_CORES" \
      --driver-memory "$DRIVER_MEMORY" \
      --deploy-mode cluster \
      /home/spark/asm//current/myapp-*.jar 2>&1 &`

       

       

      Is there any other way I can access the java params with out using extraJavaOptions. 

        Attachments

          Activity

            People

            • Assignee:
              kabhwan Jungtaek Lim
              Reporter:
              rrb441 Ravindra
            • Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: