Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-15067

YARN executors are launched with fixed perm gen size

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.6.0, 1.6.1
    • Fix Version/s: 2.0.0
    • Component/s: YARN
    • Labels:
      None
    • Target Version/s:

      Description

      It is impossible to change the executors max perm gen size using the property "spark.executor.extraJavaOptions" when you are running on YARN.

      When the JVM option "-XX:MaxPermSize" is set through the property "spark.executor.extraJavaOptions", Spark put it properly in the shell command that will start the JVM container but, in the ending of command, it sets again this option using a fixed value of 256m, as you can see in the log I've extracted:

      2016-04-30 17:20:12 INFO ExecutorRunnable:58 -
      ===============================================================================
      YARN executor launch context:
      env:
      CLASSPATH -> PWD<CPS>PWD/_spark_.jar<CPS>$HADOOP_CONF_DIR<CPS>/usr/hdp/current/hadoop-client/<CPS>/usr/hdp/current/hadoop-client/lib/<CPS>/usr/hdp/current/hadoop-hdfs-client/<CPS>/usr/hdp/current/hadoop-hdfs-client/lib/<CPS>/usr/hdp/current/hadoop-yarn-client/<CPS>/usr/hdp/current/hadoop-yarn-client/lib/<CPS>/usr/hdp/mr-framework/hadoop/share/hadoop/mapreduce/:/usr/hdp/mr-framework/hadoop/share/hadoop/mapreduce/lib/:/usr/hdp/mr-framework/hadoop/share/hadoop/common/:/usr/hdp/mr-framework/hadoop/share/hadoop/common/lib/:/usr/hdp/mr-framework/hadoop/share/hadoop/yarn/:/usr/hdp/mr-framework/hadoop/share/hadoop/yarn/lib/:/usr/hdp/mr-framework/hadoop/share/hadoop/hdfs/:/usr/hdp/mr-framework/hadoop/share/hadoop/hdfs/lib/:/usr/hdp/current/hadoop/lib/hadoop-lzo-0.6.0.jar:/etc/hadoop/conf/secure
      SPARK_LOG_URL_STDERR -> http://xxxxx0668sl.xxxxx.br:8042/node/containerlogs/container_1456962126505_329993_01_000002/h_loadbd/stderr?start=-4096
      SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1456962126505_329993
      SPARK_YARN_CACHE_FILES_FILE_SIZES -> 191719054,166
      SPARK_USER -> h_loadbd
      SPARK_YARN_CACHE_FILES_VISIBILITIES -> PUBLIC,PUBLIC
      SPARK_YARN_MODE -> true
      SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1459806496093,1459808508343
      SPARK_LOG_URL_STDOUT -> http://xxxxx0668sl.xxxxx.br:8042/node/containerlogs/container_1456962126505_329993_01_000002/h_loadbd/stdout?start=-4096
      SPARK_YARN_CACHE_FILES -> hdfs://xxxxx/user/datalab/hdp/spark/lib/spark-assembly-1.6.0.2.3.4.1-10-hadoop2.7.1.2.3.4.1-10.jar#_spark_.jar,hdfs://tlvcluster/user/datalab/hdp/spark/conf/hive-site.xml#hive-site.xml

      command:
      JAVA_HOME/bin/java -server -XX:OnOutOfMemoryError='kill %p' -Xms6144m -Xmx6144m '-XX:+PrintGCDetails' '-XX:MaxPermSize=1024M' '-XX:+PrintGCTimeStamps' -Djava.io.tmpdir=PWD/tmp '-Dspark.akka.timeout=300000' '-Dspark.driver.port=62875' '-Dspark.rpc.askTimeout=300000' '-Dspark.rpc.lookupTimeout=300000' -Dspark.yarn.app.container.log.dir=<LOG_DIR> -XX:MaxPermSize=256m org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@10.125.81.42:62875 --executor-id 1 --hostname xxxxx0668sl.xxxxx.br --cores 1 --app-id application_1456962126505_329993 --user-class-path file:$PWD/__app__.jar 1> <LOG_DIR>/stdout 2> <LOG_DIR>/stderr

      Analyzing the code is possible to see that all the options set in the property "spark.executor.extraJavaOptions" are enclosed, one by one, in single quotes (ExecutorRunnable.scala:151) before the launcher take the decision if a default value has to be provided or not for the option "-XX:MaxPermSize" (ExecutorRunnable.scala:202).

      This decision is taken examining all the options set and looking for a string starting with the value "-XX:MaxPermSize" (CommandBuilderUtils.java:328). If that value is not found, the default value is set.

      A string option starting without single quote will never be found, then, a default value will always be provided.

      A possible solution is change the source code of CommandBuilderUtils.java in the line 328:
      From-> if (arg.startsWith("-XX:MaxPermSize="))
      To-> if (arg.indexOf("-XX:MaxPermSize=") > -1)

        Attachments

          Activity

            People

            • Assignee:
              srowen Sean Owen
              Reporter:
              renatojdk Renato Falchi Brandão
              Shepherd:
              Marcelo Vanzin
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: