Details
-
Bug
-
Status: Resolved
-
Minor
-
Resolution: Fixed
-
1.6.0, 1.6.1
-
None
Description
It is impossible to change the executors max perm gen size using the property "spark.executor.extraJavaOptions" when you are running on YARN.
When the JVM option "-XX:MaxPermSize" is set through the property "spark.executor.extraJavaOptions", Spark put it properly in the shell command that will start the JVM container but, in the ending of command, it sets again this option using a fixed value of 256m, as you can see in the log I've extracted:
2016-04-30 17:20:12 INFO ExecutorRunnable:58 -
===============================================================================
YARN executor launch context:
env:
CLASSPATH -> PWD<CPS>PWD/_spark_.jar<CPS>$HADOOP_CONF_DIR<CPS>/usr/hdp/current/hadoop-client/<CPS>/usr/hdp/current/hadoop-client/lib/<CPS>/usr/hdp/current/hadoop-hdfs-client/<CPS>/usr/hdp/current/hadoop-hdfs-client/lib/<CPS>/usr/hdp/current/hadoop-yarn-client/<CPS>/usr/hdp/current/hadoop-yarn-client/lib/<CPS>/usr/hdp/mr-framework/hadoop/share/hadoop/mapreduce/:/usr/hdp/mr-framework/hadoop/share/hadoop/mapreduce/lib/:/usr/hdp/mr-framework/hadoop/share/hadoop/common/:/usr/hdp/mr-framework/hadoop/share/hadoop/common/lib/:/usr/hdp/mr-framework/hadoop/share/hadoop/yarn/:/usr/hdp/mr-framework/hadoop/share/hadoop/yarn/lib/:/usr/hdp/mr-framework/hadoop/share/hadoop/hdfs/:/usr/hdp/mr-framework/hadoop/share/hadoop/hdfs/lib/:/usr/hdp/current/hadoop/lib/hadoop-lzo-0.6.0.jar:/etc/hadoop/conf/secure
SPARK_LOG_URL_STDERR -> http://xxxxx0668sl.xxxxx.br:8042/node/containerlogs/container_1456962126505_329993_01_000002/h_loadbd/stderr?start=-4096
SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1456962126505_329993
SPARK_YARN_CACHE_FILES_FILE_SIZES -> 191719054,166
SPARK_USER -> h_loadbd
SPARK_YARN_CACHE_FILES_VISIBILITIES -> PUBLIC,PUBLIC
SPARK_YARN_MODE -> true
SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1459806496093,1459808508343
SPARK_LOG_URL_STDOUT -> http://xxxxx0668sl.xxxxx.br:8042/node/containerlogs/container_1456962126505_329993_01_000002/h_loadbd/stdout?start=-4096
SPARK_YARN_CACHE_FILES -> hdfs://xxxxx/user/datalab/hdp/spark/lib/spark-assembly-1.6.0.2.3.4.1-10-hadoop2.7.1.2.3.4.1-10.jar#_spark_.jar,hdfs://tlvcluster/user/datalab/hdp/spark/conf/hive-site.xml#hive-site.xml
command:
JAVA_HOME/bin/java -server -XX:OnOutOfMemoryError='kill %p' -Xms6144m -Xmx6144m '-XX:+PrintGCDetails' '-XX:MaxPermSize=1024M' '-XX:+PrintGCTimeStamps' -Djava.io.tmpdir=PWD/tmp '-Dspark.akka.timeout=300000' '-Dspark.driver.port=62875' '-Dspark.rpc.askTimeout=300000' '-Dspark.rpc.lookupTimeout=300000' -Dspark.yarn.app.container.log.dir=<LOG_DIR> -XX:MaxPermSize=256m org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@10.125.81.42:62875 --executor-id 1 --hostname xxxxx0668sl.xxxxx.br --cores 1 --app-id application_1456962126505_329993 --user-class-path file:$PWD/__app__.jar 1> <LOG_DIR>/stdout 2> <LOG_DIR>/stderr
Analyzing the code is possible to see that all the options set in the property "spark.executor.extraJavaOptions" are enclosed, one by one, in single quotes (ExecutorRunnable.scala:151) before the launcher take the decision if a default value has to be provided or not for the option "-XX:MaxPermSize" (ExecutorRunnable.scala:202).
This decision is taken examining all the options set and looking for a string starting with the value "-XX:MaxPermSize" (CommandBuilderUtils.java:328). If that value is not found, the default value is set.
A string option starting without single quote will never be found, then, a default value will always be provided.
A possible solution is change the source code of CommandBuilderUtils.java in the line 328:
From-> if (arg.startsWith("-XX:MaxPermSize="))
To-> if (arg.indexOf("-XX:MaxPermSize=") > -1)
