Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Duplicate
-
None
-
None
-
None
-
None
Description
When we use hadoop command to launch a class, there are two places setting the -Xmx configuration.
1. The first place is located in file hadoop-common-project/hadoop-common/src/main/bin/hadoop.
exec "$JAVA" $JAVA_HEAP_MAX $HADOOP_OPTS $CLASS "$@"
Here $JAVA_HEAP_MAX is configured in hadoop-config.sh (hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh). The default value is "-Xmx1000m".
2. The second place is set with $HADOOP_OPTS in file hadoop-common-project/hadoop-common/src/main/bin/hadoop.
HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS"
Here $HADOOP_CLIENT_OPTS is set in hadoop-env.sh (hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh)
export HADOOP_CLIENT_OPTS="-Xmx512m $HADOOP_CLIENT_OPTS"
Currently the final default java command looks like:
java -Xmx1000m .... -Xmx512m CLASS_NAME ARGUMENTS"
And if users also specify the -Xmx in the $HADOOP_CLIENT_OPTS, there will be three -Xmx configurations.
The hadoop setup tutorial only discusses hadoop-env.sh, and it looks that users should not make any change in hadoop-config.sh.
We should let hadoop smart to choose the right one before launching the java command, instead of leaving for jvm to make the decision.
Attachments
Attachments
Issue Links
- Is contained by
-
HADOOP-9902 Shell script rewrite
- Resolved
- is related to
-
HDFS-5087 Allowing specific JAVA heap max setting for HDFS related services
- Resolved