[HADOOP-9870] Mixed configurations for JVM -Xmx in hadoop command

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Assignee: Unassigned
• Reporter: Wei Yan
• Created: 13/Aug/13 20:04
• Updated: 18/Aug/14 23:56
• Resolved: 18/Aug/14 23:56

Description

When we use hadoop command to launch a class, there are two places setting the -Xmx configuration.

1. The first place is located in file hadoop-common-project/hadoop-common/src/main/bin/hadoop.

exec "$JAVA" $JAVA_HEAP_MAX $HADOOP_OPTS $CLASS "$@"

Here $JAVA_HEAP_MAX is configured in hadoop-config.sh (hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh). The default value is "-Xmx1000m".

2. The second place is set with $HADOOP_OPTS in file hadoop-common-project/hadoop-common/src/main/bin/hadoop.

HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS"

Here $HADOOP_CLIENT_OPTS is set in hadoop-env.sh (hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh)

export HADOOP_CLIENT_OPTS="-Xmx512m $HADOOP_CLIENT_OPTS"

Currently the final default java command looks like:

java -Xmx1000m .... -Xmx512m CLASS_NAME ARGUMENTS"

And if users also specify the -Xmx in the $HADOOP_CLIENT_OPTS, there will be three -Xmx configurations.

The hadoop setup tutorial only discusses hadoop-env.sh, and it looks that users should not make any change in hadoop-config.sh.

We should let hadoop smart to choose the right one before launching the java command, instead of leaving for jvm to make the decision.

Issue Links

• Is contained by: HADOOP-9902 Shell script rewrite
• is related to: HDFS-5087 Allowing specific JAVA heap max setting for HDFS related services