Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-9870

Mixed configurations for JVM -Xmx in hadoop command

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      When we use hadoop command to launch a class, there are two places setting the -Xmx configuration.

      1. The first place is located in file hadoop-common-project/hadoop-common/src/main/bin/hadoop.

      exec "$JAVA" $JAVA_HEAP_MAX $HADOOP_OPTS $CLASS "$@"
      

      Here $JAVA_HEAP_MAX is configured in hadoop-config.sh (hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh). The default value is "-Xmx1000m".

      2. The second place is set with $HADOOP_OPTS in file hadoop-common-project/hadoop-common/src/main/bin/hadoop.

      HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS"
      

      Here $HADOOP_CLIENT_OPTS is set in hadoop-env.sh (hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh)

      export HADOOP_CLIENT_OPTS="-Xmx512m $HADOOP_CLIENT_OPTS"
      

      Currently the final default java command looks like:

      java -Xmx1000m .... -Xmx512m CLASS_NAME ARGUMENTS"

      And if users also specify the -Xmx in the $HADOOP_CLIENT_OPTS, there will be three -Xmx configurations.

      The hadoop setup tutorial only discusses hadoop-env.sh, and it looks that users should not make any change in hadoop-config.sh.

      We should let hadoop smart to choose the right one before launching the java command, instead of leaving for jvm to make the decision.

        Attachments

        1. HADOOP-9870.patch
          4 kB
          Jay Sen
        2. HADOOP-9870.patch
          3 kB
          Jay Sen
        3. HADOOP-9870.patch
          1 kB
          Jay Sen

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                ywskycn Wei Yan
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: