Hadoop Common
  1. Hadoop Common
  2. HADOOP-6167

bin/hadoop script doesn't allow for different memory settings for each daemon type

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.20.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      bin/hadoop assumes that all daemon types ( namenode, datanode, jobtracker, tasktracker ), all use the same memory settings.. (HADOOP_HEAPSIZE).

      I propose changes to that script to allow overriding the default memory ( HADOOP_HEAPSIZE ), with daemon specific OPTS (HADOOP_NAMENODE_OPTS, etc ).

      Basically at the bottom of the bin/hadoop script, it will check to see if the user has already set "-Xmx" in the HADOOP_OPTS variable.. if so, then it will ignore the JAVA_HEAP_SIZE variable..

      as such:

      1. run it
        if [[ $HADOOP_OPTS == -Xmx ]]; then
        exec "$JAVA" $HADOOP_OPTS -classpath "$CLASSPATH" $CLASS "$@"
        else
        exec "$JAVA" $JAVA_HEAP_MAX $HADOOP_OPTS -classpath "$CLASSPATH" $CLASS "$@"
        fi

      I will attach the file as I have modified it..

      1. hadoop
        10 kB
        Fernando Padilla
      2. hadoop-script.diff
        0.8 kB
        Fernando Padilla

        Issue Links

          Activity

          Hide
          Fernando Padilla added a comment -

          proposed new bin/hadoop script file.

          The only change is in the last few lines.

          Show
          Fernando Padilla added a comment - proposed new bin/hadoop script file. The only change is in the last few lines.
          Hide
          Fernando Padilla added a comment -

          HBase scripts have the same issue since they were just copying things from Hadoop.. so they have the same bug.

          Here is the related bug in HBase: https://issues.apache.org/jira/browse/HBASE-1687

          Show
          Fernando Padilla added a comment - HBase scripts have the same issue since they were just copying things from Hadoop.. so they have the same bug. Here is the related bug in HBase: https://issues.apache.org/jira/browse/HBASE-1687
          Hide
          ryan rawson added a comment -

          I'm not sure this is true, my datanode processes are running like so:

          /usr/java/jdk1.6.0_14/bin/java -Xmx1000m -Dcom.sun.management.jmxremote -Xmx2000m -XX:+DoEscapeAnalysis (all the other arguments)

          in hadoop-env.sh i have:
          export HADOOP_DATANODE_OPTS="-Xmx2000m -XX:+DoEscapeAnalysis ...."

          how does this not work?

          Show
          ryan rawson added a comment - I'm not sure this is true, my datanode processes are running like so: /usr/java/jdk1.6.0_14/bin/java -Xmx1000m -Dcom.sun.management.jmxremote -Xmx2000m -XX:+DoEscapeAnalysis (all the other arguments) in hadoop-env.sh i have: export HADOOP_DATANODE_OPTS="-Xmx2000m -XX:+DoEscapeAnalysis ...." how does this not work?
          Hide
          Doug Cutting added a comment -

          This might be better done by:

          • adding JAVA_HEAP_MAX to HADOOP_OPTS if it does not already contain -Xmx
          • removing JAVA_HEAP_MAX from the java command

          Also, please attach a patch file, not the altered source file.

          http://wiki.apache.org/hadoop/HowToContribute

          Show
          Doug Cutting added a comment - This might be better done by: adding JAVA_HEAP_MAX to HADOOP_OPTS if it does not already contain -Xmx removing JAVA_HEAP_MAX from the java command Also, please attach a patch file, not the altered source file. http://wiki.apache.org/hadoop/HowToContribute
          Hide
          Fernando Padilla added a comment -

          Ok. this is the diff now. Have a look, I cleaned it up a little to do what you suggested.

          Show
          Fernando Padilla added a comment - Ok. this is the diff now. Have a look, I cleaned it up a little to do what you suggested.
          Hide
          Koji Noguchi added a comment -

          how does this not work?

          I agree with ryan. It does work as is (at least in our environment).
          If the goal is to make it look better, I'm fine.

          In our rhel environment, jvm gives preference to latter if argument is provided twice.

          Show
          Koji Noguchi added a comment - how does this not work? I agree with ryan. It does work as is (at least in our environment). If the goal is to make it look better, I'm fine. In our rhel environment, jvm gives preference to latter if argument is provided twice.
          Hide
          Fernando Padilla added a comment -

          Ok. So is this -Xmx behavior documented?

          I guess not, but I guess it is working without any code change.. so you can mark this bug as "invalid" or some such..

          sorry for the confusion.

          Show
          Fernando Padilla added a comment - Ok. So is this -Xmx behavior documented? I guess not, but I guess it is working without any code change.. so you can mark this bug as "invalid" or some such.. sorry for the confusion.
          Hide
          Scott Carey added a comment -

          Having -Xmx listed multiple times is misleading and confusing. It should be listed once, preferably near the beginning. Using tools like 'top' with the front of the command line printed shouldn't lie. IMO listing things as important as -Xmx twice is a bad idea regardless of the actual behavior.

          Show
          Scott Carey added a comment - Having -Xmx listed multiple times is misleading and confusing. It should be listed once, preferably near the beginning. Using tools like 'top' with the front of the command line printed shouldn't lie. IMO listing things as important as -Xmx twice is a bad idea regardless of the actual behavior.
          Hide
          Allen Wittenauer added a comment -

          I agree with Scott. This should be fixed.

          Show
          Allen Wittenauer added a comment - I agree with Scott. This should be fixed.

            People

            • Assignee:
              Unassigned
              Reporter:
              Fernando Padilla
            • Votes:
              2 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:

                Development