HBase
  1. HBase
  2. HBASE-1687

bin/hbase script doesn't allow for different memory settings for each daemon type

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.20.1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      bin/hbase assumes that all daemon types ( master, regionserver ), all use the same memory settings.. (HBASE_HEAPSIZE).

      1) I propose adding daemon specific OPTS much like hadoop already has ( HBASE_MASTER_OPTS, HBASE_REGIONSERVER_OPTS ).

      So in bin/hbase, we would merge in the daemon specific opts within the case statement as such:

      elif [ "$COMMAND" = "master" ] ; then
      CLASS='org.apache.hadoop.hbase.master.HMaster'
      HBASE_OPTS="$HBASE_OPTS $HBASE_MASTER_OPTS"
      elif [ "$COMMAND" = "regionserver" ] ; then
      CLASS='org.apache.hadoop.hbase.regionserver.HRegionServer'
      HBASE_OPTS="$HBASE_OPTS $HBASE_REGIONSERVER_OPTS"

      the new environment variables can then be added to hbase-env.sh as empty:

      export HBASE_MASTER_OPTS=""
      export HBASE_REGIONSERVER=""
      ..etc

      2) I propose changes to that script to allow overriding the default memory ( HBASE_HEAPSIZE ), with daemon specific OPTS (HBASE_MASTER_OPTS, etc ).

      Basically at the bottom of the bin/hbase script, it will check to see if the user has already set "-Xmx" in the HBASE_OPTS variable.. if so, then it will ignore the JAVA_HEAP_SIZE variable..

      as such:

      1. run it
        if [[ $HBASE_OPTS == -Xmx ]]; then
        exec "$JAVA" $HBASE_OPTS -classpath "$CLASSPATH" $CLASS "$@"
        else
        exec "$JAVA" $JAVA_HEAP_MAX $HBASE_OPTS -classpath "$CLASSPATH" $CLASS "$@"
        fi

      I will attach the file as I have modified it..

      1. hbase
        7 kB
        Fernando Padilla
      2. hbase-script.diff
        1 kB
        Fernando Padilla
      3. HBASE-1687-ryan.patch
        1.0 kB
        ryan rawson

        Activity

        Hide
        stack added a comment -

        Committed to trunk and branch.

        Changed name of the regionserver variable from HBASE_RS_OPTS to be HBASE_REGIONSERVER_OPTS to be consistent.

        Also updated the metrics.xml doc. that gario recently contributed so it no longer describes adding above HBASE_*_OPTS.. no need now they have been committed.

        Show
        stack added a comment - Committed to trunk and branch. Changed name of the regionserver variable from HBASE_RS_OPTS to be HBASE_REGIONSERVER_OPTS to be consistent. Also updated the metrics.xml doc. that gario recently contributed so it no longer describes adding above HBASE_*_OPTS.. no need now they have been committed.
        Hide
        ryan rawson added a comment -

        here is my implementation of this, modelled after how hadoop does it. In my setup I end up with this:

        14682 ? SNl 3:25 /usr/java/jdk1.6.0_14/bin/java -Xmx4000m -Xmx1000m -XX:+DoEscapeAnalysis -XX:+AggressiveOpts -XX:+UseConcMarkSweepGC ....

        and jmap says:
        $ jmap -heap 14682
        Attaching to process ID 14682, please wait...
        Debugger attached successfully.
        Server compiler detected.
        JVM version is 14.0-b13

        using parallel threads in the new generation.
        using thread-local object allocation.
        Concurrent Mark-Sweep GC

        Heap Configuration:
        MinHeapFreeRatio = 40
        MaxHeapFreeRatio = 70
        MaxHeapSize = 1048576000 (1000.0MB)
        NewSize = 67108864 (64.0MB)
        MaxNewSize = 67108864 (64.0MB)
        OldSize = 201326592 (192.0MB)
        NewRatio = 7
        SurvivorRatio = 8
        PermSize = 21757952 (20.75MB)
        MaxPermSize = 88080384 (84.0MB)

        so java uses the last listed Xmx as the one to use.

        Show
        ryan rawson added a comment - here is my implementation of this, modelled after how hadoop does it. In my setup I end up with this: 14682 ? SNl 3:25 /usr/java/jdk1.6.0_14/bin/java -Xmx4000m -Xmx1000m -XX:+DoEscapeAnalysis -XX:+AggressiveOpts -XX:+UseConcMarkSweepGC .... and jmap says: $ jmap -heap 14682 Attaching to process ID 14682, please wait... Debugger attached successfully. Server compiler detected. JVM version is 14.0-b13 using parallel threads in the new generation. using thread-local object allocation. Concurrent Mark-Sweep GC Heap Configuration: MinHeapFreeRatio = 40 MaxHeapFreeRatio = 70 MaxHeapSize = 1048576000 (1000.0MB) NewSize = 67108864 (64.0MB) MaxNewSize = 67108864 (64.0MB) OldSize = 201326592 (192.0MB) NewRatio = 7 SurvivorRatio = 8 PermSize = 21757952 (20.75MB) MaxPermSize = 88080384 (84.0MB) so java uses the last listed Xmx as the one to use.
        Hide
        stack added a comment -

        I tried this.

        I added defaults to hbase-env.sh and to test uncommented some:

        Index: conf/hbase-env.sh
        ===================================================================
        --- conf/hbase-env.sh   (revision 798386)
        +++ conf/hbase-env.sh   (working copy)
        @@ -23,6 +23,7 @@
         
         # The java implementation to use.  Java 1.6 required.
         # export JAVA_HOME=/usr/java/jdk1.6.0/
        +export JAVA_HOME="/System/Library/Frameworks/JavaVM.framework/Versions/1.6/Home/"
         
         # Extra Java CLASSPATH elements.  Optional.
         # export HBASE_CLASSPATH=
        @@ -30,6 +31,17 @@
         # The maximum amount of heap to use, in MB. Default is 1000.
         # export HBASE_HEAPSIZE=1000
         
        +# Heap to give regionserver. Default is HBASE_HEAPSIZE.
        +# export HBASE_REGIONSERVER_OPTS=${HBASE_HEAPSIZE}
        +
        +# Heap to give master. Default is HBASE_HEAPSIZE.
        +# export HBASE_MASTER_OPTS=${HBASE_HEAPSIZE}
        +export HBASE_MASTER_OPTS="-Xmx513m -Xms513m"
        +
        +# Heap to give master. Default is HBASE_HEAPSIZE.
        +# export HBASE_QUORUMPEER_OPTS=${HBASE_HEAPSIZE}
        +export HBASE_QUORUMPEER_OPTS="-Xmx514m -Xms514m"
        +
         # Extra Java runtime options.
         # Below are what we set by default.  May only work with SUN JVM.
         # For more on why as well as other possible settings,
        

        In my process listing I see that the HBASE_HEAPSIZE always comes afterward so if java takes last -Xms, then it will always take default. So we could swap HBASE_HEAPSIZE and HBASE_REGIONSERVER_OPTS, etc.? Is that the idea ? As is it don't seem to be too useful.

        Show
        stack added a comment - I tried this. I added defaults to hbase-env.sh and to test uncommented some: Index: conf/hbase-env.sh =================================================================== --- conf/hbase-env.sh (revision 798386) +++ conf/hbase-env.sh (working copy) @@ -23,6 +23,7 @@ # The java implementation to use. Java 1.6 required. # export JAVA_HOME=/usr/java/jdk1.6.0/ +export JAVA_HOME= "/ System /Library/Frameworks/JavaVM.framework/Versions/1.6/Home/" # Extra Java CLASSPATH elements. Optional. # export HBASE_CLASSPATH= @@ -30,6 +31,17 @@ # The maximum amount of heap to use, in MB. Default is 1000. # export HBASE_HEAPSIZE=1000 +# Heap to give regionserver. Default is HBASE_HEAPSIZE. +# export HBASE_REGIONSERVER_OPTS=${HBASE_HEAPSIZE} + +# Heap to give master. Default is HBASE_HEAPSIZE. +# export HBASE_MASTER_OPTS=${HBASE_HEAPSIZE} +export HBASE_MASTER_OPTS= "-Xmx513m -Xms513m" + +# Heap to give master. Default is HBASE_HEAPSIZE. +# export HBASE_QUORUMPEER_OPTS=${HBASE_HEAPSIZE} +export HBASE_QUORUMPEER_OPTS= "-Xmx514m -Xms514m" + # Extra Java runtime options. # Below are what we set by default . May only work with SUN JVM. # For more on why as well as other possible settings, In my process listing I see that the HBASE_HEAPSIZE always comes afterward so if java takes last -Xms, then it will always take default. So we could swap HBASE_HEAPSIZE and HBASE_REGIONSERVER_OPTS, etc.? Is that the idea ? As is it don't seem to be too useful.
        Hide
        Fernando Padilla added a comment -

        1) we still need support for OPTS for each daemon type.

        2) we might not need the fancy JAVA_HEAP_SIZE handling, since it looks like java ignores duplicate Xmx settings, and keeps the last one in the command line.. so it looks ok.. as long as the default is put first in the command line.. (but changing the script does make it a little cleaner..)

        Show
        Fernando Padilla added a comment - 1) we still need support for OPTS for each daemon type. 2) we might not need the fancy JAVA_HEAP_SIZE handling, since it looks like java ignores duplicate Xmx settings, and keeps the last one in the command line.. so it looks ok.. as long as the default is put first in the command line.. (but changing the script does make it a little cleaner..)
        Hide
        Fernando Padilla added a comment -

        ok, this is the diff. Please have a look, it should be pretty straight forward.

        I changed it a little bit from a Doug Cutting suggestion to add JAVA_HEAP_SIZE into the H*_OPTS before the exec line.. and remove it from the exec line..

        Show
        Fernando Padilla added a comment - ok, this is the diff. Please have a look, it should be pretty straight forward. I changed it a little bit from a Doug Cutting suggestion to add JAVA_HEAP_SIZE into the H*_OPTS before the exec line.. and remove it from the exec line..
        Hide
        Fernando Padilla added a comment -

        This is the related hadoop bug: https://issues.apache.org/jira/browse/HADOOP-6167

        Show
        Fernando Padilla added a comment - This is the related hadoop bug: https://issues.apache.org/jira/browse/HADOOP-6167
        Hide
        Fernando Padilla added a comment -

        sadly, hadoop has this bug as well

        I submitted pretty much this exact same bug against bin/hadoop (just changed hadoop with hbase)

        I just realized, it's hard to see because this wiki thinks I'm trying to bold the code, but there are '*' around the -Xmx, those are bash globs.. so it means.. if the $var equals <glob>-Xmx<glob> so it should match -Xmx anywhere in the string.. try it out..

        you should be able to create a simple bash file.. that has a variable, then an if statement like above, then echo "CONTAINS" else echo "NOT CONTAINS", and you'll see it working..

        Show
        Fernando Padilla added a comment - sadly, hadoop has this bug as well I submitted pretty much this exact same bug against bin/hadoop (just changed hadoop with hbase) I just realized, it's hard to see because this wiki thinks I'm trying to bold the code, but there are '*' around the -Xmx, those are bash globs.. so it means.. if the $var equals <glob>-Xmx<glob> so it should match -Xmx anywhere in the string.. try it out.. you should be able to create a simple bash file.. that has a variable, then an if statement like above, then echo "CONTAINS" else echo "NOT CONTAINS", and you'll see it working..
        Hide
        ryan rawson added a comment -

        what if my HBASE_OPTS dont start with -Xmx, what then?

        I think we should copy whatever hadoop is doing here, they have multiple options for each type of daemon.

        Show
        ryan rawson added a comment - what if my HBASE_OPTS dont start with -Xmx, what then? I think we should copy whatever hadoop is doing here, they have multiple options for each type of daemon.
        Hide
        Fernando Padilla added a comment -

        Sorry about not being in a patch form, I have been a little lazy because I actually built it in ec2.. since at the moment I can't do 1.6 dev work on my workstation (soon). So I just copied the whole file

        It is from the latest trunk if you want to generate the diff.. else I'll do it later today or tomorrow...

        The bash "contains" ( == -Xmx ) I got from a StackOverflow question. I am not a bash expert, so I just googled "bash contains", and people suggested various things ( like using grep ), but this one did show up a few times, so I went with it.. it looked a little big cleaner.

        Show
        Fernando Padilla added a comment - Sorry about not being in a patch form, I have been a little lazy because I actually built it in ec2.. since at the moment I can't do 1.6 dev work on my workstation (soon). So I just copied the whole file It is from the latest trunk if you want to generate the diff.. else I'll do it later today or tomorrow... The bash "contains" ( == -Xmx ) I got from a StackOverflow question. I am not a bash expert, so I just googled "bash contains", and people suggested various things ( like using grep ), but this one did show up a few times, so I went with it.. it looked a little big cleaner.
        Hide
        stack added a comment -

        This change looks good. We need it.

        Do you know how to make a patch? That'd be better but I can work with what you've put above.

        The only thing that concerns me is whether this test:

        if [[ $HBASE_OPTS == *-Xmx* ]]; then
        

        ... is portable bash. Where did you get it from? Do you know it will work on bash on other operating systems?

        Thanks.

        Show
        stack added a comment - This change looks good. We need it. Do you know how to make a patch? That'd be better but I can work with what you've put above. The only thing that concerns me is whether this test: if [[ $HBASE_OPTS == *-Xmx* ]]; then ... is portable bash. Where did you get it from? Do you know it will work on bash on other operating systems? Thanks.

          People

          • Assignee:
            Unassigned
            Reporter:
            Fernando Padilla
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development