Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.99.0, 0.96.3, 0.98.5, 0.94.22, 2.0.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Set HBASE_SHELL_OPTS in the environment or in hbase-env.sh to supply extra VM arguments when launching the HBase command shell.

      Description

      Over in hbase-2177 Ryan goes on how enabling gc logging, it shows on stdout when you fire the shell:

      so one problem with this is the irb then logs all GC to stdout, which is ugly.  I do something like this in my scripts:
      
      export HBASE_OPTS=""
      export HBASE_LOG_DIR=<somewhere>
      
      export SERVER_GC_OPTS="$HBASE_OPTS -verbose:gc -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps -XX:+PrintGCDetails -Xloggc:$HBASE_HOME/logs/gc-hbase.log"
      
      export JMX_OPTS="-Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.password.file=$HBASE_HOME/conf/jmxremote.password -Dcom.sun.management.jmxremote"
      
      
      export HBASE_MASTER_OPTS="$SERVER_GC_OPTS -Xloggc:$HBASE_LOG_DIR/logs/gc-master.log"
      export HBASE_REGIONSERVER_OPTS="$SERVER_GC_OPTS -Xloggc:$HBASE_LOG_DIR/gc-hbase.log -Dcom.sun.management.jmxremote.port=10102 $JMX_OPTS"
      export HBASE_THRIFT_OPTS="-Xmx1000m $SERVER_GC_OPTS -Xloggc:$HBASE_LOG_DIR/gc-hbase-thrift.log -Dcom.sun.management.jmxremote.port=10103 $JMX_OPTS"
      export HBASE_ZOOKEEPER_OPTS="-Xmx1000m $SERVER_GC_OPTS -Xloggc:$HBASE_LOG_DIR/gc-zk.log -Dcom.sun.management.jmxremote.port=10104 $JMX_OPTS"
      
      
      
      now you get remote JMX with logging to whatever directory (we have to log to our large data partition since logs... can be big).  Also the shell doesnt log GC to stdout, and you can get separate GC logs for hmaster, hrs, thrift, zookeeper.
      

      Need to make an OPTS for the shell to use.... or do the above.

      1. HBASE-2217.patch
        0.7 kB
        Andrew Purtell

        Activity

        Hide
        Andrew Purtell added a comment - - edited

        Going to commit trivial patch shortly to 0.94+

        Show
        Andrew Purtell added a comment - - edited Going to commit trivial patch shortly to 0.94+
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12656154/HBASE-2217.patch
        against trunk revision .
        ATTACHMENT ID: 12656154

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        +1 site. The mvn site goal succeeds with this patch.

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/10099//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10099//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10099//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10099//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10099//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10099//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10099//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10099//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10099//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10099//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10099//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12656154/HBASE-2217.patch against trunk revision . ATTACHMENT ID: 12656154 +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/10099//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10099//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10099//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10099//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10099//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10099//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10099//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10099//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10099//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10099//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10099//console This message is automatically generated.
        Hide
        stack added a comment -

        +1

        Show
        stack added a comment - +1
        Hide
        stack added a comment -

        Is the idea that we'd set HBASE_SHELL_OPTS externally when we call bin/hbase?

        Show
        stack added a comment - Is the idea that we'd set HBASE_SHELL_OPTS externally when we call bin/hbase?
        Hide
        Andrew Purtell added a comment -

        Is the idea that we'd set HBASE_SHELL_OPTS externally when we call bin/hbase?

        Yep, or in hbase-env.sh, like HBASE_MASTER_OPTS, HBASE_REGIONSERVER_OPTS, and such

        Show
        Andrew Purtell added a comment - Is the idea that we'd set HBASE_SHELL_OPTS externally when we call bin/hbase? Yep, or in hbase-env.sh, like HBASE_MASTER_OPTS, HBASE_REGIONSERVER_OPTS, and such
        Hide
        stack added a comment -

        grand

        Show
        stack added a comment - grand
        Hide
        Andrew Purtell added a comment -

        Thanks Stack.

        Pushed trivial change to 0.94+.

        Tested manually locally with 0.94 and 0.98.

        $ mvn -DskipTests clean install
        $ ./bin/hbase shell
        (shell starts)
        ^D
        $ HBASE_SHELL_OPTS="-XX:+Die" ./bin/hbase
        Unrecognized VM option 'Die'
        Error: Could not create the Java Virtual Machine.
        ..
        
        Show
        Andrew Purtell added a comment - Thanks Stack. Pushed trivial change to 0.94+. Tested manually locally with 0.94 and 0.98. $ mvn -DskipTests clean install $ ./bin/hbase shell (shell starts) ^D $ HBASE_SHELL_OPTS="-XX:+Die" ./bin/hbase Unrecognized VM option 'Die' Error: Could not create the Java Virtual Machine. ..
        Hide
        Hudson added a comment -

        FAILURE: Integrated in hbase-0.96 #409 (See https://builds.apache.org/job/hbase-0.96/409/)
        HBASE-2217 VM OPTS for shell only (apurtell: rev a2b70c7efaf5781c0a44e31213b32bbaa67abe77)

        • bin/hbase
        Show
        Hudson added a comment - FAILURE: Integrated in hbase-0.96 #409 (See https://builds.apache.org/job/hbase-0.96/409/ ) HBASE-2217 VM OPTS for shell only (apurtell: rev a2b70c7efaf5781c0a44e31213b32bbaa67abe77) bin/hbase
        Hide
        Hudson added a comment -

        FAILURE: Integrated in HBase-1.0 #51 (See https://builds.apache.org/job/HBase-1.0/51/)
        HBASE-2217 VM OPTS for shell only (apurtell: rev 94f767fcc410d38648f21d449da2f5faaf7d7432)

        • bin/hbase
        Show
        Hudson added a comment - FAILURE: Integrated in HBase-1.0 #51 (See https://builds.apache.org/job/HBase-1.0/51/ ) HBASE-2217 VM OPTS for shell only (apurtell: rev 94f767fcc410d38648f21d449da2f5faaf7d7432) bin/hbase
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-0.94-security #500 (See https://builds.apache.org/job/HBase-0.94-security/500/)
        HBASE-2217 VM OPTS for shell only (apurtell: rev 225561cc6d6f69fe948b908bb305a73faab9c336)

        • bin/hbase
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-0.94-security #500 (See https://builds.apache.org/job/HBase-0.94-security/500/ ) HBASE-2217 VM OPTS for shell only (apurtell: rev 225561cc6d6f69fe948b908bb305a73faab9c336) bin/hbase
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-TRUNK #5316 (See https://builds.apache.org/job/HBase-TRUNK/5316/)
        HBASE-2217 VM OPTS for shell only (apurtell: rev a46299076512faf631936167dd2ab34c25c84735)

        • bin/hbase
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-TRUNK #5316 (See https://builds.apache.org/job/HBase-TRUNK/5316/ ) HBASE-2217 VM OPTS for shell only (apurtell: rev a46299076512faf631936167dd2ab34c25c84735) bin/hbase
        Hide
        Hudson added a comment -

        FAILURE: Integrated in HBase-0.94-JDK7 #155 (See https://builds.apache.org/job/HBase-0.94-JDK7/155/)
        HBASE-2217 VM OPTS for shell only (apurtell: rev 225561cc6d6f69fe948b908bb305a73faab9c336)

        • bin/hbase
        Show
        Hudson added a comment - FAILURE: Integrated in HBase-0.94-JDK7 #155 (See https://builds.apache.org/job/HBase-0.94-JDK7/155/ ) HBASE-2217 VM OPTS for shell only (apurtell: rev 225561cc6d6f69fe948b908bb305a73faab9c336) bin/hbase
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #383 (See https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/383/)
        HBASE-2217 VM OPTS for shell only (apurtell: rev e42883f2579e9ee2c4feab91794b83066a8e29bb)

        • bin/hbase
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #383 (See https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/383/ ) HBASE-2217 VM OPTS for shell only (apurtell: rev e42883f2579e9ee2c4feab91794b83066a8e29bb) bin/hbase
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-0.98 #403 (See https://builds.apache.org/job/HBase-0.98/403/)
        HBASE-2217 VM OPTS for shell only (apurtell: rev e42883f2579e9ee2c4feab91794b83066a8e29bb)

        • bin/hbase
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-0.98 #403 (See https://builds.apache.org/job/HBase-0.98/403/ ) HBASE-2217 VM OPTS for shell only (apurtell: rev e42883f2579e9ee2c4feab91794b83066a8e29bb) bin/hbase
        Hide
        Hudson added a comment -

        FAILURE: Integrated in HBase-0.94 #1387 (See https://builds.apache.org/job/HBase-0.94/1387/)
        HBASE-2217 VM OPTS for shell only (apurtell: rev 225561cc6d6f69fe948b908bb305a73faab9c336)

        • bin/hbase
        Show
        Hudson added a comment - FAILURE: Integrated in HBase-0.94 #1387 (See https://builds.apache.org/job/HBase-0.94/1387/ ) HBASE-2217 VM OPTS for shell only (apurtell: rev 225561cc6d6f69fe948b908bb305a73faab9c336) bin/hbase
        Hide
        Hudson added a comment -

        FAILURE: Integrated in hbase-0.96-hadoop2 #283 (See https://builds.apache.org/job/hbase-0.96-hadoop2/283/)
        HBASE-2217 VM OPTS for shell only (apurtell: rev a2b70c7efaf5781c0a44e31213b32bbaa67abe77)

        • bin/hbase
        Show
        Hudson added a comment - FAILURE: Integrated in hbase-0.96-hadoop2 #283 (See https://builds.apache.org/job/hbase-0.96-hadoop2/283/ ) HBASE-2217 VM OPTS for shell only (apurtell: rev a2b70c7efaf5781c0a44e31213b32bbaa67abe77) bin/hbase

          People

          • Assignee:
            Andrew Purtell
            Reporter:
            stack
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development