HBase
  1. HBase
  2. HBASE-7153

print gc option in hbase-env.sh affects hbase zkcli

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.94.0
    • Fix Version/s: 0.98.0, 0.94.6, 0.95.0
    • Component/s: scripts
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      I un-commented the -verbose:gc option in hbase-env.sh, which print out the gc info.
      but when I use hbase zkcli to check zk, it can not connect to the server.
      the problem is zkcli uses "hbase org.apache.hadoop.hbase.zookeeper.ZooKeeperMainServerArg" to get the server_arg in the script hbase. when gc verbose option is open, the output of ZooKeeperMainServerArg is with gc info, which masses up with server_arg. and this is the reason stop zkcli working.
      I think the easiest way to fix this is to trim the gc info out of server_arg in the hbase script.

      1. 7153-0.94.txt
        0.4 kB
        Lars Hofhansl
      2. HBASE-7153-0.94.patch
        0.7 kB
        Dave Latham

        Issue Links

          Activity

          Hide
          Zavier Gao added a comment -

          replace the line [ SERVER_ARG=`"$bin"/hbase org.apache.hadoop.hbase.zookeeper.ZooKeeperMainServerArg`] with [ SERVER_ARG=`"$bin"/hbase org.apache.hadoop.hbase.zookeeper.ZooKeeperMainServerArg | sed -n -e '1p'` ] will solve this issue.

          Show
          Zavier Gao added a comment - replace the line [ SERVER_ARG=`"$bin"/hbase org.apache.hadoop.hbase.zookeeper.ZooKeeperMainServerArg`] with [ SERVER_ARG=`"$bin"/hbase org.apache.hadoop.hbase.zookeeper.ZooKeeperMainServerArg | sed -n -e '1p'` ] will solve this issue.
          Hide
          Dave Latham added a comment -

          This issue was introduced by HBASE-7091.
          If SERVER_GC_OPTS is uncommented in hbase-env.sh then GC output appears with everything aside from hbase shell, because the bash array syntax is broken instead.

          I propose to fix the array syntax and change the list so that the server GC args apply to a whitelist of known server commands so that additional misc client tools (like org.apache.hadoop.hbase.zookeeper.ZooKeeperMainServerArg or region_mover.rb) are not affected.

          Show
          Dave Latham added a comment - This issue was introduced by HBASE-7091 . If SERVER_GC_OPTS is uncommented in hbase-env.sh then GC output appears with everything aside from hbase shell, because the bash array syntax is broken instead. I propose to fix the array syntax and change the list so that the server GC args apply to a whitelist of known server commands so that additional misc client tools (like org.apache.hadoop.hbase.zookeeper.ZooKeeperMainServerArg or region_mover.rb) are not affected.
          Hide
          Dave Latham added a comment -

          Patch for 0.94 branch - may work on trunk as well.

          Show
          Dave Latham added a comment - Patch for 0.94 branch - may work on trunk as well.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12572255/HBASE-7153-0.94.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.master.TestSplitLogManager
          org.apache.hadoop.hbase.master.TestMasterFailover
          org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4692//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4692//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4692//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4692//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4692//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4692//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4692//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4692//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4692//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4692//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12572255/HBASE-7153-0.94.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.master.TestSplitLogManager org.apache.hadoop.hbase.master.TestMasterFailover org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4692//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4692//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4692//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4692//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4692//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4692//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4692//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4692//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4692//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4692//console This message is automatically generated.
          Hide
          Lars Hofhansl added a comment -

          Why not just fix the array syntax?
          Jesse Yates please have a look.

          Show
          Lars Hofhansl added a comment - Why not just fix the array syntax? Jesse Yates please have a look.
          Hide
          Jesse Yates added a comment -

          The idea for HBASE-7091 was to avoid these cases. Do you have special GC options for the shell that wouldn't necessarily apply to zkcli? I can't think of any off the top of my head. It seems like fixing the array syntax should be sufficient

          Show
          Jesse Yates added a comment - The idea for HBASE-7091 was to avoid these cases. Do you have special GC options for the shell that wouldn't necessarily apply to zkcli? I can't think of any off the top of my head. It seems like fixing the array syntax should be sufficient
          Hide
          Lars Hofhansl added a comment -

          I prefer this.
          If there are no objection I would like to commit this tonight so that I can cut a new RC.

          Show
          Lars Hofhansl added a comment - I prefer this. If there are no objection I would like to commit this tonight so that I can cut a new RC.
          Hide
          Jesse Yates added a comment -

          +1 on latest attachment.

          Show
          Jesse Yates added a comment - +1 on latest attachment.
          Hide
          Lars Hofhansl added a comment -

          Committed to 0.94, 0.95, 0.98.
          Thanks for pointing out the flaw Dave!
          (We can change the meaning around in another jira if you think that would be better going forward)

          Show
          Lars Hofhansl added a comment - Committed to 0.94, 0.95, 0.98. Thanks for pointing out the flaw Dave! (We can change the meaning around in another jira if you think that would be better going forward)
          Hide
          Hudson added a comment -

          Integrated in HBase-0.94-security #118 (See https://builds.apache.org/job/HBase-0.94-security/118/)
          HBASE-7153 print gc option in hbase-env.sh affects hbase zkcli (Dave Latham and LarsH) (Revision 1453684)

          Result = SUCCESS
          larsh :
          Files :

          • /hbase/branches/0.94/bin/hbase
          Show
          Hudson added a comment - Integrated in HBase-0.94-security #118 (See https://builds.apache.org/job/HBase-0.94-security/118/ ) HBASE-7153 print gc option in hbase-env.sh affects hbase zkcli (Dave Latham and LarsH) (Revision 1453684) Result = SUCCESS larsh : Files : /hbase/branches/0.94/bin/hbase
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12572489/7153-0.94.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.master.TestMasterFailover

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4710//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4710//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4710//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4710//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4710//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4710//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4710//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4710//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4710//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4710//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12572489/7153-0.94.txt against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.master.TestMasterFailover Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4710//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4710//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4710//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4710//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4710//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4710//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4710//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4710//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4710//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4710//console This message is automatically generated.
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #3922 (See https://builds.apache.org/job/HBase-TRUNK/3922/)
          HBASE-7153 print gc option in hbase-env.sh affects hbase zkcli (Dave Latham and LarsH) (Revision 1453686)

          Result = FAILURE
          larsh :
          Files :

          • /hbase/trunk/bin/hbase
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #3922 (See https://builds.apache.org/job/HBase-TRUNK/3922/ ) HBASE-7153 print gc option in hbase-env.sh affects hbase zkcli (Dave Latham and LarsH) (Revision 1453686) Result = FAILURE larsh : Files : /hbase/trunk/bin/hbase
          Hide
          Hudson added a comment -

          Integrated in hbase-0.95 #33 (See https://builds.apache.org/job/hbase-0.95/33/)
          HBASE-7153 print gc option in hbase-env.sh affects hbase zkcli (Dave Latham and LarsH) (Revision 1453685)

          Result = FAILURE
          larsh :
          Files :

          • /hbase/branches/0.95/bin/hbase
          Show
          Hudson added a comment - Integrated in hbase-0.95 #33 (See https://builds.apache.org/job/hbase-0.95/33/ ) HBASE-7153 print gc option in hbase-env.sh affects hbase zkcli (Dave Latham and LarsH) (Revision 1453685) Result = FAILURE larsh : Files : /hbase/branches/0.95/bin/hbase
          Hide
          Hudson added a comment -

          Integrated in HBase-0.94 #885 (See https://builds.apache.org/job/HBase-0.94/885/)
          HBASE-7153 print gc option in hbase-env.sh affects hbase zkcli (Dave Latham and LarsH) (Revision 1453684)

          Result = FAILURE
          larsh :
          Files :

          • /hbase/branches/0.94/bin/hbase
          Show
          Hudson added a comment - Integrated in HBase-0.94 #885 (See https://builds.apache.org/job/HBase-0.94/885/ ) HBASE-7153 print gc option in hbase-env.sh affects hbase zkcli (Dave Latham and LarsH) (Revision 1453684) Result = FAILURE larsh : Files : /hbase/branches/0.94/bin/hbase
          Hide
          Hudson added a comment -

          Integrated in hbase-0.95-on-hadoop2 #15 (See https://builds.apache.org/job/hbase-0.95-on-hadoop2/15/)
          HBASE-7153 print gc option in hbase-env.sh affects hbase zkcli (Dave Latham and LarsH) (Revision 1453685)

          Result = FAILURE
          larsh :
          Files :

          • /hbase/branches/0.95/bin/hbase
          Show
          Hudson added a comment - Integrated in hbase-0.95-on-hadoop2 #15 (See https://builds.apache.org/job/hbase-0.95-on-hadoop2/15/ ) HBASE-7153 print gc option in hbase-env.sh affects hbase zkcli (Dave Latham and LarsH) (Revision 1453685) Result = FAILURE larsh : Files : /hbase/branches/0.95/bin/hbase
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #434 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/434/)
          HBASE-7153 print gc option in hbase-env.sh affects hbase zkcli (Dave Latham and LarsH) (Revision 1453686)

          Result = FAILURE
          larsh :
          Files :

          • /hbase/trunk/bin/hbase
          Show
          Hudson added a comment - Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #434 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/434/ ) HBASE-7153 print gc option in hbase-env.sh affects hbase zkcli (Dave Latham and LarsH) (Revision 1453686) Result = FAILURE larsh : Files : /hbase/trunk/bin/hbase
          Hide
          Dave Latham added a comment -

          Thanks guys for taking a look.

          Fixing the array syntax alone is not enough. When you invoke "hbase zkcli" it turns around and first invokes "hbase org.apache.hadoop.hbase.zookeeper.ZooKeeperMainServerArg" which will then pick up the server options, cause GC output to appear in standard out, which then gets picked up on the command line for zkcli causing it to fail.

          Also, there are many other tools you can invoke that are more likely to "client" than "server" options. For example, "bin/hbase org.jruby.Main region_mover.rb" or "bin/hbase org.apache.hadoop.hbase.mapreduce.CopyTable" or "bin/hbase version" or "bin/hbase org.apache.hadoop.hbase.mapreduce.Export". The whitelist of server commands is shorter and easier to maintain than a whitelist of client commands.

          Show
          Dave Latham added a comment - Thanks guys for taking a look. Fixing the array syntax alone is not enough. When you invoke "hbase zkcli" it turns around and first invokes "hbase org.apache.hadoop.hbase.zookeeper.ZooKeeperMainServerArg" which will then pick up the server options, cause GC output to appear in standard out, which then gets picked up on the command line for zkcli causing it to fail. Also, there are many other tools you can invoke that are more likely to "client" than "server" options. For example, "bin/hbase org.jruby.Main region_mover.rb" or "bin/hbase org.apache.hadoop.hbase.mapreduce.CopyTable" or "bin/hbase version" or "bin/hbase org.apache.hadoop.hbase.mapreduce.Export". The whitelist of server commands is shorter and easier to maintain than a whitelist of client commands.
          Hide
          Dave Latham added a comment -

          Created HBASE-8025 for follow up.

          Show
          Dave Latham added a comment - Created HBASE-8025 for follow up.
          Hide
          Lars Hofhansl added a comment -

          Thanks Dave. I see, you're right. Since this is wrong, we can it here or in HBASE-8025.

          Show
          Lars Hofhansl added a comment - Thanks Dave. I see, you're right. Since this is wrong, we can it here or in HBASE-8025 .
          Hide
          Dave Latham added a comment -

          Since this one is committed, let's continue in HBASE-8025.

          Show
          Dave Latham added a comment - Since this one is committed, let's continue in HBASE-8025 .
          Hide
          Hudson added a comment -

          Integrated in HBase-0.94-security-on-Hadoop-23 #13 (See https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/13/)
          HBASE-7153 print gc option in hbase-env.sh affects hbase zkcli (Dave Latham and LarsH) (Revision 1453684)

          Result = FAILURE
          larsh :
          Files :

          • /hbase/branches/0.94/bin/hbase
          Show
          Hudson added a comment - Integrated in HBase-0.94-security-on-Hadoop-23 #13 (See https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/13/ ) HBASE-7153 print gc option in hbase-env.sh affects hbase zkcli (Dave Latham and LarsH) (Revision 1453684) Result = FAILURE larsh : Files : /hbase/branches/0.94/bin/hbase

            People

            • Assignee:
              Dave Latham
              Reporter:
              wonderyl
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development