Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-11658

Piped commands to hbase shell should return non-zero if shell command failed.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0
    • Fix Version/s: 2.0.0, 0.98.13, 1.2.0
    • Component/s: shell
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Adds a noninteractive mode (-n or --noninteractive) to the hbase shell that exits with a non-zero error code on failed or invalid shell command executions, and exits with a zero error code upon successful execution.

      Description

      See HBASE-11655. We would like the shell to return non-zero when a command that has been piped to it fails in "scripting mode". This could be due to invalid commands or commands issued with invalid arguments.

      This would lower the barrier to entry for hbase admins so they could effectively script some operations.

      1. HBASE_11658-v2.patch
        10 kB
        Sean Busbey
      2. HBASE-11658.manual_test.log
        18 kB
        Sean Busbey
      3. HBASE-11658.patch
        10 kB
        Sean Busbey
      4. test_shell.sh
        3 kB
        Sean Busbey

        Issue Links

          Activity

          Hide
          busbey Sean Busbey added a comment -

          Since this ended up touching a lot of lines, I'm attaching a reviewboard for the change against master.

          Because the shell relies on IRB and IRB eats all exceptions, there's no way to do this without a new flag.

          The new flag '--noninteractive' skips IRB entirely and relies on a small REPL that reuses the underlying components of IRB but allows failure in the case of exceptions.

          I believe I've kept this backwards compatible with previous behavior when the new flag is not present. The new flag also tries to be close to the old behavior (only show stacktraces when debug is on, give a grep-able string that there's an error).

          TestShell passes. cherry pick is clean to branch-1 and 0.98, TestShell passes there as well.

          Show
          busbey Sean Busbey added a comment - Since this ended up touching a lot of lines, I'm attaching a reviewboard for the change against master. Because the shell relies on IRB and IRB eats all exceptions, there's no way to do this without a new flag. The new flag '--noninteractive' skips IRB entirely and relies on a small REPL that reuses the underlying components of IRB but allows failure in the case of exceptions. I believe I've kept this backwards compatible with previous behavior when the new flag is not present. The new flag also tries to be close to the old behavior (only show stacktraces when debug is on, give a grep-able string that there's an error). TestShell passes. cherry pick is clean to branch-1 and 0.98, TestShell passes there as well.
          Hide
          busbey Sean Busbey added a comment -

          attaching patch so that QA can take a run at it.

          Also attaching bash script used for manually verifying behavior and the log from a run of it.

          Show
          busbey Sean Busbey added a comment - attaching patch so that QA can take a run at it. Also attaching bash script used for manually verifying behavior and the log from a run of it.
          Hide
          jmhsieh Jonathan Hsieh added a comment -

          This is nice. I tested on trunk, currently testing on branch-1 and 0.98.

          Enis Soztutar, Andrew Purtell do you guys want this in your respective branches?

          Show
          jmhsieh Jonathan Hsieh added a comment - This is nice. I tested on trunk, currently testing on branch-1 and 0.98. Enis Soztutar , Andrew Purtell do you guys want this in your respective branches?
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12660191/HBASE-11658.patch
          against trunk revision .
          ATTACHMENT ID: 12660191

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 8 new or modified tests.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.procedure.TestProcedureManager
          org.apache.hadoop.hbase.ipc.TestIPC
          org.apache.hadoop.hbase.master.TestClockSkewDetection

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/10323//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10323//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10323//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10323//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10323//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10323//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10323//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10323//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10323//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10323//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10323//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12660191/HBASE-11658.patch against trunk revision . ATTACHMENT ID: 12660191 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 8 new or modified tests. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.procedure.TestProcedureManager org.apache.hadoop.hbase.ipc.TestIPC org.apache.hadoop.hbase.master.TestClockSkewDetection Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/10323//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10323//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10323//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10323//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10323//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10323//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10323//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10323//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10323//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10323//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10323//console This message is automatically generated.
          Hide
          busbey Sean Busbey added a comment -

          Attaching v2 from the RB.

          I took a look at the QA results. Since this patch only touches the shell ruby code, I don't see how it could have caused the test failures in hbase-server or the change in findbugs warnings.

          Show
          busbey Sean Busbey added a comment - Attaching v2 from the RB. I took a look at the QA results. Since this patch only touches the shell ruby code, I don't see how it could have caused the test failures in hbase-server or the change in findbugs warnings.
          Hide
          stack stack added a comment -

          Skimmed. It looks great.

          Failures are infrastructure issues. See https://issues.apache.org/jira/browse/BUILDS-4

          Show
          stack stack added a comment - Skimmed. It looks great. Failures are infrastructure issues. See https://issues.apache.org/jira/browse/BUILDS-4
          Hide
          jmhsieh Jonathan Hsieh added a comment -

          lgtm – applies to branch-1 and 0.98 and looks like it works on both. waiting for their feedback before I commit to master and those branches.

          Show
          jmhsieh Jonathan Hsieh added a comment - lgtm – applies to branch-1 and 0.98 and looks like it works on both. waiting for their feedback before I commit to master and those branches.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12660200/HBASE_11658-v2.patch
          against trunk revision .
          ATTACHMENT ID: 12660200

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 8 new or modified tests.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.master.TestRestartCluster
          org.apache.hadoop.hbase.regionserver.TestRegionReplicas
          org.apache.hadoop.hbase.client.TestReplicasClient

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/10324//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10324//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10324//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10324//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10324//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10324//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10324//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10324//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10324//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10324//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10324//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12660200/HBASE_11658-v2.patch against trunk revision . ATTACHMENT ID: 12660200 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 8 new or modified tests. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.master.TestRestartCluster org.apache.hadoop.hbase.regionserver.TestRegionReplicas org.apache.hadoop.hbase.client.TestReplicasClient Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/10324//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10324//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10324//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10324//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10324//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10324//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10324//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10324//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10324//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10324//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10324//console This message is automatically generated.
          Hide
          jmhsieh Jonathan Hsieh added a comment -

          Been a few days. Just going to commit to trunk.

          Thanks Sean!

          Show
          jmhsieh Jonathan Hsieh added a comment - Been a few days. Just going to commit to trunk. Thanks Sean!
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in HBase-TRUNK #5387 (See https://builds.apache.org/job/HBase-TRUNK/5387/)
          HBASE-11658 Piped commands to hbase shell should return non-zero if shell command failed (Sean Busbey) (jmhsieh: rev 4b8a38ebb5db171e9928ca1645045b2c149bdb74)

          • hbase-shell/src/test/ruby/shell/noninteractive_test.rb
          • hbase-shell/src/main/ruby/shell.rb
          • hbase-shell/src/test/ruby/shell/shell_test.rb
          • hbase-shell/src/main/ruby/shell/commands.rb
          • bin/hirb.rb
          Show
          hudson Hudson added a comment - FAILURE: Integrated in HBase-TRUNK #5387 (See https://builds.apache.org/job/HBase-TRUNK/5387/ ) HBASE-11658 Piped commands to hbase shell should return non-zero if shell command failed (Sean Busbey) (jmhsieh: rev 4b8a38ebb5db171e9928ca1645045b2c149bdb74) hbase-shell/src/test/ruby/shell/noninteractive_test.rb hbase-shell/src/main/ruby/shell.rb hbase-shell/src/test/ruby/shell/shell_test.rb hbase-shell/src/main/ruby/shell/commands.rb bin/hirb.rb
          Hide
          enis Enis Soztutar added a comment -

          Enis Soztutar, Andrew Purtell do you guys want this in your respective branches?

          Sorry I missed this. I'll just commit this to 1.2 seems useful.

          Show
          enis Enis Soztutar added a comment - Enis Soztutar, Andrew Purtell do you guys want this in your respective branches? Sorry I missed this. I'll just commit this to 1.2 seems useful.
          Hide
          apurtell Andrew Purtell added a comment -

          I'll commit to 0.98.

          Show
          apurtell Andrew Purtell added a comment - I'll commit to 0.98.
          Hide
          apurtell Andrew Purtell added a comment -

          Picked to 0.98. Applied cleanly. TestShell passed

          Show
          apurtell Andrew Purtell added a comment - Picked to 0.98. Applied cleanly. TestShell passed
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in HBase-1.2 #55 (See https://builds.apache.org/job/HBase-1.2/55/)
          HBASE-11658 Piped commands to hbase shell should return non-zero if shell command failed (Sean Busbey) (enis: rev d90523355e16d9acb9c44e8e72726df481398d75)

          • hbase-shell/src/main/ruby/shell.rb
          • hbase-shell/src/test/ruby/shell/shell_test.rb
          • hbase-shell/src/test/ruby/shell/noninteractive_test.rb
          • bin/hirb.rb
          • hbase-shell/src/main/ruby/shell/commands.rb
          Show
          hudson Hudson added a comment - FAILURE: Integrated in HBase-1.2 #55 (See https://builds.apache.org/job/HBase-1.2/55/ ) HBASE-11658 Piped commands to hbase shell should return non-zero if shell command failed (Sean Busbey) (enis: rev d90523355e16d9acb9c44e8e72726df481398d75) hbase-shell/src/main/ruby/shell.rb hbase-shell/src/test/ruby/shell/shell_test.rb hbase-shell/src/test/ruby/shell/noninteractive_test.rb bin/hirb.rb hbase-shell/src/main/ruby/shell/commands.rb
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in HBase-0.98 #974 (See https://builds.apache.org/job/HBase-0.98/974/)
          HBASE-11658 Piped commands to hbase shell should return non-zero if shell command failed (Sean Busbey) (apurtell: rev 2f09b39f27707ed05bac35bc3806e9cdb1210cf3)

          • bin/hirb.rb
          • hbase-shell/src/test/ruby/shell/shell_test.rb
          • hbase-shell/src/main/ruby/shell/commands.rb
          • hbase-shell/src/test/ruby/shell/noninteractive_test.rb
          • hbase-shell/src/main/ruby/shell.rb
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in HBase-0.98 #974 (See https://builds.apache.org/job/HBase-0.98/974/ ) HBASE-11658 Piped commands to hbase shell should return non-zero if shell command failed (Sean Busbey) (apurtell: rev 2f09b39f27707ed05bac35bc3806e9cdb1210cf3) bin/hirb.rb hbase-shell/src/test/ruby/shell/shell_test.rb hbase-shell/src/main/ruby/shell/commands.rb hbase-shell/src/test/ruby/shell/noninteractive_test.rb hbase-shell/src/main/ruby/shell.rb
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #927 (See https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/927/)
          HBASE-11658 Piped commands to hbase shell should return non-zero if shell command failed (Sean Busbey) (apurtell: rev 2f09b39f27707ed05bac35bc3806e9cdb1210cf3)

          • hbase-shell/src/main/ruby/shell.rb
          • hbase-shell/src/main/ruby/shell/commands.rb
          • hbase-shell/src/test/ruby/shell/shell_test.rb
          • bin/hirb.rb
          • hbase-shell/src/test/ruby/shell/noninteractive_test.rb
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #927 (See https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/927/ ) HBASE-11658 Piped commands to hbase shell should return non-zero if shell command failed (Sean Busbey) (apurtell: rev 2f09b39f27707ed05bac35bc3806e9cdb1210cf3) hbase-shell/src/main/ruby/shell.rb hbase-shell/src/main/ruby/shell/commands.rb hbase-shell/src/test/ruby/shell/shell_test.rb bin/hirb.rb hbase-shell/src/test/ruby/shell/noninteractive_test.rb

            People

            • Assignee:
              busbey Sean Busbey
              Reporter:
              jmhsieh Jonathan Hsieh
            • Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development