HBase
  1. HBase
  2. HBASE-7204

Make hbck ErrorReporter pluggable

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.94.4, 0.95.0
    • Component/s: hbck
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Now hbck runs with ToolRunner, and can accept configurations from command line.

      Description

      Make hbck ErrorReporter pluggable so that it can be replaced dynamically.

      1. trunk-7204.patch
        7 kB
        Jimmy Xiang
      2. trunk-7204.addendum
        2 kB
        Jimmy Xiang
      3. trunk-7204_v2.patch
        21 kB
        Jimmy Xiang
      4. trunk-7204_v2.1.patch
        21 kB
        Jimmy Xiang
      5. 0.94-7204.patch
        20 kB
        Jimmy Xiang

        Activity

        Hide
        Hudson added a comment -

        Integrated in HBase-0.94-security-on-Hadoop-23 #10 (See https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/10/)
        HBASE-7204 Make hbck ErrorReporter pluggable, ADDENDUM (Revision 1415898)
        HBASE-7204 Make hbck ErrorReporter pluggable (Revision 1415876)

        Result = FAILURE
        jxiang :
        Files :

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
        • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java

        jxiang :
        Files :

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
        • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
        Show
        Hudson added a comment - Integrated in HBase-0.94-security-on-Hadoop-23 #10 (See https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/10/ ) HBASE-7204 Make hbck ErrorReporter pluggable, ADDENDUM (Revision 1415898) HBASE-7204 Make hbck ErrorReporter pluggable (Revision 1415876) Result = FAILURE jxiang : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java jxiang : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
        Hide
        Hudson added a comment -

        Integrated in HBase-0.94-security #86 (See https://builds.apache.org/job/HBase-0.94-security/86/)
        HBASE-7204 Make hbck ErrorReporter pluggable, ADDENDUM (Revision 1415898)
        HBASE-7204 Make hbck ErrorReporter pluggable (Revision 1415876)

        Result = SUCCESS
        jxiang :
        Files :

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
        • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java

        jxiang :
        Files :

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
        • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
        Show
        Hudson added a comment - Integrated in HBase-0.94-security #86 (See https://builds.apache.org/job/HBase-0.94-security/86/ ) HBASE-7204 Make hbck ErrorReporter pluggable, ADDENDUM (Revision 1415898) HBASE-7204 Make hbck ErrorReporter pluggable (Revision 1415876) Result = SUCCESS jxiang : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java jxiang : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #283 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/283/)
        HBASE-7204 Make hbck ErrorReporter pluggable, ADDENDUM (Revision 1415897)

        Result = FAILURE
        jxiang :
        Files :

        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #283 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/283/ ) HBASE-7204 Make hbck ErrorReporter pluggable, ADDENDUM (Revision 1415897) Result = FAILURE jxiang : Files : /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
        Hide
        Hudson added a comment -

        Integrated in HBase-0.94 #609 (See https://builds.apache.org/job/HBase-0.94/609/)
        HBASE-7204 Make hbck ErrorReporter pluggable, ADDENDUM (Revision 1415898)

        Result = SUCCESS
        jxiang :
        Files :

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
        • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
        Show
        Hudson added a comment - Integrated in HBase-0.94 #609 (See https://builds.apache.org/job/HBase-0.94/609/ ) HBASE-7204 Make hbck ErrorReporter pluggable, ADDENDUM (Revision 1415898) Result = SUCCESS jxiang : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #3585 (See https://builds.apache.org/job/HBase-TRUNK/3585/)
        HBASE-7204 Make hbck ErrorReporter pluggable, ADDENDUM (Revision 1415897)

        Result = FAILURE
        jxiang :
        Files :

        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK #3585 (See https://builds.apache.org/job/HBase-TRUNK/3585/ ) HBASE-7204 Make hbck ErrorReporter pluggable, ADDENDUM (Revision 1415897) Result = FAILURE jxiang : Files : /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
        Hide
        Jimmy Xiang added a comment -

        Attached an addendum to fix the jdk 1.7 test issue.

        Show
        Jimmy Xiang added a comment - Attached an addendum to fix the jdk 1.7 test issue.
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #282 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/282/)
        HBASE-7204 Make hbck ErrorReporter pluggable (Revision 1415871)

        Result = FAILURE
        jxiang :
        Files :

        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #282 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/282/ ) HBASE-7204 Make hbck ErrorReporter pluggable (Revision 1415871) Result = FAILURE jxiang : Files : /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
        Hide
        Hudson added a comment -

        Integrated in HBase-0.94 #608 (See https://builds.apache.org/job/HBase-0.94/608/)
        HBASE-7204 Make hbck ErrorReporter pluggable (Revision 1415876)

        Result = SUCCESS
        jxiang :
        Files :

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
        • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
        Show
        Hudson added a comment - Integrated in HBase-0.94 #608 (See https://builds.apache.org/job/HBase-0.94/608/ ) HBASE-7204 Make hbck ErrorReporter pluggable (Revision 1415876) Result = SUCCESS jxiang : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #3584 (See https://builds.apache.org/job/HBase-TRUNK/3584/)
        HBASE-7204 Make hbck ErrorReporter pluggable (Revision 1415871)

        Result = FAILURE
        jxiang :
        Files :

        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK #3584 (See https://builds.apache.org/job/HBase-TRUNK/3584/ ) HBASE-7204 Make hbck ErrorReporter pluggable (Revision 1415871) Result = FAILURE jxiang : Files : /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
        Hide
        Jimmy Xiang added a comment -

        Ran tests locally. Integrated into trunk and 0.94.

        Show
        Jimmy Xiang added a comment - Ran tests locally. Integrated into trunk and 0.94.
        Hide
        Jimmy Xiang added a comment -

        Try hadoopqa again.

        Show
        Jimmy Xiang added a comment - Try hadoopqa again.
        Hide
        Jonathan Hsieh added a comment -

        Thanks for making the changes! feel free to commit if the test fix is trivial

        Show
        Jonathan Hsieh added a comment - Thanks for making the changes! feel free to commit if the test fix is trivial
        Hide
        Jimmy Xiang added a comment -

        Thanks for the review. I need to do a minor change to fix a test failure. I will do that before I commit.

        Show
        Jimmy Xiang added a comment - Thanks for the review. I need to do a minor change to fix a test failure. I will do that before I commit.
        Hide
        Jonathan Hsieh added a comment -

        +1 lgtm.

        Show
        Jonathan Hsieh added a comment - +1 lgtm.
        Hide
        Jimmy Xiang added a comment -

        Addressed Jon and Ram's comments. Now hbck runs with ToolRunner, and can accept configurations from command line.

        Show
        Jimmy Xiang added a comment - Addressed Jon and Ram's comments. Now hbck runs with ToolRunner, and can accept configurations from command line.
        Hide
        Jimmy Xiang added a comment -

        Okay. Let me convert it to Tool and post a new patch.

        Show
        Jimmy Xiang added a comment - Okay. Let me convert it to Tool and post a new patch.
        Hide
        Jonathan Hsieh added a comment -

        I'm -0 on the current approach (not going to block but don't like), +1 if tool is used.

        Using Tool is really simple – just rename main to run and replace main with the equivalent of this.

          public static void main(String[] args) throws Exception {
            int ret = ToolRunner.run(new LoadIncrementalHFiles(HBaseConfiguration.create()), args);
            System.exit(ret);
          }
        
        Show
        Jonathan Hsieh added a comment - I'm -0 on the current approach (not going to block but don't like), +1 if tool is used. Using Tool is really simple – just rename main to run and replace main with the equivalent of this. public static void main( String [] args) throws Exception { int ret = ToolRunner.run( new LoadIncrementalHFiles(HBaseConfiguration.create()), args); System .exit(ret); }
        Hide
        Jimmy Xiang added a comment -

        Cool. Can we handle that in a separate jira so that we can have this capability now (overriding this conf from command line for hbck)?

        Show
        Jimmy Xiang added a comment - Cool. Can we handle that in a separate jira so that we can have this capability now (overriding this conf from command line for hbck)?
        Hide
        Jonathan Hsieh added a comment -

        Ok, well that needs to be documented (release notes?). Honestly, I'd still prefer if this was turned into a Tool since the rest of the hbase tools like import/export/copytable/LoadIncrementalHFiles do this. I think this is a hadoop platform convention, not just an MR convention.

        Show
        Jonathan Hsieh added a comment - Ok, well that needs to be documented (release notes?). Honestly, I'd still prefer if this was turned into a Tool since the rest of the hbase tools like import/export/copytable/LoadIncrementalHFiles do this. I think this is a hadoop platform convention, not just an MR convention.
        Hide
        Jimmy Xiang added a comment -

        I tried it yesterday. We can run it like this:

        HBASE_OPTS=-Dkey=value hbase hbck ...

        Show
        Jimmy Xiang added a comment - I tried it yesterday. We can run it like this: HBASE_OPTS=-Dkey=value hbase hbck ...
        Hide
        Jonathan Hsieh added a comment -

        Right – what I'm saying is that I don't think we can specify the system property (as opposed to xml config property) if we use the normal hbase script. e.g. – I don't think this will work as expected.

        hbase hbck -Dsystem.property=foo blah blah

        Show
        Jonathan Hsieh added a comment - Right – what I'm saying is that I don't think we can specify the system property (as opposed to xml config property) if we use the normal hbase script. e.g. – I don't think this will work as expected. hbase hbck -Dsystem.property=foo blah blah
        Hide
        Jimmy Xiang added a comment -

        Tool is more for MR. The patch still uses the regular configuration if it's not specified in system properties.

        Show
        Jimmy Xiang added a comment - Tool is more for MR. The patch still uses the regular configuration if it's not specified in system properties.
        Hide
        Jonathan Hsieh added a comment -

        First I'd double check what I said to make sure it is true. (I'm going off of documentation, haven't actually tried it).

        I wasn't aware of Tool when we overhauled hbck a while back. From a usability point of view, I'd really prefer it if we "Tool"'ed hbck. (look at copy table, or import/export for examples – it is really simple).

        The system properties approach won't work intuitively with the hbase script, and the extra wrapper code will save trouble down the line.

        Show
        Jonathan Hsieh added a comment - First I'd double check what I said to make sure it is true. (I'm going off of documentation, haven't actually tried it). I wasn't aware of Tool when we overhauled hbck a while back. From a usability point of view, I'd really prefer it if we "Tool"'ed hbck. (look at copy table, or import/export for examples – it is really simple). The system properties approach won't work intuitively with the hbase script, and the extra wrapper code will save trouble down the line.
        Hide
        Jimmy Xiang added a comment -

        I see. However, hbck doesn't use the Tool class for now.

        This way (with System property), we don't have to put the configuration in a file. If it is in a configuration file, it can be overridden from command line.

        Show
        Jimmy Xiang added a comment - I see. However, hbck doesn't use the Tool class for now. This way (with System property), we don't have to put the configuration in a file. If it is in a configuration file, it can be overridden from command line.
        Hide
        Jonathan Hsieh added a comment -

        It depends on where you set it. Let's say you write out the entire command line (including java) like this:

        java -Dsystem.property=foo org.apache.hadoop.hbase.util.HBaseFsck -Dconf.property=bar
        

        The -Dsystem.property options come from the jvm invocation (its a jvm thing). The -Dconf.property options come form the Tool class that handles cmd line parsing of argv[]. They are gathered and put into the Configuration. (its a hadoop convention).

        Show
        Jonathan Hsieh added a comment - It depends on where you set it. Let's say you write out the entire command line (including java) like this: java -Dsystem.property=foo org.apache.hadoop.hbase.util.HBaseFsck -Dconf.property=bar The -Dsystem.property options come from the jvm invocation (its a jvm thing). The -Dconf.property options come form the Tool class that handles cmd line parsing of argv[]. They are gathered and put into the Configuration. (its a hadoop convention).
        Hide
        Jimmy Xiang added a comment -

        -Dhbasefsck.errorreporter=x will set it in the system properties, right?

        Show
        Jimmy Xiang added a comment - -Dhbasefsck.errorreporter=x will set it in the system properties, right?
        Hide
        ramkrishna.s.vasudevan added a comment -

        Patch looks good. Same as what Jon said..Thanks

        Show
        ramkrishna.s.vasudevan added a comment - Patch looks good. Same as what Jon said..Thanks
        Hide
        Jonathan Hsieh added a comment -
        +    String className = System.getProperty("hbasefsck.errorreporter");
        

        Why is this System property? Why not a configuration property? (command line overrideable via -Dhbasefsck.errorreporter)?

        otherwise, lgtm.

        Show
        Jonathan Hsieh added a comment - + String className = System .getProperty( "hbasefsck.errorreporter" ); Why is this System property? Why not a configuration property? (command line overrideable via -Dhbasefsck.errorreporter)? otherwise, lgtm.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12554606/trunk-7204.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        -1 javadoc. The javadoc tool appears to have generated 98 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 24 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.client.TestHCM

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/3396//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3396//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3396//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3396//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3396//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3396//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3396//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3396//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3396//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12554606/trunk-7204.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. -1 javadoc . The javadoc tool appears to have generated 98 warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 24 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.client.TestHCM Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/3396//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3396//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3396//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3396//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3396//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3396//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3396//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3396//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3396//console This message is automatically generated.

          People

          • Assignee:
            Jimmy Xiang
            Reporter:
            Jimmy Xiang
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development