HBase
  1. HBase
  2. HBASE-5892

[hbck] Refactor parallel WorkItem* to Futures.

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.90.6, 0.92.1, 0.94.0
    • Fix Version/s: 0.92.2, 0.94.1
    • Component/s: hbck
    • Labels:
    • Hadoop Flags:
      Reviewed

      Description

      This would convert WorkItem* logic (with low level notifies, and rough exception handling) into a more canonical Futures pattern.

      Currently there are two instances of this pattern (for loading hdfs dirs, for contacting regionservers for assignments, and soon – for loading hdfs .regioninfo files).

      1. hbase-5892.patch
        12 kB
        Andrew Wang
      2. hbase-5892-1.patch
        12 kB
        Andrew Wang
      3. hbase-5892-2.patch
        12 kB
        Andrew Wang
      4. hbase-5892-2-0.90.patch
        12 kB
        Andrew Wang
      5. hbase-5892-3.patch
        12 kB
        Andrew Wang
      6. hbase-5892-4.patch
        12 kB
        Andrew Wang
      7. hbase-5892-4-0.90.patch
        12 kB
        Andrew Wang

        Activity

        Hide
        Hudson added a comment -

        Integrated in HBase-0.94-security #34 (See https://builds.apache.org/job/HBase-0.94-security/34/)
        HBASE-5892 [hbck] Refactor parallel WorkItem* to Futures (Andrew Wang) (Revision 1345889)

        Result = SUCCESS
        jmhsieh :
        Files :

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
        Show
        Hudson added a comment - Integrated in HBase-0.94-security #34 (See https://builds.apache.org/job/HBase-0.94-security/34/ ) HBASE-5892 [hbck] Refactor parallel WorkItem* to Futures (Andrew Wang) (Revision 1345889) Result = SUCCESS jmhsieh : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
        Hide
        Hudson added a comment -

        Integrated in HBase-0.92-security #110 (See https://builds.apache.org/job/HBase-0.92-security/110/)
        HBASE-5892 [hbck] Refactor parallel WorkItem* to Futures (Andrew Wang) (Revision 1345888)

        Result = FAILURE
        jmhsieh :
        Files :

        • /hbase/branches/0.92/CHANGES.txt
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
        Show
        Hudson added a comment - Integrated in HBase-0.92-security #110 (See https://builds.apache.org/job/HBase-0.92-security/110/ ) HBASE-5892 [hbck] Refactor parallel WorkItem* to Futures (Andrew Wang) (Revision 1345888) Result = FAILURE jmhsieh : Files : /hbase/branches/0.92/CHANGES.txt /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
        Hide
        Hudson added a comment -

        Integrated in HBase-0.92 #443 (See https://builds.apache.org/job/HBase-0.92/443/)
        HBASE-5892 [hbck] Refactor parallel WorkItem* to Futures (Andrew Wang) (Revision 1345888)

        Result = FAILURE
        jmhsieh :
        Files :

        • /hbase/branches/0.92/CHANGES.txt
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
        Show
        Hudson added a comment - Integrated in HBase-0.92 #443 (See https://builds.apache.org/job/HBase-0.92/443/ ) HBASE-5892 [hbck] Refactor parallel WorkItem* to Futures (Andrew Wang) (Revision 1345888) Result = FAILURE jmhsieh : Files : /hbase/branches/0.92/CHANGES.txt /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #39 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/39/)
        HBASE-5892 [hbck] Refactor parallel WorkItem* to Futures (Andrew Wang) (Revision 1345890)

        Result = FAILURE
        jmhsieh :
        Files :

        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #39 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/39/ ) HBASE-5892 [hbck] Refactor parallel WorkItem* to Futures (Andrew Wang) (Revision 1345890) Result = FAILURE jmhsieh : Files : /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
        Hide
        Hudson added a comment -

        Integrated in HBase-0.94 #244 (See https://builds.apache.org/job/HBase-0.94/244/)
        HBASE-5892 [hbck] Refactor parallel WorkItem* to Futures (Andrew Wang) (Revision 1345889)

        Result = SUCCESS
        jmhsieh :
        Files :

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
        Show
        Hudson added a comment - Integrated in HBase-0.94 #244 (See https://builds.apache.org/job/HBase-0.94/244/ ) HBASE-5892 [hbck] Refactor parallel WorkItem* to Futures (Andrew Wang) (Revision 1345889) Result = SUCCESS jmhsieh : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #2982 (See https://builds.apache.org/job/HBase-TRUNK/2982/)
        HBASE-5892 [hbck] Refactor parallel WorkItem* to Futures (Andrew Wang) (Revision 1345890)

        Result = FAILURE
        jmhsieh :
        Files :

        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK #2982 (See https://builds.apache.org/job/HBase-TRUNK/2982/ ) HBASE-5892 [hbck] Refactor parallel WorkItem* to Futures (Andrew Wang) (Revision 1345890) Result = FAILURE jmhsieh : Files : /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
        Hide
        Jonathan Hsieh added a comment -

        Committed to 0.90/0.92/0.94/trunk branches. Thanks Andrew!

        Show
        Jonathan Hsieh added a comment - Committed to 0.90/0.92/0.94/trunk branches. Thanks Andrew!
        Hide
        Jonathan Hsieh added a comment -

        Looks good to me. I was able to apply the v4 version to trunk and to 0.92/0.94 via 'patch -p1 < hbase-5892-4.patch' but at the moment am having problems with my account. Will commit when I get fixed.

        Show
        Jonathan Hsieh added a comment - Looks good to me. I was able to apply the v4 version to trunk and to 0.92/0.94 via 'patch -p1 < hbase-5892-4.patch' but at the moment am having problems with my account. Will commit when I get fixed.
        Hide
        Anoop Sam John added a comment -

        @Andrew
        The issue with findbugs is solved as part of HBASE-6138

        Show
        Anoop Sam John added a comment - @Andrew The issue with findbugs is solved as part of HBASE-6138
        Hide
        Andrew Wang added a comment -

        I don't know why Findbugs is erroring. Maybe the modularization change?

        [ERROR] Could not find resource '${parent.basedir}/dev-support/findbugs-exclude.xml'. -> [Help 1]
        

        No tests because there's no functionality change, it's a refactor.

        Show
        Andrew Wang added a comment - I don't know why Findbugs is erroring. Maybe the modularization change? [ERROR] Could not find resource '${parent.basedir}/dev-support/findbugs-exclude.xml'. -> [Help 1] No tests because there's no functionality change, it's a refactor.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12530256/hbase-5892-4.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2059//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2059//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12530256/hbase-5892-4.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2059//testReport/ Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2059//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12530243/hbase-5892-3.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.coprocessor.TestRowProcessorEndpoint
        org.apache.hadoop.hbase.master.TestAssignmentManager
        org.apache.hadoop.hbase.replication.TestReplication

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2058//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2058//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12530243/hbase-5892-3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.coprocessor.TestRowProcessorEndpoint org.apache.hadoop.hbase.master.TestAssignmentManager org.apache.hadoop.hbase.replication.TestReplication Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2058//testReport/ Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2058//console This message is automatically generated.
        Hide
        Andrew Wang added a comment -

        Okay, respun both versions to fix the code style comments.

        Show
        Andrew Wang added a comment - Okay, respun both versions to fix the code style comments.
        Hide
        Ted Yu added a comment -

        Minor comment:

        +    List<WorkItemRegion> work = new ArrayList<WorkItemRegion>(regionServerList.size());
        

        variable work should be named works.

        +        if(hdfsEntry.hri != null) {
        

        Insert space between if and (.

        Show
        Ted Yu added a comment - Minor comment: + List<WorkItemRegion> work = new ArrayList<WorkItemRegion>(regionServerList.size()); variable work should be named works. + if (hdfsEntry.hri != null ) { Insert space between if and (.
        Hide
        Andrew Wang added a comment -

        Re-diffed to get the right prefix. Thanks Zhihong.

        Show
        Andrew Wang added a comment - Re-diffed to get the right prefix. Thanks Zhihong.
        Hide
        Ted Yu added a comment -

        @Andrew:
        The path for source file should have hbase-server/ prefix. This is due to modularization change.

        Show
        Ted Yu added a comment - @Andrew: The path for source file should have hbase-server/ prefix. This is due to modularization change.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12530128/hbase-5892-2.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2042//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12530128/hbase-5892-2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2042//console This message is automatically generated.
        Hide
        Ted Yu added a comment -

        The practice is to attach trunk patch separately after attaching the other patches.

        Show
        Ted Yu added a comment - The practice is to attach trunk patch separately after attaching the other patches.
        Hide
        Andrew Wang added a comment -

        Patch failed to apply because Jenkins tried to apply the version I made for 0.90 to trunk. I don't know how to kick the build bot to make it do the right thing for different versions...

        Show
        Andrew Wang added a comment - Patch failed to apply because Jenkins tried to apply the version I made for 0.90 to trunk. I don't know how to kick the build bot to make it do the right thing for different versions...
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12529803/hbase-5892-2-0.90.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2003//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12529803/hbase-5892-2-0.90.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2003//console This message is automatically generated.
        Hide
        Andrew Wang added a comment -

        Another rev, tried to fix the one relevant FindBugs error I saw. The patch for trunk applied cleanly to 0.94 and 0.92; provided is the slightly modified patch for 0.90.

        Show
        Andrew Wang added a comment - Another rev, tried to fix the one relevant FindBugs error I saw. The patch for trunk applied cleanly to 0.94 and 0.92; provided is the slightly modified patch for 0.90.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12529780/hbase-5892-1.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 hadoop23. The patch compiles against the hadoop 0.23.x profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 33 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2001//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2001//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2001//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12529780/hbase-5892-1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop23. The patch compiles against the hadoop 0.23.x profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 33 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2001//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2001//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2001//console This message is automatically generated.
        Hide
        Andrew Wang added a comment -

        Ran TestHBaseFsck, had to fix a null pointer thus new version of the patch. I'll port it to prior versions too.

        Show
        Andrew Wang added a comment - Ran TestHBaseFsck, had to fix a null pointer thus new version of the patch. I'll port it to prior versions too.
        Hide
        Jonathan Hsieh added a comment - - edited

        Andrew, looks good. I'm going to wait for the hadoopqa robot to execute the test suite. Alternately, since this just modifies hbck, can you try this command and share results: 'mvn test -PlocalTests -Dtest=TestHBaseFsck'?

        I'd like to keep all hbck across versions essentially the same – would you be willing to port to 0.90/0.92/0.94? I'd bet that this may apply to 0.94 and 0.92, and that 0.90 would require some near trivial tweaks.

        Show
        Jonathan Hsieh added a comment - - edited Andrew, looks good. I'm going to wait for the hadoopqa robot to execute the test suite. Alternately, since this just modifies hbck, can you try this command and share results: 'mvn test -PlocalTests -Dtest=TestHBaseFsck'? I'd like to keep all hbck across versions essentially the same – would you be willing to port to 0.90/0.92/0.94? I'd bet that this may apply to 0.94 and 0.92, and that 0.90 would require some near trivial tweaks.
        Hide
        Andrew Wang added a comment -

        I tried to do this refactor, essentially switching out Runnable for Callable and adding some more logging in the process. Let me know if it's not what you were thinking of.

        I didn't do any testing beyond running hbck on my local machine, which seemed to work.

        Show
        Andrew Wang added a comment - I tried to do this refactor, essentially switching out Runnable for Callable and adding some more logging in the process. Let me know if it's not what you were thinking of. I didn't do any testing beyond running hbck on my local machine, which seemed to work.

          People

          • Assignee:
            Andrew Wang
            Reporter:
            Jonathan Hsieh
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development