HBase
  1. HBase
  2. HBASE-10101

testOfflineRegionReAssginedAfterMasterRestart times out sometimes.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.98.0, 0.96.1, 0.99.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Sometimes, I got this test timed out. The log is attached. It could be because the new cluster takes a while to process the dead server, or assign meta.

      1. trunk-10101.patch
        9 kB
        Jimmy Xiang
      2. trunk-10101_v3.patch
        17 kB
        Jimmy Xiang
      3. trunk-10101_v2.patch
        16 kB
        Jimmy Xiang
      4. test.log
        270 kB
        Jimmy Xiang
      5. hbase-10101-v3-update.patch
        4 kB
        Jeffrey Zhong
      6. hbase-10101-v3.patch
        3 kB
        Jeffrey Zhong
      7. hbase-10101-v2.patch
        1 kB
        Jeffrey Zhong
      8. hbase-10101.patch
        1 kB
        Jeffrey Zhong

        Activity

        Hide
        Jimmy Xiang added a comment -

        Attached a wrong log. Let me get the right one.

        Show
        Jimmy Xiang added a comment - Attached a wrong log. Let me get the right one.
        Hide
        Jimmy Xiang added a comment -

        Here is the right log.

        Show
        Jimmy Xiang added a comment - Here is the right log.
        Hide
        Jeffrey Zhong added a comment -

        The test case did uncover a race condition which should pre-exists. Basically it relies on the old source server to be processed by SSH firstly otherwise you can see that even the region assignment(second one) triggered by SSH is also skipped. SSH region assignment should not be skipped at all. The issue should happen for RITs in failed open state before. Below are related log lines:

        2013-12-06 20:47:45,903 INFO  [AM.-pool62-t1] master.AssignmentManager(1764): Skip assigning testOfflineRegionReAssginedAfterMasterRestart,I,1386391663080.be7906f27d850789818867916aa08c93., it is on a dead but not processed yet server
        ...
        2013-12-06 20:47:45,926 INFO  [localhost.localdomain,59276,1386391665426-GeneralBulkAssigner-2] master.AssignmentManager(1447): Skip assigning testOfflineRegionReAssginedAfterMasterRestart,I,1386391663080.be7906f27d850789818867916aa08c93., it's host localhost.localdomain,47661,1386391655958 is dead but not processed yet
        
        Show
        Jeffrey Zhong added a comment - The test case did uncover a race condition which should pre-exists. Basically it relies on the old source server to be processed by SSH firstly otherwise you can see that even the region assignment(second one) triggered by SSH is also skipped. SSH region assignment should not be skipped at all. The issue should happen for RITs in failed open state before. Below are related log lines: 2013-12-06 20:47:45,903 INFO [AM.-pool62-t1] master.AssignmentManager(1764): Skip assigning testOfflineRegionReAssginedAfterMasterRestart,I,1386391663080.be7906f27d850789818867916aa08c93., it is on a dead but not processed yet server ... 2013-12-06 20:47:45,926 INFO [localhost.localdomain,59276,1386391665426-GeneralBulkAssigner-2] master.AssignmentManager(1447): Skip assigning testOfflineRegionReAssginedAfterMasterRestart,I,1386391663080.be7906f27d850789818867916aa08c93., it's host localhost.localdomain,47661,1386391655958 is dead but not processed yet
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12617549/hbase-10101.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

        +1 hadoop1.1. The patch compiles against the hadoop 1.1 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        -1 site. The patch appears to cause mvn site goal to fail.

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8085//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8085//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8085//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8085//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8085//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8085//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8085//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8085//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8085//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8085//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8085//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12617549/hbase-10101.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop1.1 . The patch compiles against the hadoop 1.1 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8085//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8085//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8085//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8085//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8085//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8085//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8085//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8085//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8085//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8085//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8085//console This message is automatically generated.
        Hide
        Jeffrey Zhong added a comment -

        Jimmy Xiang Could you please take a quick look at the fix which is trivial? Thanks.

        Show
        Jeffrey Zhong added a comment - Jimmy Xiang Could you please take a quick look at the fix which is trivial? Thanks.
        Hide
        Jimmy Xiang added a comment -

        Jeffrey Zhong, good finding. This fix has a little issue. We can clear the last assignment only if it is a different server. I was wondering if we need to clear that when we move the region to offline state in processRegionInTransition. For other state like failed_close/opening/pending_open, the last assignment should have already been cleared. It seems to me HBASE-10085 is a partial fix?

        Show
        Jimmy Xiang added a comment - Jeffrey Zhong , good finding. This fix has a little issue. We can clear the last assignment only if it is a different server. I was wondering if we need to clear that when we move the region to offline state in processRegionInTransition. For other state like failed_close/opening/pending_open, the last assignment should have already been cleared. It seems to me HBASE-10085 is a partial fix?
        Hide
        Jeffrey Zhong added a comment -

        Jimmy Xiang Thanks for the comments. I've updated the patch with your feedbacks. I run the test 20 times in a loop and all passed.

        Show
        Jeffrey Zhong added a comment - Jimmy Xiang Thanks for the comments. I've updated the patch with your feedbacks. I run the test 20 times in a loop and all passed.
        Hide
        Jimmy Xiang added a comment - - edited

        How about we do it this way? We add some function like setLastRegionServerOfRegion(..) to RegionStates and call it in AM#processRegionsInTransition around line 621 to reset the last region server of the region transitioning on a dead server to the dead server.

        Patch v2 is good. It's kind of a safety net to cover all scenarios. That means it may hide some issues we may not know yet, if any. That's why I think it may be better to fix it in AM#processRegionsInTransition which is the place we know we missed something there. What do you think?

        Show
        Jimmy Xiang added a comment - - edited How about we do it this way? We add some function like setLastRegionServerOfRegion(..) to RegionStates and call it in AM#processRegionsInTransition around line 621 to reset the last region server of the region transitioning on a dead server to the dead server. Patch v2 is good. It's kind of a safety net to cover all scenarios. That means it may hide some issues we may not know yet, if any. That's why I think it may be better to fix it in AM#processRegionsInTransition which is the place we know we missed something there. What do you think?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12617662/hbase-10101-v2.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

        +1 hadoop1.1. The patch compiles against the hadoop 1.1 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        -1 site. The patch appears to cause mvn site goal to fail.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8090//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8090//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8090//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8090//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8090//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8090//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8090//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8090//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8090//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8090//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8090//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12617662/hbase-10101-v2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop1.1 . The patch compiles against the hadoop 1.1 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8090//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8090//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8090//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8090//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8090//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8090//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8090//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8090//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8090//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8090//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8090//console This message is automatically generated.
        Hide
        Jeffrey Zhong added a comment -

        Ok, make sense. I moved the changes into the AM#processRegionsInTransition and adjust the test case a little bit to make it more reliable. Thanks for the comments.

        Show
        Jeffrey Zhong added a comment - Ok, make sense. I moved the changes into the AM#processRegionsInTransition and adjust the test case a little bit to make it more reliable. Thanks for the comments.
        Hide
        stack added a comment -

        Hey Jibin, on v3, I don't get this bit:

               ZKAssign.createOrForceNodeOffline(this.watcher, regionInfo, sn);
               regionStates.updateRegionState(regionInfo, State.OFFLINE, sn);
        +      // We need to reset last assignment info for the region because
        +      // 1) we force current RIT state to be offline which means the region isn't assigned anywhere.
        +      // 2) Otherwise they may be skipped by regionStates.wasRegionOnDeadServer in SSH region
        +      // assignment
        +      regionStates.setLastRegionServerOfRegion(sn, regionInfo);
        

        We are setting the servername but didn't we just do it up in createOrForceNodeOffline?

        We are doing the above because region could have come from another RS altogether, one that did not get fully processed? Now we force it to be on the current dead server?

        Show
        stack added a comment - Hey Jibin, on v3, I don't get this bit: ZKAssign.createOrForceNodeOffline( this .watcher, regionInfo, sn); regionStates.updateRegionState(regionInfo, State.OFFLINE, sn); + // We need to reset last assignment info for the region because + // 1) we force current RIT state to be offline which means the region isn't assigned anywhere. + // 2) Otherwise they may be skipped by regionStates.wasRegionOnDeadServer in SSH region + // assignment + regionStates.setLastRegionServerOfRegion(sn, regionInfo); We are setting the servername but didn't we just do it up in createOrForceNodeOffline? We are doing the above because region could have come from another RS altogether, one that did not get fully processed? Now we force it to be on the current dead server?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12617762/hbase-10101-v3.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

        +1 hadoop1.1. The patch compiles against the hadoop 1.1 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        -1 site. The patch appears to cause mvn site goal to fail.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.master.TestRestartCluster
        org.apache.hadoop.hbase.master.TestMasterFailover

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8091//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8091//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8091//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8091//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8091//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8091//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8091//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8091//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8091//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8091//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8091//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12617762/hbase-10101-v3.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop1.1 . The patch compiles against the hadoop 1.1 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.master.TestRestartCluster org.apache.hadoop.hbase.master.TestMasterFailover Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8091//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8091//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8091//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8091//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8091//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8091//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8091//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8091//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8091//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8091//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8091//console This message is automatically generated.
        Hide
        Jimmy Xiang added a comment -

        +1 on v3. For those two failed unit tests, if they are just flaky, I can take a look in HBASE-10090.

        We are setting the servername but didn't we just do it up in createOrForceNodeOffline?

        createOrForceNodeOffline is just set the info in ZK, not the region states map in the mater. Actually, this line is not much use, which will removed in in HBASE-10090.

        We are doing the above because region could have come from another RS altogether, one that did not get fully processed? Now we force it to be on the current dead server?

        We are doing this because the region is transitioning on the new/current dead server now. This happens when the master restarts. In meta, the region could be still assigned to the old dead server (as recorded in the lastAssignment when the master loads the meta after restart). Since ZK shows the region is already transition on the new dead server, we should not worry about log splitting of the old dead server. That's why I agree with Jeffrey Zhong on clear the last assignment info here.

        Show
        Jimmy Xiang added a comment - +1 on v3. For those two failed unit tests, if they are just flaky, I can take a look in HBASE-10090 . We are setting the servername but didn't we just do it up in createOrForceNodeOffline? createOrForceNodeOffline is just set the info in ZK, not the region states map in the mater. Actually, this line is not much use, which will removed in in HBASE-10090 . We are doing the above because region could have come from another RS altogether, one that did not get fully processed? Now we force it to be on the current dead server? We are doing this because the region is transitioning on the new/current dead server now. This happens when the master restarts. In meta, the region could be still assigned to the old dead server (as recorded in the lastAssignment when the master loads the meta after restart). Since ZK shows the region is already transition on the new dead server, we should not worry about log splitting of the old dead server. That's why I agree with Jeffrey Zhong on clear the last assignment info here.
        Hide
        Jimmy Xiang added a comment -

        I see why the tests failed. It's because of the meta region. If we change the fix to (just added a meta region check):

              if (!regionInfo.isMetaRegion()) {
                regionStates.setLastRegionServerOfRegion(sn, regionInfo);
              }
        

        then the test are ok. For the meta region, when master splits the meta log, it uses the meta location in zk, instead of the dead sever we set here. What should we do? Can we tackle it in a separate issue?

        Show
        Jimmy Xiang added a comment - I see why the tests failed. It's because of the meta region. If we change the fix to (just added a meta region check): if (!regionInfo.isMetaRegion()) { regionStates.setLastRegionServerOfRegion(sn, regionInfo); } then the test are ok. For the meta region, when master splits the meta log, it uses the meta location in zk, instead of the dead sever we set here. What should we do? Can we tackle it in a separate issue?
        Hide
        Jimmy Xiang added a comment -

        +1 on commit with the minor change mentioned above. You can fix the trailing spaces as well when you commit. Thanks. We can work on the remaining meta region issue separately.

        Show
        Jimmy Xiang added a comment - +1 on commit with the minor change mentioned above. You can fix the trailing spaces as well when you commit. Thanks. We can work on the remaining meta region issue separately.
        Hide
        stack added a comment -

        Ok. +1. Add a comment on commit on what is going on here (or maybe if the line that sets in zk is of no use, removing it will make it clearer for the slow pokes like myself when they come reading this code)

        Show
        stack added a comment - Ok. +1. Add a comment on commit on what is going on here (or maybe if the line that sets in zk is of no use, removing it will make it clearer for the slow pokes like myself when they come reading this code)
        Hide
        Jeffrey Zhong added a comment -

        Thanks Jimmy Xiang and Stack for the good inputs! I updated the patch a little instead of handling META region differently, I clear LastAssignment info for the region which is in transition to different server.

        Since lastAssignment info is built from META, the lastAssignment clear is less confusion because the lastAssignment of the region is stale so we clear it from out internal memory state. Late region assignment can keep it upto date.

        The other test case failure is due to master can't initialized which I remove master znode & rs znodes after a cluster shutdown in HBaseTestUtility to clear possible stale states preparing for next cluster restart.

        I verified the two failed tests are passed in 5 times in a loop. If the updated version is good, I can check it in tonight. Thanks!

        Show
        Jeffrey Zhong added a comment - Thanks Jimmy Xiang and Stack for the good inputs! I updated the patch a little instead of handling META region differently, I clear LastAssignment info for the region which is in transition to different server. Since lastAssignment info is built from META, the lastAssignment clear is less confusion because the lastAssignment of the region is stale so we clear it from out internal memory state. Late region assignment can keep it upto date. The other test case failure is due to master can't initialized which I remove master znode & rs znodes after a cluster shutdown in HBaseTestUtility to clear possible stale states preparing for next cluster restart. I verified the two failed tests are passed in 5 times in a loop. If the updated version is good, I can check it in tonight. Thanks!
        Hide
        stack added a comment -

        +1 from me. Reads much better now. I can follow what is going on.

        Would wait on Jimmy Xiang +1 before commit. His +1 carries the weight in this area (I'll not cut an RC before this goes in). Thanks Jeffrey Zhong

        Show
        stack added a comment - +1 from me. Reads much better now. I can follow what is going on. Would wait on Jimmy Xiang +1 before commit. His +1 carries the weight in this area (I'll not cut an RC before this goes in). Thanks Jeffrey Zhong
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12617777/hbase-10101-v3-update.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

        +1 hadoop1.1. The patch compiles against the hadoop 1.1 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        -1 site. The patch appears to cause mvn site goal to fail.

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8094//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8094//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8094//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8094//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8094//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8094//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8094//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8094//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8094//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8094//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8094//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12617777/hbase-10101-v3-update.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop1.1 . The patch compiles against the hadoop 1.1 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8094//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8094//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8094//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8094//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8094//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8094//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8094//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8094//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8094//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8094//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8094//console This message is automatically generated.
        Hide
        Jimmy Xiang added a comment -

        I don't think we need to change the test utility. In live cluster, we don't clean up zk before a restart right? Let me post a patch soon, which may make the discussion a little easier?

        Show
        Jimmy Xiang added a comment - I don't think we need to change the test utility. In live cluster, we don't clean up zk before a restart right? Let me post a patch soon, which may make the discussion a little easier?
        Hide
        Jeffrey Zhong added a comment -

        Jimmy Xiang There are some issues in the test utility shutdown and that's why I made change to make sure we have a clean cluster restart:
        1) We close ZookeeperWatcher before we shutdown a cluster so the shut down may leave unclean state because ZK is already closed and shutdown code has no way talk to ZK.
        2) ZK Ephemeral nodes take a little bit time be cleaned depends on how ZooKeeper handles zk session timeout. Therefore, if a cluster restarts so quick, the following cluster start up may see stale ZK nodes which may cause the issue.

        Show
        Jeffrey Zhong added a comment - Jimmy Xiang There are some issues in the test utility shutdown and that's why I made change to make sure we have a clean cluster restart: 1) We close ZookeeperWatcher before we shutdown a cluster so the shut down may leave unclean state because ZK is already closed and shutdown code has no way talk to ZK. 2) ZK Ephemeral nodes take a little bit time be cleaned depends on how ZooKeeper handles zk session timeout. Therefore, if a cluster restarts so quick, the following cluster start up may see stale ZK nodes which may cause the issue.
        Hide
        Jimmy Xiang added a comment -

        Attached a patch that based on patch v3. Jeffrey Zhong, stack, could you please take a look?

        Show
        Jimmy Xiang added a comment - Attached a patch that based on patch v3. Jeffrey Zhong , stack , could you please take a look?
        Hide
        Jimmy Xiang added a comment - - edited

        I see, about the test utility shutdown change. But if we do clean restart all the time, TestRestartCluster may be of not much use any more? I was wondering should we move testOfflineRegionReAssginedAfterMasterRestart to TestMasterFailover. How about that?

        Show
        Jimmy Xiang added a comment - - edited I see, about the test utility shutdown change. But if we do clean restart all the time, TestRestartCluster may be of not much use any more? I was wondering should we move testOfflineRegionReAssginedAfterMasterRestart to TestMasterFailover. How about that?
        Hide
        Jimmy Xiang added a comment -

        It looks like that TestMasterFailover already covers this case. Why doesn't it catch it (HBASE-10085)?

        Show
        Jimmy Xiang added a comment - It looks like that TestMasterFailover already covers this case. Why doesn't it catch it ( HBASE-10085 )?
        Hide
        Jimmy Xiang added a comment -

        Attached trunk-10101_v2.patch that moved testOfflineRegionReAssginedAfterMasterRestart to TestMasterFailOver, increased the timeout time while using less region servers.

        Show
        Jimmy Xiang added a comment - Attached trunk-10101_v2.patch that moved testOfflineRegionReAssginedAfterMasterRestart to TestMasterFailOver, increased the timeout time while using less region servers.
        Hide
        Jeffrey Zhong added a comment - - edited

        The ZK clean is only clear the master address node and RS nodes which should be removed when a cluster is shut down. The added steps make sure we have a clean restart for normal unit tests and there are special test cases for master(cluster) restart scenarios.

        I prefer the test case in TestAssignmentManagerOnCluster because it's about region aren't be assigned during a cluster restart.

        Below are my comments on the trunk patch:

        +      regionStates.setLastRegionServerOfRegion(sn, encodedName);
        +      if (regionInfo.isMetaRegion()) {
        +        // If it's meta region, reset the meta location.
        +        // So that master knows the right meta region server.
        +        MetaRegionTracker.setMetaLocation(watcher, sn);
        +      }
        

        The above is a little dramatic because we just set internal Memory state to some server. This'll cause confusion for the future readers.

        -          if (expireIfOnline(currentMetaServer)) {
        +          if (!serverManager.isServerDead(currentMetaServer)) {
        

        This isn't ideal because we could have a race condition that a dead meta server may not report(SessionException) in time. We could skip meta re-assign and cause master can't be started.

        Jimmy Xiang For your latest patch, it looks good to me except the changes in HMaster.java. I'd prefer my v3-update patch unless you have a strong feeling about your trunk patch.

        I'll let you decide which to choose and move on. Thanks.

        Show
        Jeffrey Zhong added a comment - - edited The ZK clean is only clear the master address node and RS nodes which should be removed when a cluster is shut down. The added steps make sure we have a clean restart for normal unit tests and there are special test cases for master(cluster) restart scenarios. I prefer the test case in TestAssignmentManagerOnCluster because it's about region aren't be assigned during a cluster restart. Below are my comments on the trunk patch: + regionStates.setLastRegionServerOfRegion(sn, encodedName); + if (regionInfo.isMetaRegion()) { + // If it's meta region, reset the meta location. + // So that master knows the right meta region server. + MetaRegionTracker.setMetaLocation(watcher, sn); + } The above is a little dramatic because we just set internal Memory state to some server. This'll cause confusion for the future readers. - if (expireIfOnline(currentMetaServer)) { + if (!serverManager.isServerDead(currentMetaServer)) { This isn't ideal because we could have a race condition that a dead meta server may not report(SessionException) in time. We could skip meta re-assign and cause master can't be started. Jimmy Xiang For your latest patch, it looks good to me except the changes in HMaster.java. I'd prefer my v3-update patch unless you have a strong feeling about your trunk patch. I'll let you decide which to choose and move on. Thanks.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12617873/trunk-10101.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 9 new or modified tests.

        +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

        +1 hadoop1.1. The patch compiles against the hadoop 1.1 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        -1 site. The patch appears to cause mvn site goal to fail.

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8100//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8100//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8100//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8100//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8100//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8100//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8100//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8100//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8100//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8100//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8100//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12617873/trunk-10101.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 9 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop1.1 . The patch compiles against the hadoop 1.1 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8100//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8100//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8100//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8100//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8100//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8100//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8100//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8100//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8100//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8100//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8100//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12617876/trunk-10101_v2.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 12 new or modified tests.

        +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

        +1 hadoop1.1. The patch compiles against the hadoop 1.1 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        -1 site. The patch appears to cause mvn site goal to fail.

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8101//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8101//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8101//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8101//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8101//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8101//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8101//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8101//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8101//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8101//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8101//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12617876/trunk-10101_v2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 12 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop1.1 . The patch compiles against the hadoop 1.1 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8101//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8101//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8101//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8101//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8101//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8101//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8101//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8101//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8101//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8101//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8101//console This message is automatically generated.
        Hide
        Jimmy Xiang added a comment -

        Attached patch trunk-v3 that makes sure server not online in AM#processRegionsInTransition is known to be dead since master already considers it's dead. Otherwise, region could be double assigned.

        Show
        Jimmy Xiang added a comment - Attached patch trunk-v3 that makes sure server not online in AM#processRegionsInTransition is known to be dead since master already considers it's dead. Otherwise, region could be double assigned.
        Hide
        Jimmy Xiang added a comment -

        Jeffrey Zhong, for patch v3-update, you clear the last assignment. However, the region could have been open on the dead server for a while, so we need to wait till the current dead server is done with log splitting to avoid possible data loss. For reset the meta location, the purpose is the same, to make sure the right meta log is split. As to the change in the master, originally, it expires the sever if it is online; the change is to expire the server as long as it is not known to be dead, i.e., it is online or unknown but not queued for SSH. For unknown, as you said it could have not reported to the master. But in this case, it is not going to report to master any more, since verifymetalocation is already failed.

        The test can be in TestAssignmentManagerOnCluster if the tests don't share the same cluster. Since, the cluster is shared, if we restarts the cluster in the middle and the test fails, other tests are impacted as well. TestMasterFailove seems to be a good place.

        Show
        Jimmy Xiang added a comment - Jeffrey Zhong , for patch v3-update, you clear the last assignment. However, the region could have been open on the dead server for a while, so we need to wait till the current dead server is done with log splitting to avoid possible data loss. For reset the meta location, the purpose is the same, to make sure the right meta log is split. As to the change in the master, originally, it expires the sever if it is online; the change is to expire the server as long as it is not known to be dead, i.e., it is online or unknown but not queued for SSH. For unknown, as you said it could have not reported to the master. But in this case, it is not going to report to master any more, since verifymetalocation is already failed. The test can be in TestAssignmentManagerOnCluster if the tests don't share the same cluster. Since, the cluster is shared, if we restarts the cluster in the middle and the test fails, other tests are impacted as well. TestMasterFailove seems to be a good place.
        Hide
        Jeffrey Zhong added a comment -

        However, the region could have been open on the dead server for a while, so we need to wait till the current dead server is done with log splitting to avoid possible data loss.

        I thought this already last night. This can't be because the RIT state is in transition not (opened) so the region can't get requests. During region assignment, the same check(in am.forceRegionStateToOffline for offline case) happens to prevent us from assigning a region prematurely(before logsplitting work). Setting the last region location is logically same as removing the last assignment because last assignment is only used to skip region assign inside SSH.

        expire the server as long as it is not known to be dead

        I c. It's a good change but unrelated to this issue, right?

        In overall, I'm good(+1) on your trunk v3 patch while I still prefer to my hbase-10101-v3-update.patch because it has less changes and readable.

        Jimmy Xiang If you check in your trunk v3 patch, please assign the JIRA to yourself. Thanks.

        Show
        Jeffrey Zhong added a comment - However, the region could have been open on the dead server for a while, so we need to wait till the current dead server is done with log splitting to avoid possible data loss. I thought this already last night. This can't be because the RIT state is in transition not (opened) so the region can't get requests. During region assignment, the same check(in am.forceRegionStateToOffline for offline case) happens to prevent us from assigning a region prematurely(before logsplitting work). Setting the last region location is logically same as removing the last assignment because last assignment is only used to skip region assign inside SSH. expire the server as long as it is not known to be dead I c. It's a good change but unrelated to this issue, right? In overall, I'm good(+1) on your trunk v3 patch while I still prefer to my hbase-10101-v3-update.patch because it has less changes and readable. Jimmy Xiang If you check in your trunk v3 patch, please assign the JIRA to yourself. Thanks.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12617911/trunk-10101_v3.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 12 new or modified tests.

        +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

        +1 hadoop1.1. The patch compiles against the hadoop 1.1 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        -1 site. The patch appears to cause mvn site goal to fail.

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8105//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8105//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8105//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8105//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8105//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8105//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8105//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8105//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8105//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8105//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8105//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12617911/trunk-10101_v3.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 12 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop1.1 . The patch compiles against the hadoop 1.1 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8105//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8105//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8105//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8105//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8105//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8105//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8105//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8105//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8105//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8105//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8105//console This message is automatically generated.
        Hide
        Jimmy Xiang added a comment -

        Integrated v3 into 0.96, 0.98, and trunk. Thanks Jeffrey Zhong a lot for finding out the root cause, initial patches, and review. Probably your v3-update works too. I haven't thought your approach through since 0.96.1 rc is going to be cut soon, so I'd like to go with my approach. Thanks again.

        Show
        Jimmy Xiang added a comment - Integrated v3 into 0.96, 0.98, and trunk. Thanks Jeffrey Zhong a lot for finding out the root cause, initial patches, and review. Probably your v3-update works too. I haven't thought your approach through since 0.96.1 rc is going to be cut soon, so I'd like to go with my approach. Thanks again.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12617911/trunk-10101_v3.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 12 new or modified tests.

        +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

        +1 hadoop1.1. The patch compiles against the hadoop 1.1 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        -1 site. The patch appears to cause mvn site goal to fail.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.regionserver.TestSplitLogWorker

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8106//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8106//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8106//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8106//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8106//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8106//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8106//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8106//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8106//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8106//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8106//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12617911/trunk-10101_v3.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 12 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop1.1 . The patch compiles against the hadoop 1.1 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.TestSplitLogWorker Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8106//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8106//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8106//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8106//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8106//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8106//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8106//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8106//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8106//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8106//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8106//console This message is automatically generated.
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in hbase-0.96 #220 (See https://builds.apache.org/job/hbase-0.96/220/)
        HBASE-10101 testOfflineRegionReAssginedAfterMasterRestart times out sometimes (jxiang: rev 1549709)

        • /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
        • /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
        • /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java
        • /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java
        • /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManagerOnCluster.java
        • /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java
        • /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
        Show
        Hudson added a comment - SUCCESS: Integrated in hbase-0.96 #220 (See https://builds.apache.org/job/hbase-0.96/220/ ) HBASE-10101 testOfflineRegionReAssginedAfterMasterRestart times out sometimes (jxiang: rev 1549709) /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManagerOnCluster.java /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-TRUNK #4718 (See https://builds.apache.org/job/HBase-TRUNK/4718/)
        HBASE-10101 testOfflineRegionReAssginedAfterMasterRestart times out sometimes (jxiang: rev 1549708)

        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManagerOnCluster.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-TRUNK #4718 (See https://builds.apache.org/job/HBase-TRUNK/4718/ ) HBASE-10101 testOfflineRegionReAssginedAfterMasterRestart times out sometimes (jxiang: rev 1549708) /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManagerOnCluster.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-0.98 #6 (See https://builds.apache.org/job/HBase-0.98/6/)
        HBASE-10101 testOfflineRegionReAssginedAfterMasterRestart times out sometimes (jxiang: rev 1549712)

        • /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
        • /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
        • /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java
        • /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java
        • /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManagerOnCluster.java
        • /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java
        • /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-0.98 #6 (See https://builds.apache.org/job/HBase-0.98/6/ ) HBASE-10101 testOfflineRegionReAssginedAfterMasterRestart times out sometimes (jxiang: rev 1549712) /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManagerOnCluster.java /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #5 (See https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/5/)
        HBASE-10101 testOfflineRegionReAssginedAfterMasterRestart times out sometimes (jxiang: rev 1549712)

        • /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
        • /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
        • /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java
        • /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java
        • /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManagerOnCluster.java
        • /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java
        • /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #5 (See https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/5/ ) HBASE-10101 testOfflineRegionReAssginedAfterMasterRestart times out sometimes (jxiang: rev 1549712) /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManagerOnCluster.java /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in hbase-0.96-hadoop2 #146 (See https://builds.apache.org/job/hbase-0.96-hadoop2/146/)
        HBASE-10101 testOfflineRegionReAssginedAfterMasterRestart times out sometimes (jxiang: rev 1549709)

        • /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
        • /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
        • /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java
        • /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java
        • /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManagerOnCluster.java
        • /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java
        • /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
        Show
        Hudson added a comment - SUCCESS: Integrated in hbase-0.96-hadoop2 #146 (See https://builds.apache.org/job/hbase-0.96-hadoop2/146/ ) HBASE-10101 testOfflineRegionReAssginedAfterMasterRestart times out sometimes (jxiang: rev 1549709) /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManagerOnCluster.java /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in HBase-TRUNK-on-Hadoop-1.1 #3 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-1.1/3/)
        HBASE-10101 testOfflineRegionReAssginedAfterMasterRestart times out sometimes (jxiang: rev 1549708)

        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManagerOnCluster.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
        Show
        Hudson added a comment - FAILURE: Integrated in HBase-TRUNK-on-Hadoop-1.1 #3 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-1.1/3/ ) HBASE-10101 testOfflineRegionReAssginedAfterMasterRestart times out sometimes (jxiang: rev 1549708) /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManagerOnCluster.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
        Hide
        stack added a comment -

        Released in 0.96.1. Issue closed.

        Show
        stack added a comment - Released in 0.96.1. Issue closed.

          People

          • Assignee:
            Jimmy Xiang
            Reporter:
            Jimmy Xiang
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development