HBase
  1. HBase
  2. HBASE-6143

Make region assignment smarter when regions are re-enabled.

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Critical Critical
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Balancer
    • Labels:
      None

      Description

      Right now a random region server is picked when re-enabling a table. This could be much smarter.

      1. 6143-v1.txt
        0.7 kB
        Ted Yu
      2. 6143-v2.txt
        3 kB
        Ted Yu
      3. 6143-v3.txt
        6 kB
        Ted Yu
      4. HBASE-6143-0.patch
        46 kB
        Elliott Clark
      5. HBASE-6143-1.patch
        48 kB
        Elliott Clark
      6. HBASE-6143-v1.patch
        89 kB
        Elliott Clark
      7. HBASE-6143-v2.patch
        127 kB
        Elliott Clark

        Issue Links

          Activity

          Hide
          Nicolas Spiegelberg added a comment -

          The default behavior should most likely take load balancing into account (See HBASE-4191). In FB, our master has a concept of 'favored nodes' that are assigned upon Region creation and map directly to the HDFS replication pipeline.

          Show
          Nicolas Spiegelberg added a comment - The default behavior should most likely take load balancing into account (See HBASE-4191 ). In FB, our master has a concept of 'favored nodes' that are assigned upon Region creation and map directly to the HDFS replication pipeline.
          Hide
          Elliott Clark added a comment -

          I agree. Though sometimes choosing a balance of the regions can be an expensive operation so the balancer should provide an api for a cheap balance.

          Show
          Elliott Clark added a comment - I agree. Though sometimes choosing a balance of the regions can be an expensive operation so the balancer should provide an api for a cheap balance.
          Show
          Vladimir Rodionov added a comment - This is Ted's quick analysis on this issue: http://mail-archives.apache.org/mod_mbox/hbase-dev/201307.mbox/%3CCALte62yCN7RKcgHxr6nNy6A7UyC8g-uztt7epu%3D9OevmbSQE1A%40mail.gmail.com%3E The full thread starts here: http://mail-archives.apache.org/mod_mbox/hbase-dev/201307.mbox/%3CDC5EBE7F3610EB4CA5C7E92D76873E1517ECF1DCA9@exchange2007.carrieriq.com%3E
          Hide
          Elliott Clark added a comment -

          I pretty vehemently disagree that we need to add more apis on the balancer. What we really need to do is simplify and make sure that everything uses the same code path (that code path will include locality and placement).

          Show
          Elliott Clark added a comment - I pretty vehemently disagree that we need to add more apis on the balancer. What we really need to do is simplify and make sure that everything uses the same code path (that code path will include locality and placement).
          Hide
          Lars Hofhansl added a comment -

          I agree. We should just default to attempting to keep the previous region assignment. No need to make that optional via a new API.

          Show
          Lars Hofhansl added a comment - I agree. We should just default to attempting to keep the previous region assignment. No need to make that optional via a new API.
          Hide
          Ted Yu added a comment -

          How about this patch ?

          Show
          Ted Yu added a comment - How about this patch ?
          Hide
          Lars Hofhansl added a comment -

          Both I and Vladimir said that much on the email-thread.
          Ted's analysis is only 1/2 the story, though. The retainAssignment flag is tied to skipTableStateCheck right now, and only enabled during recovery.

          It is a shame that we go through some length to keep assignments during a full cluster bounce, but drop the ball for a simple disable/enable cycle.

          Show
          Lars Hofhansl added a comment - Both I and Vladimir said that much on the email-thread. Ted's analysis is only 1/2 the story, though. The retainAssignment flag is tied to skipTableStateCheck right now, and only enabled during recovery. It is a shame that we go through some length to keep assignments during a full cluster bounce, but drop the ball for a simple disable/enable cycle.
          Hide
          Lars Hofhansl added a comment -

          I do not think it is going to be that simple, Ted.
          Simply changing that flag has other implication, too. For example we'd no longer check for concurrent enable request neither would we now check for non-existent tables (among others).

          Show
          Lars Hofhansl added a comment - I do not think it is going to be that simple, Ted. Simply changing that flag has other implication, too. For example we'd no longer check for concurrent enable request neither would we now check for non-existent tables (among others).
          Hide
          Ted Yu added a comment -

          How about patch v2 ?

          Show
          Ted Yu added a comment - How about patch v2 ?
          Hide
          Ted Yu added a comment -

          Note on patch v2: EnableTableHandler.retainAssignment was improperly named. I renamed it this.skipTableStateCheck to be consistent with the name of parameter passed to ctor.

          Show
          Ted Yu added a comment - Note on patch v2: EnableTableHandler.retainAssignment was improperly named. I renamed it this.skipTableStateCheck to be consistent with the name of parameter passed to ctor.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12593947/6143-v2.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.TestIOFencing

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6450//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6450//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6450//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6450//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6450//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6450//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6450//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6450//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6450//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6450//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12593947/6143-v2.txt against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.TestIOFencing Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6450//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6450//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6450//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6450//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6450//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6450//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6450//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6450//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6450//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6450//console This message is automatically generated.
          Hide
          Lars Hofhansl added a comment -

          v2 could work. The code path for assignment after enabling is now quite different, though.

          Might still be better to call AssignmentManager.assign(Map<HRegionInfo, ServerName> regions) BulkEnabler.populatePool, I think we can build the map ahead of time.

          This is the same call used by AssignmentManager.joinCluster(), so we'd use the same code.

          Show
          Lars Hofhansl added a comment - v2 could work. The code path for assignment after enabling is now quite different, though. Might still be better to call AssignmentManager.assign(Map<HRegionInfo, ServerName> regions) BulkEnabler.populatePool , I think we can build the map ahead of time. This is the same call used by AssignmentManager.joinCluster(), so we'd use the same code.
          Hide
          stack added a comment -

          Unit test? Is bulk assigner safe? (Jimmy Xiang?) In the past, bulk assigner was known to not be safe and if we failed, because we were using it on startup only, then we'd just exit so user would retry. Jimmy may have fixed this.

          Show
          stack added a comment - Unit test? Is bulk assigner safe? ( Jimmy Xiang ?) In the past, bulk assigner was known to not be safe and if we failed, because we were using it on startup only, then we'd just exit so user would retry. Jimmy may have fixed this.
          Hide
          Lars Hofhansl added a comment -

          If we use AssignmentManager.assign(Map<HRegionInfo, ServerName> regions) we're using the same code as used in startup, hopefully that is safe. We absolutely would need a test for this.

          Show
          Lars Hofhansl added a comment - If we use AssignmentManager.assign(Map<HRegionInfo, ServerName> regions) we're using the same code as used in startup, hopefully that is safe. We absolutely would need a test for this.
          Hide
          Ted Yu added a comment -

          Turns out that there was a test in TestAdmin involving enabling table.

          With patch v2, the modified test passes.

          Without patch v2, I got:

          Failed tests:   testEnableTableRetainAssignment(org.apache.hadoop.hbase.client.TestAdmin): expected:<192.168.0.13,50614,1374729419850> but was:<192.168.0.13,50613,1374729419822>
          
          Show
          Ted Yu added a comment - Turns out that there was a test in TestAdmin involving enabling table. With patch v2, the modified test passes. Without patch v2, I got: Failed tests: testEnableTableRetainAssignment(org.apache.hadoop.hbase.client.TestAdmin): expected:<192.168.0.13,50614,1374729419850> but was:<192.168.0.13,50613,1374729419822>
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12594104/6143-v3.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6460//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6460//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6460//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6460//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6460//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6460//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6460//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6460//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6460//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6460//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12594104/6143-v3.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6460//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6460//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6460//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6460//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6460//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6460//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6460//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6460//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6460//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6460//console This message is automatically generated.
          Hide
          Jimmy Xiang added a comment -

          Stack, in trunk and 0.95, bulk assigner is safe.

          Show
          Jimmy Xiang added a comment - Stack , in trunk and 0.95, bulk assigner is safe.
          Hide
          Elliott Clark added a comment -

          I was thinking something more along the lines of this. It moves everything onto fewer code paths. And it also allows for bulk assigns.

          Show
          Elliott Clark added a comment - I was thinking something more along the lines of this. It moves everything onto fewer code paths. And it also allows for bulk assigns.
          Hide
          Elliott Clark added a comment -

          It's really raw and I doubt it even runs yet. But I think the simplification of the code is worth it.

          Show
          Elliott Clark added a comment - It's really raw and I doubt it even runs yet. But I think the simplification of the code is worth it.
          Hide
          stack added a comment -

          Did a skim. I like all the removes and the emergence of a simple pattern all follow when they want to do region moving. Let me know when you want a closer review.

          Show
          stack added a comment - Did a skim. I like all the removes and the emergence of a simple pattern all follow when they want to do region moving. Let me know when you want a closer review.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12594295/HBASE-6143-0.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 20 new or modified tests.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 lineLengths. The patch introduces lines longer than 100

          -1 site. The patch appears to cause mvn site goal to fail.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.coprocessor.TestMasterObserver
          org.apache.hadoop.hbase.regionserver.TestRegionMergeTransactionOnCluster
          org.apache.hadoop.hbase.security.access.TestAccessController
          org.apache.hadoop.hbase.master.TestRegionPlacement
          org.apache.hadoop.hbase.TestRegionRebalancing
          org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster
          org.apache.hadoop.hbase.regionserver.TestHRegionOnCluster
          org.apache.hadoop.hbase.snapshot.TestFlushSnapshotFromClient
          org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster
          org.apache.hadoop.hbase.master.TestRollingRestart

          -1 core zombie tests. There are 10 zombie test(s): at org.apache.hadoop.hbase.client.TestFromClientSide.testScanMetrics(TestFromClientSide.java:4740)
          at org.apache.hadoop.hbase.util.TestHBaseFsck.testValidLingeringSplitParent(TestHBaseFsck.java:1283)
          at org.apache.hadoop.hbase.client.TestScannersFromClientSide.testScanBatch(TestScannersFromClientSide.java:131)
          at org.apache.hadoop.hbase.client.TestHCM.testRegionCaching(TestHCM.java:365)
          at org.apache.hadoop.hbase.regionserver.wal.TestWALReplay.testReplayEditsAfterRegionMovedWithMultiCF(TestWALReplay.java:190)
          at org.apache.hadoop.hbase.TestZooKeeper.testRegionAssignmentAfterMasterRecoveryDueToZKExpiry(TestZooKeeper.java:488)
          at org.apache.hadoop.hbase.client.TestFromClientSide.testScanMetrics(TestFromClientSide.java:4740)

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6473//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6473//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6473//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6473//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6473//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6473//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6473//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6473//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6473//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6473//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12594295/HBASE-6143-0.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 20 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. -1 javadoc . The javadoc tool appears to have generated 1 warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 lineLengths . The patch introduces lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.coprocessor.TestMasterObserver org.apache.hadoop.hbase.regionserver.TestRegionMergeTransactionOnCluster org.apache.hadoop.hbase.security.access.TestAccessController org.apache.hadoop.hbase.master.TestRegionPlacement org.apache.hadoop.hbase.TestRegionRebalancing org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster org.apache.hadoop.hbase.regionserver.TestHRegionOnCluster org.apache.hadoop.hbase.snapshot.TestFlushSnapshotFromClient org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster org.apache.hadoop.hbase.master.TestRollingRestart -1 core zombie tests . There are 10 zombie test(s): at org.apache.hadoop.hbase.client.TestFromClientSide.testScanMetrics(TestFromClientSide.java:4740) at org.apache.hadoop.hbase.util.TestHBaseFsck.testValidLingeringSplitParent(TestHBaseFsck.java:1283) at org.apache.hadoop.hbase.client.TestScannersFromClientSide.testScanBatch(TestScannersFromClientSide.java:131) at org.apache.hadoop.hbase.client.TestHCM.testRegionCaching(TestHCM.java:365) at org.apache.hadoop.hbase.regionserver.wal.TestWALReplay.testReplayEditsAfterRegionMovedWithMultiCF(TestWALReplay.java:190) at org.apache.hadoop.hbase.TestZooKeeper.testRegionAssignmentAfterMasterRecoveryDueToZKExpiry(TestZooKeeper.java:488) at org.apache.hadoop.hbase.client.TestFromClientSide.testScanMetrics(TestFromClientSide.java:4740) Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6473//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6473//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6473//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6473//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6473//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6473//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6473//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6473//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6473//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6473//console This message is automatically generated.
          Hide
          Lars Hofhansl added a comment -

          We should probably be doing different patches in 0.94 and 0.95+.
          Generally, loosing data locality after a simple disable/enable is absolutely unacceptable!

          Show
          Lars Hofhansl added a comment - We should probably be doing different patches in 0.94 and 0.95+. Generally, loosing data locality after a simple disable/enable is absolutely unacceptable!
          Hide
          Lars Hofhansl added a comment -

          In 0.94 I still think it would be best to just do the same that AssignmentManager.joinCluster() does, namely eventually calling AssignmentManager.assign(Map<HRegionInfo, ServerName> regions). If there's a agreement around that I'll come up with a patch.

          Show
          Lars Hofhansl added a comment - In 0.94 I still think it would be best to just do the same that AssignmentManager.joinCluster() does, namely eventually calling AssignmentManager.assign(Map<HRegionInfo, ServerName> regions). If there's a agreement around that I'll come up with a patch.
          Hide
          Lars Hofhansl added a comment -

          Looking at the code again. I think Ted's patch is good. I didn't realize the RegionPlans are cached in the AssignmentManager.

          So +1 for v3 for 0.94 from me.

          We should have separate jiras for 0.94 and 0.95+. Should we leave this for 0.95 and make a new one for 0.94 or the other way 'round?

          Show
          Lars Hofhansl added a comment - Looking at the code again. I think Ted's patch is good. I didn't realize the RegionPlans are cached in the AssignmentManager. So +1 for v3 for 0.94 from me. We should have separate jiras for 0.94 and 0.95+. Should we leave this for 0.95 and make a new one for 0.94 or the other way 'round?
          Hide
          Ted Yu added a comment -

          I would say open a new JIRA for 0.94 and link that JIRA to this one.

          The subject of this JIRA implies that more can be done in this subject area.

          Show
          Ted Yu added a comment - I would say open a new JIRA for 0.94 and link that JIRA to this one. The subject of this JIRA implies that more can be done in this subject area.
          Hide
          Lars Hofhansl added a comment - - edited

          Need to add a test, and we need to performance test this. The old code path used the StartupBulkAssigner (if that is not safe in 0.94, we do have another bug here), the new code assigns region by region.

          Show
          Lars Hofhansl added a comment - - edited Need to add a test, and we need to performance test this. The old code path used the StartupBulkAssigner (if that is not safe in 0.94, we do have another bug here), the new code assigns region by region.
          Hide
          Ted Yu added a comment -

          @Lars:

          +  public void testEnableTableRetainAssignment() throws IOException, InterruptedException {
          

          Can you let me know what other test should be added beyond the one above ?

          Thanks

          Show
          Ted Yu added a comment - @Lars: + public void testEnableTableRetainAssignment() throws IOException, InterruptedException { Can you let me know what other test should be added beyond the one above ? Thanks
          Hide
          Elliott Clark added a comment -

          New version that passes more tests. Lets see what jenkins thinks of this.

          Still need to clean up AM there are still code paths that I haven't really cleaned up yet.

          Show
          Elliott Clark added a comment - New version that passes more tests. Lets see what jenkins thinks of this. Still need to clean up AM there are still code paths that I haven't really cleaned up yet.
          Hide
          Lars Hofhansl added a comment -

          Ted Yu Sorry, missed the test namechange. Also turns out my previous comment was wrong again. hbase.master.enabletable.roundrobin is false by default, so the current pass would always use the single assignment in 0.94. Wanna create a 0.94 issue, Ted, or want me to do that?

          Show
          Lars Hofhansl added a comment - Ted Yu Sorry, missed the test namechange. Also turns out my previous comment was wrong again. hbase.master.enabletable.roundrobin is false by default, so the current pass would always use the single assignment in 0.94. Wanna create a 0.94 issue, Ted, or want me to do that?
          Hide
          Lars Hofhansl added a comment -

          Elliott Clark Love all that remove code! Will digest the path a bit more.

          Show
          Lars Hofhansl added a comment - Elliott Clark Love all that remove code! Will digest the path a bit more.
          Hide
          Ted Yu added a comment -

          @Lars:
          I logged HBASE-9080 for 0.94 branch.

          Show
          Ted Yu added a comment - @Lars: I logged HBASE-9080 for 0.94 branch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12594831/HBASE-6143-1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 20 new or modified tests.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 lineLengths. The patch introduces lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.regionserver.TestRegionMergeTransactionOnCluster
          org.apache.hadoop.hbase.client.TestAdmin
          org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster
          org.apache.hadoop.hbase.TestDrainingServer
          org.apache.hadoop.hbase.snapshot.TestFlushSnapshotFromClient
          org.apache.hadoop.hbase.client.TestFromClientSide
          org.apache.hadoop.hbase.master.TestRegionPlacement
          org.apache.hadoop.hbase.client.TestFromClientSideWithCoprocessor
          org.apache.hadoop.hbase.security.access.TestAccessController
          org.apache.hadoop.hbase.master.TestDistributedLogSplitting

          -1 core zombie tests. There are 3 zombie test(s): at org.apache.hadoop.hbase.client.TestHCM.testMulti(TestHCM.java:771)
          at org.apache.hadoop.hbase.regionserver.wal.TestWALReplay.testReplayEditsAfterRegionMovedWithMultiCF(TestWALReplay.java:190)
          at org.apache.hadoop.hbase.TestZooKeeper.testRegionAssignmentAfterMasterRecoveryDueToZKExpiry(TestZooKeeper.java:488)

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6516//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6516//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6516//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6516//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6516//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6516//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6516//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6516//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6516//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6516//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12594831/HBASE-6143-1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 20 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. -1 javadoc . The javadoc tool appears to have generated 1 warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 lineLengths . The patch introduces lines longer than 100 +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.TestRegionMergeTransactionOnCluster org.apache.hadoop.hbase.client.TestAdmin org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster org.apache.hadoop.hbase.TestDrainingServer org.apache.hadoop.hbase.snapshot.TestFlushSnapshotFromClient org.apache.hadoop.hbase.client.TestFromClientSide org.apache.hadoop.hbase.master.TestRegionPlacement org.apache.hadoop.hbase.client.TestFromClientSideWithCoprocessor org.apache.hadoop.hbase.security.access.TestAccessController org.apache.hadoop.hbase.master.TestDistributedLogSplitting -1 core zombie tests . There are 3 zombie test(s): at org.apache.hadoop.hbase.client.TestHCM.testMulti(TestHCM.java:771) at org.apache.hadoop.hbase.regionserver.wal.TestWALReplay.testReplayEditsAfterRegionMovedWithMultiCF(TestWALReplay.java:190) at org.apache.hadoop.hbase.TestZooKeeper.testRegionAssignmentAfterMasterRecoveryDueToZKExpiry(TestZooKeeper.java:488) Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6516//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6516//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6516//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6516//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6516//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6516//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6516//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6516//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6516//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6516//console This message is automatically generated.
          Hide
          Vladimir Rodionov added a comment -

          OK, this is probably not the right place but :

          requestsPerSecond=0, numberOfOnlineRegions=398, numberOfStores=415, numberOfStorefiles=135, storefileIndexSizeMB=1, rootIndexSizeKB=1865, totalStaticIndexSizeKB=25208, totalStaticBloomSizeKB=30481, memstoreSizeMB=0, mbInMemoryWithoutWAL=0, numberOfPutsWithoutWAL=0, readRequestsCount=0, writeRequestsCount=0, compactionQueueSize=0, flushQueueSize=0, usedHeapMB=118, maxHeapMB=5987, blockCacheSizeMB=12.28, blockCacheFreeMB=1484.53, blockCacheCount=0, blockCacheHitCount=0, blockCacheMissCount=0, blockCacheEvictedCount=0, blockCacheHitRatio=0%, blockCacheHitCachingRatio=0%, hdfsBlocksLocalityIndex=24, slowHLogAppendCount=0, fsReadLatencyHistogramMean=0.00, fsReadLatencyHistogramCount=0.00, fsReadLatencyHistogramMedian=0.00, fsReadLatencyHistogram75th=0.00, fsReadLatencyHistogram95th=0.00, fsReadLatencyHistogram99th=0.00, fsReadLatencyHistogram999th=0.00, fsPreadLatencyHistogramMean=0.00, fsPreadLatencyHistogramCount=0.00, fsPreadLatencyHistogramMedian=0.00, fsPreadLatencyHistogram75th=0.00, fsPreadLatencyHistogram95th=0.00, fsPreadLatencyHistogram99th=0.00, fsPreadLatencyHistogram999th=0.00, fsWriteLatencyHistogramMean=0.00, fsWriteLatencyHistogramCount=0.00, fsWriteLatencyHistogramMedian=0.00, fsWriteLatencyHistogram75th=0.00, fsWriteLatencyHistogram95th=0.00, fsWriteLatencyHistogram99th=0.00, fsWriteLatencyHistogram999th=0.00
          

          I do have very low hdfsBlocksLocalityIndex (< 25) on ALL RS in my cluster. I restarted cluster several times but the locality is still very poor. It seems that smart algorithm of region assignments is not 100% correct?

          0.94.6

          Show
          Vladimir Rodionov added a comment - OK, this is probably not the right place but : requestsPerSecond=0, numberOfOnlineRegions=398, numberOfStores=415, numberOfStorefiles=135, storefileIndexSizeMB=1, rootIndexSizeKB=1865, totalStaticIndexSizeKB=25208, totalStaticBloomSizeKB=30481, memstoreSizeMB=0, mbInMemoryWithoutWAL=0, numberOfPutsWithoutWAL=0, readRequestsCount=0, writeRequestsCount=0, compactionQueueSize=0, flushQueueSize=0, usedHeapMB=118, maxHeapMB=5987, blockCacheSizeMB=12.28, blockCacheFreeMB=1484.53, blockCacheCount=0, blockCacheHitCount=0, blockCacheMissCount=0, blockCacheEvictedCount=0, blockCacheHitRatio=0%, blockCacheHitCachingRatio=0%, hdfsBlocksLocalityIndex=24, slowHLogAppendCount=0, fsReadLatencyHistogramMean=0.00, fsReadLatencyHistogramCount=0.00, fsReadLatencyHistogramMedian=0.00, fsReadLatencyHistogram75th=0.00, fsReadLatencyHistogram95th=0.00, fsReadLatencyHistogram99th=0.00, fsReadLatencyHistogram999th=0.00, fsPreadLatencyHistogramMean=0.00, fsPreadLatencyHistogramCount=0.00, fsPreadLatencyHistogramMedian=0.00, fsPreadLatencyHistogram75th=0.00, fsPreadLatencyHistogram95th=0.00, fsPreadLatencyHistogram99th=0.00, fsPreadLatencyHistogram999th=0.00, fsWriteLatencyHistogramMean=0.00, fsWriteLatencyHistogramCount=0.00, fsWriteLatencyHistogramMedian=0.00, fsWriteLatencyHistogram75th=0.00, fsWriteLatencyHistogram95th=0.00, fsWriteLatencyHistogram99th=0.00, fsWriteLatencyHistogram999th=0.00 I do have very low hdfsBlocksLocalityIndex (< 25) on ALL RS in my cluster. I restarted cluster several times but the locality is still very poor. It seems that smart algorithm of region assignments is not 100% correct? 0.94.6
          Hide
          Lars Hofhansl added a comment -

          Regions are not rebalanced based on locality (currently - but see HBASE-4191). The only way to restore locality (after it was lost) is to trigger compactions, so that data gets rewritten locally.

          Facebook also worked on HDFS level placement groups in order to improve locality (I do not think this is in any official Hadoop release, though).

          Show
          Lars Hofhansl added a comment - Regions are not rebalanced based on locality (currently - but see HBASE-4191 ). The only way to restore locality (after it was lost) is to trigger compactions, so that data gets rewritten locally. Facebook also worked on HDFS level placement groups in order to improve locality (I do not think this is in any official Hadoop release, though).
          Hide
          Elliott Clark added a comment -

          Regions are not rebalanced based on locality

          Trunk should already have locality awareness built in. Since not every file will be created on the same set of datanodes it will be hard to have 100% until the favored nodes functionality is in a shipped version of HDFS.

          https://github.com/apache/hbase/blob/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java#L524

          0.94.6

          HBASE-9080 was opened up for 0.94 branch, to deal with the full restart case. But like Lars Hofhansl said in 0.94 the balancer doesn't know about locality so the only way to get it for sure is to have compactions run (either minor or major).

          Show
          Elliott Clark added a comment - Regions are not rebalanced based on locality Trunk should already have locality awareness built in. Since not every file will be created on the same set of datanodes it will be hard to have 100% until the favored nodes functionality is in a shipped version of HDFS. https://github.com/apache/hbase/blob/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java#L524 0.94.6 HBASE-9080 was opened up for 0.94 branch, to deal with the full restart case. But like Lars Hofhansl said in 0.94 the balancer doesn't know about locality so the only way to get it for sure is to have compactions run (either minor or major).
          Hide
          Elliott Clark added a comment -

          Attaching a patch just to get QA to chew on it. I still need to work on the tests but the rest of the code is pretty close.

          Balancer now has two different methods. One fast, and one regular. This should mean that when server shut down handler, or table enable handler are called the balance they use can be much better.

          Show
          Elliott Clark added a comment - Attaching a patch just to get QA to chew on it. I still need to work on the tests but the rest of the code is pretty close. Balancer now has two different methods. One fast, and one regular. This should mean that when server shut down handler, or table enable handler are called the balance they use can be much better.
          Hide
          Elliott Clark added a comment -
          Show
          Elliott Clark added a comment - Review is up here: https://reviews.facebook.net/D36267
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12708729/HBASE-6143-v1.patch
          against master branch at commit 485800830a58da1af31beaac859d99b56d823f99.
          ATTACHMENT ID: 12708729

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 20 new or modified tests.

          +1 hadoop versions. The patch compiles with all supported hadoop versions (2.4.1 2.5.2 2.6.0)

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 protoc. The applied patch does not increase the total number of protoc compiler warnings.

          -1 javadoc. The javadoc tool appears to have generated 6 warning messages.

          -1 checkstyle. The applied patch generated 1937 checkstyle errors (more than the master's current 1924 errors).

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          -1 release audit. The applied patch generated 1 release audit warnings (more than the master's current 0 warnings).

          -1 lineLengths. The patch introduces the following lines longer than 100:
          + BulkAssigner ba = new GeneralBulkAssigner(this.server, bulkPlan, this, bulkAssignWaitTillAllAssigned);
          + Map<HRegionInfo, ServerName> suggest) throws HBaseIOException {
          +// LOG.debug("testImmediateAssignment with " + mock[0] + " regions and " + mock[1] + " servers");

          +1 site. The mvn site goal succeeds with this patch.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.master.TestMasterOperationsForRegionReplicas
          org.apache.hadoop.hbase.master.handler.TestEnableTableHandler
          org.apache.hadoop.hbase.TestRegionRebalancing
          org.apache.hadoop.hbase.master.TestMaster
          org.apache.hadoop.hbase.client.TestAdmin1
          org.apache.hadoop.hbase.master.TestRestartCluster
          org.apache.hadoop.hbase.master.TestRegionPlacement

          -1 core zombie tests. There are 2 zombie test(s): at org.apache.hadoop.hbase.coprocessor.TestRegionServerObserver.testCoprocessorHooksInRegionsMerge(TestRegionServerObserver.java:100)
          at org.apache.hadoop.hbase.regionserver.TestEncryptionKeyRotation.testMasterKeyRotation(TestEncryptionKeyRotation.java:189)

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/13527//testReport/
          Release audit warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13527//artifact/patchprocess/patchReleaseAuditWarnings.txt
          Release Findbugs (version 2.0.3) warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13527//artifact/patchprocess/newFindbugsWarnings.html
          Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/13527//artifact/patchprocess/checkstyle-aggregate.html

          Javadoc warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13527//artifact/patchprocess/patchJavadocWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/13527//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12708729/HBASE-6143-v1.patch against master branch at commit 485800830a58da1af31beaac859d99b56d823f99. ATTACHMENT ID: 12708729 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 20 new or modified tests. +1 hadoop versions . The patch compiles with all supported hadoop versions (2.4.1 2.5.2 2.6.0) +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 protoc . The applied patch does not increase the total number of protoc compiler warnings. -1 javadoc . The javadoc tool appears to have generated 6 warning messages. -1 checkstyle . The applied patch generated 1937 checkstyle errors (more than the master's current 1924 errors). +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. -1 release audit . The applied patch generated 1 release audit warnings (more than the master's current 0 warnings). -1 lineLengths . The patch introduces the following lines longer than 100: + BulkAssigner ba = new GeneralBulkAssigner(this.server, bulkPlan, this, bulkAssignWaitTillAllAssigned); + Map<HRegionInfo, ServerName> suggest) throws HBaseIOException { +// LOG.debug("testImmediateAssignment with " + mock [0] + " regions and " + mock [1] + " servers"); +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.master.TestMasterOperationsForRegionReplicas org.apache.hadoop.hbase.master.handler.TestEnableTableHandler org.apache.hadoop.hbase.TestRegionRebalancing org.apache.hadoop.hbase.master.TestMaster org.apache.hadoop.hbase.client.TestAdmin1 org.apache.hadoop.hbase.master.TestRestartCluster org.apache.hadoop.hbase.master.TestRegionPlacement -1 core zombie tests . There are 2 zombie test(s): at org.apache.hadoop.hbase.coprocessor.TestRegionServerObserver.testCoprocessorHooksInRegionsMerge(TestRegionServerObserver.java:100) at org.apache.hadoop.hbase.regionserver.TestEncryptionKeyRotation.testMasterKeyRotation(TestEncryptionKeyRotation.java:189) Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/13527//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13527//artifact/patchprocess/patchReleaseAuditWarnings.txt Release Findbugs (version 2.0.3) warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13527//artifact/patchprocess/newFindbugsWarnings.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/13527//artifact/patchprocess/checkstyle-aggregate.html Javadoc warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13527//artifact/patchprocess/patchJavadocWarnings.txt Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/13527//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12709085/HBASE-6143-v2.patch
          against master branch at commit 2365932d78180255be7145d8d247ee1f801aed1d.
          ATTACHMENT ID: 12709085

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 20 new or modified tests.

          +1 hadoop versions. The patch compiles with all supported hadoop versions (2.4.1 2.5.2 2.6.0)

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 protoc. The applied patch does not increase the total number of protoc compiler warnings.

          -1 javadoc. The javadoc tool appears to have generated 2 warning messages.

          -1 checkstyle. The applied patch generated 1940 checkstyle errors (more than the master's current 1924 errors).

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 lineLengths. The patch introduces the following lines longer than 100:
          + BulkAssigner ba = new GeneralBulkAssigner(this.server, bulkPlan, this, bulkAssignWaitTillAllAssigned);
          + Map<ServerName, ServerName> noTimeToCurrent = new HashMap<ServerName,ServerName>(clusterState.size());
          + Map<HRegionInfo, ServerName> initialLocation = new HashMap<HRegionInfo, ServerName>(regions.size());
          + sn = noTimeToCurrent.get(ServerName.valueOf(old.getHostname(), old.getPort(), ServerName.NON_STARTCODE));
          + Map<HRegionInfo, ServerName> suggest) throws HBaseIOException {
          +// conf.setClass("hbase.util.ip.to.rack.determiner", MockMapping.class, DNSToSwitchMapping.class);
          +// clusterState.put(servers[5], list1); //servers[5], rack2 hosts replica_of_region1 and replica_of_region2
          +// clusterState.put(servers[10], new ArrayList<HRegionInfo>()); //servers[10], rack3 hosts no region
          +// Map<ServerName, List<HRegionInfo>> clusterState = new HashMap<ServerName, List<HRegionInfo>>();
          +// assertEquals(10, cluster.numServers); // only 10 servers because they share the same host + port

          +1 site. The mvn site goal succeeds with this patch.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.master.TestMaster
          org.apache.hadoop.hbase.master.TestRegionPlacement
          org.apache.hadoop.hbase.util.TestHBaseFsck
          org.apache.hadoop.hbase.util.TestMergeTable
          org.apache.hadoop.hbase.master.TestMasterOperationsForRegionReplicas
          org.apache.hadoop.hbase.replication.TestReplicationSyncUpTool
          org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildBase
          org.apache.hadoop.hbase.master.TestRestartCluster
          org.apache.hadoop.hbase.regionserver.TestRSKilledWhenInitializing
          org.apache.hadoop.hbase.TestRegionRebalancing
          org.apache.hadoop.hbase.client.TestAdmin1

          -1 core zombie tests. There are 6 zombie test(s): at org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClient.testRestoreSnapshot(TestRestoreSnapshotFromClient.java:147)
          at org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClient.testRestoreSnapshot(TestRestoreSnapshotFromClient.java:147)
          at org.apache.hadoop.hbase.client.TestHCM.testConnectionClose(TestHCM.java:327)
          at org.apache.hadoop.hbase.client.TestHCM.testConnectionCloseAllowsInterrupt(TestHCM.java:280)
          at org.apache.hadoop.hbase.snapshot.TestRestoreFlushSnapshotFromClient.testRestoreSnapshot(TestRestoreFlushSnapshotFromClient.java:151)
          at org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels.testVisibilityLabelsInScanThatDoesNotMatchAnyDefinedLabels(TestVisibilityLabels.java:255)
          at org.apache.hadoop.hbase.client.TestSnapshotMetadata.testDescribeOnEmptyTableMatchesAfterMetadataChangeAndRestore(TestSnapshotMetadata.java:248)

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/13545//testReport/
          Release Findbugs (version 2.0.3) warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13545//artifact/patchprocess/newFindbugsWarnings.html
          Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/13545//artifact/patchprocess/checkstyle-aggregate.html

          Javadoc warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13545//artifact/patchprocess/patchJavadocWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/13545//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12709085/HBASE-6143-v2.patch against master branch at commit 2365932d78180255be7145d8d247ee1f801aed1d. ATTACHMENT ID: 12709085 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 20 new or modified tests. +1 hadoop versions . The patch compiles with all supported hadoop versions (2.4.1 2.5.2 2.6.0) +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 protoc . The applied patch does not increase the total number of protoc compiler warnings. -1 javadoc . The javadoc tool appears to have generated 2 warning messages. -1 checkstyle . The applied patch generated 1940 checkstyle errors (more than the master's current 1924 errors). +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 lineLengths . The patch introduces the following lines longer than 100: + BulkAssigner ba = new GeneralBulkAssigner(this.server, bulkPlan, this, bulkAssignWaitTillAllAssigned); + Map<ServerName, ServerName> noTimeToCurrent = new HashMap<ServerName,ServerName>(clusterState.size()); + Map<HRegionInfo, ServerName> initialLocation = new HashMap<HRegionInfo, ServerName>(regions.size()); + sn = noTimeToCurrent.get(ServerName.valueOf(old.getHostname(), old.getPort(), ServerName.NON_STARTCODE)); + Map<HRegionInfo, ServerName> suggest) throws HBaseIOException { +// conf.setClass("hbase.util.ip.to.rack.determiner", MockMapping.class, DNSToSwitchMapping.class); +// clusterState.put(servers [5] , list1); //servers [5] , rack2 hosts replica_of_region1 and replica_of_region2 +// clusterState.put(servers [10] , new ArrayList<HRegionInfo>()); //servers [10] , rack3 hosts no region +// Map<ServerName, List<HRegionInfo>> clusterState = new HashMap<ServerName, List<HRegionInfo>>(); +// assertEquals(10, cluster.numServers); // only 10 servers because they share the same host + port +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.master.TestMaster org.apache.hadoop.hbase.master.TestRegionPlacement org.apache.hadoop.hbase.util.TestHBaseFsck org.apache.hadoop.hbase.util.TestMergeTable org.apache.hadoop.hbase.master.TestMasterOperationsForRegionReplicas org.apache.hadoop.hbase.replication.TestReplicationSyncUpTool org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildBase org.apache.hadoop.hbase.master.TestRestartCluster org.apache.hadoop.hbase.regionserver.TestRSKilledWhenInitializing org.apache.hadoop.hbase.TestRegionRebalancing org.apache.hadoop.hbase.client.TestAdmin1 -1 core zombie tests . There are 6 zombie test(s): at org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClient.testRestoreSnapshot(TestRestoreSnapshotFromClient.java:147) at org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClient.testRestoreSnapshot(TestRestoreSnapshotFromClient.java:147) at org.apache.hadoop.hbase.client.TestHCM.testConnectionClose(TestHCM.java:327) at org.apache.hadoop.hbase.client.TestHCM.testConnectionCloseAllowsInterrupt(TestHCM.java:280) at org.apache.hadoop.hbase.snapshot.TestRestoreFlushSnapshotFromClient.testRestoreSnapshot(TestRestoreFlushSnapshotFromClient.java:151) at org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels.testVisibilityLabelsInScanThatDoesNotMatchAnyDefinedLabels(TestVisibilityLabels.java:255) at org.apache.hadoop.hbase.client.TestSnapshotMetadata.testDescribeOnEmptyTableMatchesAfterMetadataChangeAndRestore(TestSnapshotMetadata.java:248) Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/13545//testReport/ Release Findbugs (version 2.0.3) warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13545//artifact/patchprocess/newFindbugsWarnings.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/13545//artifact/patchprocess/checkstyle-aggregate.html Javadoc warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13545//artifact/patchprocess/patchJavadocWarnings.txt Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/13545//console This message is automatically generated.

            People

            • Assignee:
              Unassigned
              Reporter:
              Elliott Clark
            • Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

              • Created:
                Updated:

                Development