Details
-
Type:
Improvement
-
Status: Open
-
Priority:
Critical
-
Resolution: Unresolved
-
Affects Version/s: None
-
Fix Version/s: None
-
Component/s: Balancer
-
Labels:None
Description
Right now a random region server is picked when re-enabling a table. This could be much smarter.
Issue Links
- is related to
-
HBASE-9080
Retain assignment should be used when re-enabling table(s)
-
- Closed
-
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
I agree. Though sometimes choosing a balance of the regions can be an expensive operation so the balancer should provide an api for a cheap balance.
This is Ted's quick analysis on this issue:
http://mail-archives.apache.org/mod_mbox/hbase-dev/201307.mbox/%3CCALte62yCN7RKcgHxr6nNy6A7UyC8g-uztt7epu%3D9OevmbSQE1A%40mail.gmail.com%3E
The full thread starts here:
http://mail-archives.apache.org/mod_mbox/hbase-dev/201307.mbox/%3CDC5EBE7F3610EB4CA5C7E92D76873E1517ECF1DCA9@exchange2007.carrieriq.com%3E
I pretty vehemently disagree that we need to add more apis on the balancer. What we really need to do is simplify and make sure that everything uses the same code path (that code path will include locality and placement).
I agree. We should just default to attempting to keep the previous region assignment. No need to make that optional via a new API.
Both I and Vladimir said that much on the email-thread.
Ted's analysis is only 1/2 the story, though. The retainAssignment flag is tied to skipTableStateCheck right now, and only enabled during recovery.
It is a shame that we go through some length to keep assignments during a full cluster bounce, but drop the ball for a simple disable/enable cycle.
I do not think it is going to be that simple, Ted.
Simply changing that flag has other implication, too. For example we'd no longer check for concurrent enable request neither would we now check for non-existent tables (among others).
Note on patch v2: EnableTableHandler.retainAssignment was improperly named. I renamed it this.skipTableStateCheck to be consistent with the name of parameter passed to ctor.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12593947/6143-v2.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.
+1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 lineLengths. The patch does not introduce lines longer than 100
+1 site. The mvn site goal succeeds with this patch.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.TestIOFencing
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6450//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6450//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6450//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6450//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6450//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6450//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6450//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6450//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6450//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6450//console
This message is automatically generated.
v2 could work. The code path for assignment after enabling is now quite different, though.
Might still be better to call AssignmentManager.assign(Map<HRegionInfo, ServerName> regions) BulkEnabler.populatePool, I think we can build the map ahead of time.
This is the same call used by AssignmentManager.joinCluster(), so we'd use the same code.
Unit test? Is bulk assigner safe? (Jimmy Xiang?) In the past, bulk assigner was known to not be safe and if we failed, because we were using it on startup only, then we'd just exit so user would retry. Jimmy may have fixed this.
If we use AssignmentManager.assign(Map<HRegionInfo, ServerName> regions) we're using the same code as used in startup, hopefully that is safe. We absolutely would need a test for this.
Turns out that there was a test in TestAdmin involving enabling table.
With patch v2, the modified test passes.
Without patch v2, I got:
Failed tests: testEnableTableRetainAssignment(org.apache.hadoop.hbase.client.TestAdmin): expected:<192.168.0.13,50614,1374729419850> but was:<192.168.0.13,50613,1374729419822>
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12594104/6143-v3.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.
+1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 lineLengths. The patch does not introduce lines longer than 100
+1 site. The mvn site goal succeeds with this patch.
+1 core tests. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6460//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6460//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6460//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6460//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6460//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6460//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6460//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6460//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6460//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6460//console
This message is automatically generated.
Stack, in trunk and 0.95, bulk assigner is safe.
I was thinking something more along the lines of this. It moves everything onto fewer code paths. And it also allows for bulk assigns.
It's really raw and I doubt it even runs yet. But I think the simplification of the code is worth it.
Did a skim. I like all the removes and the emergence of a simple pattern all follow when they want to do region moving. Let me know when you want a closer review.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12594295/HBASE-6143-0.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 20 new or modified tests.
+1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.
+1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.
-1 javadoc. The javadoc tool appears to have generated 1 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 lineLengths. The patch introduces lines longer than 100
-1 site. The patch appears to cause mvn site goal to fail.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.coprocessor.TestMasterObserver
org.apache.hadoop.hbase.regionserver.TestRegionMergeTransactionOnCluster
org.apache.hadoop.hbase.security.access.TestAccessController
org.apache.hadoop.hbase.master.TestRegionPlacement
org.apache.hadoop.hbase.TestRegionRebalancing
org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster
org.apache.hadoop.hbase.regionserver.TestHRegionOnCluster
org.apache.hadoop.hbase.snapshot.TestFlushSnapshotFromClient
org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster
org.apache.hadoop.hbase.master.TestRollingRestart
-1 core zombie tests. There are 10 zombie test(s): at org.apache.hadoop.hbase.client.TestFromClientSide.testScanMetrics(TestFromClientSide.java:4740)
at org.apache.hadoop.hbase.util.TestHBaseFsck.testValidLingeringSplitParent(TestHBaseFsck.java:1283)
at org.apache.hadoop.hbase.client.TestScannersFromClientSide.testScanBatch(TestScannersFromClientSide.java:131)
at org.apache.hadoop.hbase.client.TestHCM.testRegionCaching(TestHCM.java:365)
at org.apache.hadoop.hbase.regionserver.wal.TestWALReplay.testReplayEditsAfterRegionMovedWithMultiCF(TestWALReplay.java:190)
at org.apache.hadoop.hbase.TestZooKeeper.testRegionAssignmentAfterMasterRecoveryDueToZKExpiry(TestZooKeeper.java:488)
at org.apache.hadoop.hbase.client.TestFromClientSide.testScanMetrics(TestFromClientSide.java:4740)
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6473//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6473//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6473//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6473//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6473//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6473//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6473//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6473//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6473//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6473//console
This message is automatically generated.
We should probably be doing different patches in 0.94 and 0.95+.
Generally, loosing data locality after a simple disable/enable is absolutely unacceptable!
In 0.94 I still think it would be best to just do the same that AssignmentManager.joinCluster() does, namely eventually calling AssignmentManager.assign(Map<HRegionInfo, ServerName> regions). If there's a agreement around that I'll come up with a patch.
Looking at the code again. I think Ted's patch is good. I didn't realize the RegionPlans are cached in the AssignmentManager.
So +1 for v3 for 0.94 from me.
We should have separate jiras for 0.94 and 0.95+. Should we leave this for 0.95 and make a new one for 0.94 or the other way 'round?
I would say open a new JIRA for 0.94 and link that JIRA to this one.
The subject of this JIRA implies that more can be done in this subject area.
Need to add a test, and we need to performance test this. The old code path used the StartupBulkAssigner (if that is not safe in 0.94, we do have another bug here), the new code assigns region by region.
@Lars:
+ public void testEnableTableRetainAssignment() throws IOException, InterruptedException {
Can you let me know what other test should be added beyond the one above ?
Thanks
New version that passes more tests. Lets see what jenkins thinks of this.
Still need to clean up AM there are still code paths that I haven't really cleaned up yet.
Ted Yu Sorry, missed the test namechange. Also turns out my previous comment was wrong again. hbase.master.enabletable.roundrobin is false by default, so the current pass would always use the single assignment in 0.94. Wanna create a 0.94 issue, Ted, or want me to do that?
Elliott Clark Love all that remove code! Will digest the path a bit more.
@Lars:
I logged HBASE-9080 for 0.94 branch.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12594831/HBASE-6143-1.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 20 new or modified tests.
+1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.
+1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.
-1 javadoc. The javadoc tool appears to have generated 1 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 lineLengths. The patch introduces lines longer than 100
+1 site. The mvn site goal succeeds with this patch.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestRegionMergeTransactionOnCluster
org.apache.hadoop.hbase.client.TestAdmin
org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster
org.apache.hadoop.hbase.TestDrainingServer
org.apache.hadoop.hbase.snapshot.TestFlushSnapshotFromClient
org.apache.hadoop.hbase.client.TestFromClientSide
org.apache.hadoop.hbase.master.TestRegionPlacement
org.apache.hadoop.hbase.client.TestFromClientSideWithCoprocessor
org.apache.hadoop.hbase.security.access.TestAccessController
org.apache.hadoop.hbase.master.TestDistributedLogSplitting
-1 core zombie tests. There are 3 zombie test(s): at org.apache.hadoop.hbase.client.TestHCM.testMulti(TestHCM.java:771)
at org.apache.hadoop.hbase.regionserver.wal.TestWALReplay.testReplayEditsAfterRegionMovedWithMultiCF(TestWALReplay.java:190)
at org.apache.hadoop.hbase.TestZooKeeper.testRegionAssignmentAfterMasterRecoveryDueToZKExpiry(TestZooKeeper.java:488)
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6516//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6516//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6516//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6516//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6516//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6516//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6516//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6516//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6516//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6516//console
This message is automatically generated.
OK, this is probably not the right place but :
requestsPerSecond=0, numberOfOnlineRegions=398, numberOfStores=415, numberOfStorefiles=135, storefileIndexSizeMB=1, rootIndexSizeKB=1865, totalStaticIndexSizeKB=25208, totalStaticBloomSizeKB=30481, memstoreSizeMB=0, mbInMemoryWithoutWAL=0, numberOfPutsWithoutWAL=0, readRequestsCount=0, writeRequestsCount=0, compactionQueueSize=0, flushQueueSize=0, usedHeapMB=118, maxHeapMB=5987, blockCacheSizeMB=12.28, blockCacheFreeMB=1484.53, blockCacheCount=0, blockCacheHitCount=0, blockCacheMissCount=0, blockCacheEvictedCount=0, blockCacheHitRatio=0%, blockCacheHitCachingRatio=0%, hdfsBlocksLocalityIndex=24, slowHLogAppendCount=0, fsReadLatencyHistogramMean=0.00, fsReadLatencyHistogramCount=0.00, fsReadLatencyHistogramMedian=0.00, fsReadLatencyHistogram75th=0.00, fsReadLatencyHistogram95th=0.00, fsReadLatencyHistogram99th=0.00, fsReadLatencyHistogram999th=0.00, fsPreadLatencyHistogramMean=0.00, fsPreadLatencyHistogramCount=0.00, fsPreadLatencyHistogramMedian=0.00, fsPreadLatencyHistogram75th=0.00, fsPreadLatencyHistogram95th=0.00, fsPreadLatencyHistogram99th=0.00, fsPreadLatencyHistogram999th=0.00, fsWriteLatencyHistogramMean=0.00, fsWriteLatencyHistogramCount=0.00, fsWriteLatencyHistogramMedian=0.00, fsWriteLatencyHistogram75th=0.00, fsWriteLatencyHistogram95th=0.00, fsWriteLatencyHistogram99th=0.00, fsWriteLatencyHistogram999th=0.00
I do have very low hdfsBlocksLocalityIndex (< 25) on ALL RS in my cluster. I restarted cluster several times but the locality is still very poor. It seems that smart algorithm of region assignments is not 100% correct?
0.94.6
Regions are not rebalanced based on locality (currently - but see HBASE-4191). The only way to restore locality (after it was lost) is to trigger compactions, so that data gets rewritten locally.
Facebook also worked on HDFS level placement groups in order to improve locality (I do not think this is in any official Hadoop release, though).
Regions are not rebalanced based on locality
Trunk should already have locality awareness built in. Since not every file will be created on the same set of datanodes it will be hard to have 100% until the favored nodes functionality is in a shipped version of HDFS.
0.94.6
HBASE-9080 was opened up for 0.94 branch, to deal with the full restart case. But like Lars Hofhansl said in 0.94 the balancer doesn't know about locality so the only way to get it for sure is to have compactions run (either minor or major).
Attaching a patch just to get QA to chew on it. I still need to work on the tests but the rest of the code is pretty close.
Balancer now has two different methods. One fast, and one regular. This should mean that when server shut down handler, or table enable handler are called the balance they use can be much better.
Review is up here: https://reviews.facebook.net/D36267
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12708729/HBASE-6143-v1.patch
against master branch at commit 485800830a58da1af31beaac859d99b56d823f99.
ATTACHMENT ID: 12708729
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 20 new or modified tests.
+1 hadoop versions. The patch compiles with all supported hadoop versions (2.4.1 2.5.2 2.6.0)
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 protoc. The applied patch does not increase the total number of protoc compiler warnings.
-1 javadoc. The javadoc tool appears to have generated 6 warning messages.
-1 checkstyle. The applied patch generated 1937 checkstyle errors (more than the master's current 1924 errors).
+1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.
-1 release audit. The applied patch generated 1 release audit warnings (more than the master's current 0 warnings).
-1 lineLengths. The patch introduces the following lines longer than 100:
+ BulkAssigner ba = new GeneralBulkAssigner(this.server, bulkPlan, this, bulkAssignWaitTillAllAssigned);
+ Map<HRegionInfo, ServerName> suggest) throws HBaseIOException {
+// LOG.debug("testImmediateAssignment with " + mock[0] + " regions and " + mock[1] + " servers");
+1 site. The mvn site goal succeeds with this patch.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.master.TestMasterOperationsForRegionReplicas
org.apache.hadoop.hbase.master.handler.TestEnableTableHandler
org.apache.hadoop.hbase.TestRegionRebalancing
org.apache.hadoop.hbase.master.TestMaster
org.apache.hadoop.hbase.client.TestAdmin1
org.apache.hadoop.hbase.master.TestRestartCluster
org.apache.hadoop.hbase.master.TestRegionPlacement
-1 core zombie tests. There are 2 zombie test(s): at org.apache.hadoop.hbase.coprocessor.TestRegionServerObserver.testCoprocessorHooksInRegionsMerge(TestRegionServerObserver.java:100)
at org.apache.hadoop.hbase.regionserver.TestEncryptionKeyRotation.testMasterKeyRotation(TestEncryptionKeyRotation.java:189)
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/13527//testReport/
Release audit warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13527//artifact/patchprocess/patchReleaseAuditWarnings.txt
Release Findbugs (version 2.0.3) warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13527//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/13527//artifact/patchprocess/checkstyle-aggregate.html
Javadoc warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13527//artifact/patchprocess/patchJavadocWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/13527//console
This message is automatically generated.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12709085/HBASE-6143-v2.patch
against master branch at commit 2365932d78180255be7145d8d247ee1f801aed1d.
ATTACHMENT ID: 12709085
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 20 new or modified tests.
+1 hadoop versions. The patch compiles with all supported hadoop versions (2.4.1 2.5.2 2.6.0)
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 protoc. The applied patch does not increase the total number of protoc compiler warnings.
-1 javadoc. The javadoc tool appears to have generated 2 warning messages.
-1 checkstyle. The applied patch generated 1940 checkstyle errors (more than the master's current 1924 errors).
+1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 lineLengths. The patch introduces the following lines longer than 100:
+ BulkAssigner ba = new GeneralBulkAssigner(this.server, bulkPlan, this, bulkAssignWaitTillAllAssigned);
+ Map<ServerName, ServerName> noTimeToCurrent = new HashMap<ServerName,ServerName>(clusterState.size());
+ Map<HRegionInfo, ServerName> initialLocation = new HashMap<HRegionInfo, ServerName>(regions.size());
+ sn = noTimeToCurrent.get(ServerName.valueOf(old.getHostname(), old.getPort(), ServerName.NON_STARTCODE));
+ Map<HRegionInfo, ServerName> suggest) throws HBaseIOException {
+// conf.setClass("hbase.util.ip.to.rack.determiner", MockMapping.class, DNSToSwitchMapping.class);
+// clusterState.put(servers[5], list1); //servers[5], rack2 hosts replica_of_region1 and replica_of_region2
+// clusterState.put(servers[10], new ArrayList<HRegionInfo>()); //servers[10], rack3 hosts no region
+// Map<ServerName, List<HRegionInfo>> clusterState = new HashMap<ServerName, List<HRegionInfo>>();
+// assertEquals(10, cluster.numServers); // only 10 servers because they share the same host + port
+1 site. The mvn site goal succeeds with this patch.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.master.TestMaster
org.apache.hadoop.hbase.master.TestRegionPlacement
org.apache.hadoop.hbase.util.TestHBaseFsck
org.apache.hadoop.hbase.util.TestMergeTable
org.apache.hadoop.hbase.master.TestMasterOperationsForRegionReplicas
org.apache.hadoop.hbase.replication.TestReplicationSyncUpTool
org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildBase
org.apache.hadoop.hbase.master.TestRestartCluster
org.apache.hadoop.hbase.regionserver.TestRSKilledWhenInitializing
org.apache.hadoop.hbase.TestRegionRebalancing
org.apache.hadoop.hbase.client.TestAdmin1
-1 core zombie tests. There are 6 zombie test(s): at org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClient.testRestoreSnapshot(TestRestoreSnapshotFromClient.java:147)
at org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClient.testRestoreSnapshot(TestRestoreSnapshotFromClient.java:147)
at org.apache.hadoop.hbase.client.TestHCM.testConnectionClose(TestHCM.java:327)
at org.apache.hadoop.hbase.client.TestHCM.testConnectionCloseAllowsInterrupt(TestHCM.java:280)
at org.apache.hadoop.hbase.snapshot.TestRestoreFlushSnapshotFromClient.testRestoreSnapshot(TestRestoreFlushSnapshotFromClient.java:151)
at org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels.testVisibilityLabelsInScanThatDoesNotMatchAnyDefinedLabels(TestVisibilityLabels.java:255)
at org.apache.hadoop.hbase.client.TestSnapshotMetadata.testDescribeOnEmptyTableMatchesAfterMetadataChangeAndRestore(TestSnapshotMetadata.java:248)
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/13545//testReport/
Release Findbugs (version 2.0.3) warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13545//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/13545//artifact/patchprocess/checkstyle-aggregate.html
Javadoc warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13545//artifact/patchprocess/patchJavadocWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/13545//console
This message is automatically generated.
The default behavior should most likely take load balancing into account (See HBASE-4191). In FB, our master has a concept of 'favored nodes' that are assigned upon Region creation and map directly to the HDFS replication pipeline.