2015-06-15 03:02:11,679 INFO [main] hbase.ResourceChecker(147): before: master.TestAssignmentManager#testSSHTimesOutOpeningRegionTransition Thread=8, OpenFileDescriptor=197, MaxFileDescriptor=32768, SystemLoadAverage=717, ProcessCount=101, AvailableMemoryMB=5704, ConnectionCount=0 2015-06-15 03:02:12,564 INFO [main] zookeeper.RecoverableZooKeeper(121): Process identifier=mockedServer connecting to ZooKeeper ensemble=localhost:59952 2015-06-15 03:02:12,687 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer0x0, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-15 03:02:12,693 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(592): mockedServer-0x14df52ac8120000 connected 2015-06-15 03:02:13,738 WARN [main] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2015-06-15 03:02:15,721 DEBUG [Thread-3] executor.ExecutorService(99): Starting executor service name=MASTER_OPEN_REGION-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:02:15,726 DEBUG [Thread-3] executor.ExecutorService(99): Starting executor service name=MASTER_CLOSE_REGION-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:02:15,727 DEBUG [Thread-3] executor.ExecutorService(99): Starting executor service name=MASTER_SERVER_OPERATIONS-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:02:15,727 DEBUG [Thread-3] executor.ExecutorService(99): Starting executor service name=MASTER_META_SERVER_OPERATIONS-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:02:15,942 WARN [Thread-3] impl.MetricsConfig(124): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2015-06-15 03:02:16,173 INFO [Thread-3] balancer.StochasticLoadBalancer(135): loading config 2015-06-15 03:02:16,301 INFO [Thread-3] master.RegionStates(1112): Transition null to {cae5ea3b1868354349837ee2dd0d4d89 state=OPEN, ts=1434337336301, server=example.org,0,5678} 2015-06-15 03:02:16,302 DEBUG [Thread-3] master.RegionStates(460): Onlined cae5ea3b1868354349837ee2dd0d4d89 on example.org,0,5678 {ENCODED => cae5ea3b1868354349837ee2dd0d4d89, NAME => 't,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89.', STARTKEY => '', ENDKEY => ''} 2015-06-15 03:02:16,304 INFO [Thread-3] zookeeper.ZKTableStateManager(101): Moving table t state from null to ENABLED 2015-06-15 03:02:16,635 INFO [Thread-3] handler.ServerShutdownHandler(217): Skipping log splitting for example.org,1234,5678 2015-06-15 03:02:16,635 INFO [Thread-3] master.RegionStates(661): Found region in {cae5ea3b1868354349837ee2dd0d4d89 state=OPENING, ts=1434337336301, server=example.org,1234,5678} to be reassigned by SSH for example.org,1234,5678 2015-06-15 03:02:16,644 INFO [Thread-3] zookeeper.RecoverableZooKeeper(189): Node /hbase/region-in-transition/cae5ea3b1868354349837ee2dd0d4d89 already deleted, retry=false 2015-06-15 03:02:16,644 INFO [Thread-3] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=OPEN, ts=1434337336301, server=example.org,0,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=OFFLINE, ts=1434337336644, server=example.org,0,5678} 2015-06-15 03:02:16,644 INFO [Thread-3] handler.ServerShutdownHandler(238): Reassigning 1 region(s) that example.org,1234,5678 was carrying (and 1 regions(s) that were opening on this server) 2015-06-15 03:02:16,809 INFO [Thread-3] master.AssignmentManager(1611): Assigning 1 region(s) to example.org,1234,5678 2015-06-15 03:02:16,817 WARN [Thread-3] master.RegionStates(856): Couldn't reach online server example.org,0,5678 2015-06-15 03:02:16,834 WARN [Thread-3] master.RegionStates(856): Couldn't reach online server example.org,0,5678 2015-06-15 03:02:16,834 INFO [Thread-3] master.AssignmentManager(1633): Skip assigning t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89., it's host example.org,0,5678 is dead but not processed yet 2015-06-15 03:02:36,919 DEBUG [Thread-3] master.AssignmentManager(1804): some user regions are still in transition: [{ENCODED => cae5ea3b1868354349837ee2dd0d4d89, NAME => 't,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89.', STARTKEY => '', ENDKEY => ''}] 2015-06-15 03:02:36,920 DEBUG [Thread-3] master.AssignmentManager(1806): Bulk assigning done for example.org,1234,5678 2015-06-15 03:02:36,920 DEBUG [Thread-3] master.DeadServer(118): Finished processing example.org,1234,5678 2015-06-15 03:02:36,921 INFO [Thread-3] handler.ServerShutdownHandler(347): Finished processing of shutdown of example.org,1234,5678 2015-06-15 03:02:36,922 DEBUG [main] zookeeper.ZKAssign(497): mockedServer-0x14df52ac8120000, quorum=localhost:59952, baseZNode=/hbase Deleting any existing unassigned nodes 2015-06-15 03:02:36,958 INFO [main] hbase.ResourceChecker(171): after: master.TestAssignmentManager#testSSHTimesOutOpeningRegionTransition Thread=11 (was 8) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: main-SendThread(localhost:59952) java.lang.Thread.sleep(Native Method) org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:230) org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1185) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1122) Potentially hanging thread: process reaper sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) - Thread LEAK? -, OpenFileDescriptor=203 (was 197) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=32768 (was 32768), SystemLoadAverage=635 (was 717), ProcessCount=101 (was 101), AvailableMemoryMB=5648 (was 5704), ConnectionCount=1 (was 0) - ConnectionCount LEAK? - 2015-06-15 03:02:36,969 INFO [main] hbase.ResourceChecker(147): before: master.TestAssignmentManager#testBalanceRegionOfDeletedTable Thread=11, OpenFileDescriptor=203, MaxFileDescriptor=32768, SystemLoadAverage=635, ProcessCount=101, AvailableMemoryMB=5647, ConnectionCount=1 2015-06-15 03:02:36,970 INFO [main] zookeeper.RecoverableZooKeeper(121): Process identifier=mockedServer connecting to ZooKeeper ensemble=localhost:59952 2015-06-15 03:02:36,983 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer0x0, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-15 03:02:36,984 DEBUG [main] zookeeper.RecoverableZooKeeper(595): Node /hbase already exists 2015-06-15 03:02:36,985 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(592): mockedServer-0x14df52ac8120001 connected 2015-06-15 03:02:37,018 INFO [Thread-6] master.AssignmentManager(3484): Ignored moving region not assigned: {ENCODED => cae5ea3b1868354349837ee2dd0d4d89, NAME => 't,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89.', STARTKEY => '', ENDKEY => ''}, not in region states 2015-06-15 03:02:37,018 DEBUG [main] zookeeper.ZKAssign(497): mockedServer-0x14df52ac8120001, quorum=localhost:59952, baseZNode=/hbase Deleting any existing unassigned nodes 2015-06-15 03:02:37,069 INFO [main] hbase.ResourceChecker(171): after: master.TestAssignmentManager#testBalanceRegionOfDeletedTable Thread=12 (was 11) - Thread LEAK? -, OpenFileDescriptor=208 (was 203) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=32768 (was 32768), SystemLoadAverage=635 (was 635), ProcessCount=101 (was 101), AvailableMemoryMB=5640 (was 5647), ConnectionCount=1 (was 1) 2015-06-15 03:02:37,111 INFO [main] hbase.ResourceChecker(147): before: master.TestAssignmentManager#testBalanceOnMasterFailoverScenarioWithClosedNode Thread=11, OpenFileDescriptor=205, MaxFileDescriptor=32768, SystemLoadAverage=635, ProcessCount=101, AvailableMemoryMB=5640, ConnectionCount=1 2015-06-15 03:02:37,112 INFO [main] zookeeper.RecoverableZooKeeper(121): Process identifier=mockedServer connecting to ZooKeeper ensemble=localhost:59952 2015-06-15 03:02:37,138 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer0x0, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-15 03:02:37,139 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(592): mockedServer-0x14df52ac8120002 connected 2015-06-15 03:02:37,141 DEBUG [main] zookeeper.RecoverableZooKeeper(595): Node /hbase already exists 2015-06-15 03:02:37,173 DEBUG [Thread-7] executor.ExecutorService(99): Starting executor service name=MASTER_OPEN_REGION-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:02:37,173 DEBUG [Thread-7] executor.ExecutorService(99): Starting executor service name=MASTER_CLOSE_REGION-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:02:37,174 DEBUG [Thread-7] executor.ExecutorService(99): Starting executor service name=MASTER_SERVER_OPERATIONS-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:02:37,174 DEBUG [Thread-7] executor.ExecutorService(99): Starting executor service name=MASTER_META_SERVER_OPERATIONS-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:02:37,175 INFO [Thread-7] balancer.StochasticLoadBalancer(135): loading config 2015-06-15 03:02:37,183 INFO [Thread-7] master.RegionStates(1112): Transition null to {cae5ea3b1868354349837ee2dd0d4d89 state=OPEN, ts=1434337357183, server=example.org,1234,5678} 2015-06-15 03:02:37,184 DEBUG [Thread-7] master.RegionStates(460): Onlined cae5ea3b1868354349837ee2dd0d4d89 on example.org,1234,5678 {ENCODED => cae5ea3b1868354349837ee2dd0d4d89, NAME => 't,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89.', STARTKEY => '', ENDKEY => ''} 2015-06-15 03:02:37,184 DEBUG [Thread-7] master.AssignmentManager(2470): Starting unassign of t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. (offlining), current state: {cae5ea3b1868354349837ee2dd0d4d89 state=OPEN, ts=1434337357183, server=example.org,1234,5678} 2015-06-15 03:02:37,184 DEBUG [Thread-7] zookeeper.ZKAssign(527): mockedServer-0x14df52ac8120002, quorum=localhost:59952, baseZNode=/hbase Creating unassigned node cae5ea3b1868354349837ee2dd0d4d89 in a CLOSING state 2015-06-15 03:02:37,216 INFO [Thread-7] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=OPEN, ts=1434337357183, server=example.org,1234,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=PENDING_CLOSE, ts=1434337357216, server=example.org,1234,5678} 2015-06-15 03:02:37,217 WARN [Thread-7] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:02:37,219 WARN [Thread-7] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:02:37,220 WARN [Thread-7] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:02:37,226 WARN [Thread-7] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:02:37,228 WARN [Thread-7] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:02:37,232 WARN [Thread-7] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:02:37,234 WARN [Thread-7] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:02:37,243 WARN [Thread-7] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:02:37,245 WARN [Thread-7] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:02:37,246 WARN [Thread-7] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:02:37,246 WARN [Thread-7] master.RegionStates(1103): Failed to open/close cae5ea3b1868354349837ee2dd0d4d89 on example.org,1234,5678, set to FAILED_CLOSE 2015-06-15 03:02:37,246 INFO [Thread-7] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=PENDING_CLOSE, ts=1434337357216, server=example.org,1234,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=FAILED_CLOSE, ts=1434337357246, server=example.org,1234,5678} 2015-06-15 03:02:37,263 WARN [RunAmJoinCluster] master.RegionStates(339): Tried to create a state for a region already in RegionStates, used existing: {cae5ea3b1868354349837ee2dd0d4d89 state=FAILED_CLOSE, ts=1434337357246, server=example.org,1234,5678}, ignored new: OPEN 2015-06-15 03:02:37,266 DEBUG [RunAmJoinCluster] master.AssignmentManager(540): Found {ENCODED => cae5ea3b1868354349837ee2dd0d4d89, NAME => 't,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89.', STARTKEY => '', ENDKEY => ''} out on cluster 2015-06-15 03:02:37,266 INFO [RunAmJoinCluster] master.AssignmentManager(623): Found regions out on cluster or in RIT; presuming failover 2015-06-15 03:02:37,270 INFO [Thread-7] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=FAILED_CLOSE, ts=1434337357246, server=example.org,1234,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=PENDING_CLOSE, ts=1434337357270, server=example.org,1234,5678} 2015-06-15 03:02:37,270 DEBUG [Thread-7] zookeeper.ZKAssign(805): mockedServer-0x14df52ac8120002, quorum=localhost:59952, baseZNode=/hbase Transitioning cae5ea3b1868354349837ee2dd0d4d89 from M_ZK_REGION_CLOSING to RS_ZK_REGION_CLOSED 2015-06-15 03:02:37,279 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer-0x14df52ac8120002, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/cae5ea3b1868354349837ee2dd0d4d89 2015-06-15 03:02:37,282 DEBUG [Thread-7] zookeeper.ZKAssign(880): mockedServer-0x14df52ac8120002, quorum=localhost:59952, baseZNode=/hbase Transitioned node cae5ea3b1868354349837ee2dd0d4d89 from M_ZK_REGION_CLOSING to RS_ZK_REGION_CLOSED 2015-06-15 03:02:37,290 INFO [RunAmJoinCluster] master.AssignmentManager(767): Processing cae5ea3b1868354349837ee2dd0d4d89 in state: RS_ZK_REGION_CLOSED 2015-06-15 03:02:37,294 INFO [RunAmJoinCluster] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=PENDING_CLOSE, ts=1434337357270, server=example.org,1234,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=CLOSED, ts=1434337357294, server=example.org,1234,5678} 2015-06-15 03:02:37,295 DEBUG [AM.ZK.Worker-pool8-t1] master.AssignmentManager(954): Handling RS_ZK_REGION_CLOSED, server=example.org,1234,5678, region=cae5ea3b1868354349837ee2dd0d4d89, current_state={cae5ea3b1868354349837ee2dd0d4d89 state=CLOSED, ts=1434337357294, server=example.org,1234,5678} 2015-06-15 03:02:37,295 INFO [RunAmJoinCluster] master.AssignmentManager(898): Processed region cae5ea3b1868354349837ee2dd0d4d89 in state RS_ZK_REGION_CLOSED, on server: example.org,1234,5678 2015-06-15 03:02:37,296 WARN [AM.ZK.Worker-pool8-t1] master.AssignmentManager(1022): Received CLOSED for cae5ea3b1868354349837ee2dd0d4d89 from example.org,1234,5678 but the region isn't PENDING_CLOSE/CLOSING here: {cae5ea3b1868354349837ee2dd0d4d89 state=CLOSED, ts=1434337357294, server=example.org,1234,5678} 2015-06-15 03:02:37,304 DEBUG [AM.-pool7-t1] master.AssignmentManager(1953): Force region state offline {cae5ea3b1868354349837ee2dd0d4d89 state=CLOSED, ts=1434337357294, server=example.org,1234,5678} 2015-06-15 03:02:37,306 INFO [RunAmJoinCluster] master.AssignmentManager(499): Joined the cluster in 59ms, failover=true 2015-06-15 03:02:37,311 DEBUG [AM.-pool7-t1] master.AssignmentManager(2381): Found an existing plan for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. destination server is example.org,0,5678 accepted as a dest server = true 2015-06-15 03:02:37,314 DEBUG [AM.-pool7-t1] master.AssignmentManager(2413): No previous transition plan found (or ignoring an existing plan) for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89.; generated random plan=hri=t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89., src=, dest=example.org,0,5678; 2 (online=2) available servers, forceNewPlan=true 2015-06-15 03:02:37,314 INFO [AM.-pool7-t1] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=CLOSED, ts=1434337357294, server=example.org,1234,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=OFFLINE, ts=1434337357314, server=example.org,1234,5678} 2015-06-15 03:02:37,315 DEBUG [AM.-pool7-t1] zookeeper.ZKAssign(205): mockedServer-0x14df52ac8120002, quorum=localhost:59952, baseZNode=/hbase Creating (or updating) unassigned node cae5ea3b1868354349837ee2dd0d4d89 with OFFLINE state 2015-06-15 03:02:37,322 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer-0x14df52ac8120002, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/cae5ea3b1868354349837ee2dd0d4d89 2015-06-15 03:02:37,324 INFO [AM.-pool7-t1] master.AssignmentManager(2122): Assigning t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. to example.org,0,5678 2015-06-15 03:02:37,325 INFO [AM.-pool7-t1] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=OFFLINE, ts=1434337357314, server=example.org,1234,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=PENDING_OPEN, ts=1434337357325, server=example.org,0,5678} 2015-06-15 03:02:37,326 DEBUG [Thread-7] zookeeper.ZKAssign(805): mockedServer-0x14df52ac8120002, quorum=localhost:59952, baseZNode=/hbase Transitioning cae5ea3b1868354349837ee2dd0d4d89 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-06-15 03:02:37,330 DEBUG [Thread-7] zookeeper.ZKAssign(880): mockedServer-0x14df52ac8120002, quorum=localhost:59952, baseZNode=/hbase Transitioned node cae5ea3b1868354349837ee2dd0d4d89 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-06-15 03:02:37,330 DEBUG [Thread-7] zookeeper.ZKAssign(805): mockedServer-0x14df52ac8120002, quorum=localhost:59952, baseZNode=/hbase Transitioning cae5ea3b1868354349837ee2dd0d4d89 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2015-06-15 03:02:37,330 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer-0x14df52ac8120002, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/cae5ea3b1868354349837ee2dd0d4d89 2015-06-15 03:02:37,334 DEBUG [AM.ZK.Worker-pool8-t2] master.AssignmentManager(954): Handling RS_ZK_REGION_OPENING, server=example.org,0,5678, region=cae5ea3b1868354349837ee2dd0d4d89, current_state={cae5ea3b1868354349837ee2dd0d4d89 state=PENDING_OPEN, ts=1434337357325, server=example.org,0,5678} 2015-06-15 03:02:37,334 INFO [AM.ZK.Worker-pool8-t2] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=PENDING_OPEN, ts=1434337357325, server=example.org,0,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=OPENING, ts=1434337357334, server=example.org,0,5678} 2015-06-15 03:02:37,336 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer-0x14df52ac8120002, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/cae5ea3b1868354349837ee2dd0d4d89 2015-06-15 03:02:37,336 DEBUG [Thread-7] zookeeper.ZKAssign(880): mockedServer-0x14df52ac8120002, quorum=localhost:59952, baseZNode=/hbase Transitioned node cae5ea3b1868354349837ee2dd0d4d89 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2015-06-15 03:02:37,351 DEBUG [Thread-7] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:02:37,351 DEBUG [AM.ZK.Worker-pool8-t3] master.AssignmentManager(954): Handling RS_ZK_REGION_OPENED, server=example.org,0,5678, region=cae5ea3b1868354349837ee2dd0d4d89, current_state={cae5ea3b1868354349837ee2dd0d4d89 state=OPENING, ts=1434337357334, server=example.org,0,5678} 2015-06-15 03:02:37,351 INFO [AM.ZK.Worker-pool8-t3] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=OPENING, ts=1434337357334, server=example.org,0,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=OPEN, ts=1434337357351, server=example.org,0,5678} 2015-06-15 03:02:37,353 DEBUG [AM.ZK.Worker-pool8-t3] coordination.ZkOpenRegionCoordination(350): Handling OPENED of cae5ea3b1868354349837ee2dd0d4d89 from master,1,1; deleting unassigned node 2015-06-15 03:02:37,361 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer-0x14df52ac8120002, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/cae5ea3b1868354349837ee2dd0d4d89 2015-06-15 03:02:37,362 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer-0x14df52ac8120002, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-06-15 03:02:37,363 DEBUG [AM.ZK.Worker-pool8-t3] zookeeper.ZKAssign(479): mockedServer-0x14df52ac8120002, quorum=localhost:59952, baseZNode=/hbase Deleted unassigned node cae5ea3b1868354349837ee2dd0d4d89 in expected state RS_ZK_REGION_OPENED 2015-06-15 03:02:37,380 DEBUG [AM.ZK.Worker-pool8-t6] master.AssignmentManager$4(1343): Znode t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. deleted, state: {cae5ea3b1868354349837ee2dd0d4d89 state=OPEN, ts=1434337357351, server=example.org,0,5678} 2015-06-15 03:02:37,381 DEBUG [AM.ZK.Worker-pool8-t6] master.RegionStates(460): Onlined cae5ea3b1868354349837ee2dd0d4d89 on example.org,0,5678 {ENCODED => cae5ea3b1868354349837ee2dd0d4d89, NAME => 't,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89.', STARTKEY => '', ENDKEY => ''} 2015-06-15 03:02:37,390 INFO [AM.ZK.Worker-pool8-t6] master.RegionStates(472): Offlined cae5ea3b1868354349837ee2dd0d4d89 from example.org,1234,5678 2015-06-15 03:02:37,453 DEBUG [main] zookeeper.ZKAssign(497): mockedServer-0x14df52ac8120002, quorum=localhost:59952, baseZNode=/hbase Deleting any existing unassigned nodes 2015-06-15 03:02:37,515 INFO [main] hbase.ResourceChecker(171): after: master.TestAssignmentManager#testBalanceOnMasterFailoverScenarioWithClosedNode Thread=11 (was 11), OpenFileDescriptor=204 (was 205), MaxFileDescriptor=32768 (was 32768), SystemLoadAverage=635 (was 635), ProcessCount=101 (was 101), AvailableMemoryMB=5608 (was 5640), ConnectionCount=1 (was 1) 2015-06-15 03:02:37,535 INFO [main] hbase.ResourceChecker(147): before: master.TestAssignmentManager#testAssignmentEventIgnoredIfNotExpected Thread=11, OpenFileDescriptor=205, MaxFileDescriptor=32768, SystemLoadAverage=635, ProcessCount=101, AvailableMemoryMB=5608, ConnectionCount=1 2015-06-15 03:02:37,536 INFO [main] zookeeper.RecoverableZooKeeper(121): Process identifier=mockedServer connecting to ZooKeeper ensemble=localhost:59952 2015-06-15 03:02:37,556 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer0x0, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-15 03:02:37,560 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(592): mockedServer-0x14df52ac8120003 connected 2015-06-15 03:02:37,563 DEBUG [main] zookeeper.RecoverableZooKeeper(595): Node /hbase already exists 2015-06-15 03:02:37,584 INFO [Thread-8] balancer.StochasticLoadBalancer(135): loading config 2015-06-15 03:02:37,591 INFO [Thread-8] master.RegionStates(1112): Transition null to {1588230740 state=OFFLINE, ts=1434337357591, server=null} 2015-06-15 03:02:37,601 DEBUG [Thread-8] zookeeper.ZKAssign(147): mockedServer-0x14df52ac8120003, quorum=localhost:59952, baseZNode=/hbase Creating unassigned node 1588230740 in OFFLINE state 2015-06-15 03:02:37,604 DEBUG [Thread-8] zookeeper.ZKAssign(805): mockedServer-0x14df52ac8120003, quorum=localhost:59952, baseZNode=/hbase Transitioning 1588230740 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-06-15 03:02:37,620 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer-0x14df52ac8120003, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/1588230740 2015-06-15 03:02:37,620 DEBUG [Thread-8] zookeeper.ZKAssign(880): mockedServer-0x14df52ac8120003, quorum=localhost:59952, baseZNode=/hbase Transitioned node 1588230740 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-06-15 03:02:37,640 DEBUG [AM.ZK.Worker-pool11-t1] master.AssignmentManager(954): Handling RS_ZK_REGION_OPENING, server=example.org,1234,5678, region=1588230740, current_state={1588230740 state=OFFLINE, ts=1434337357591, server=null} 2015-06-15 03:02:37,640 WARN [AM.ZK.Worker-pool11-t1] master.AssignmentManager(1076): Received OPENING for 1588230740 from example.org,1234,5678 but the region isn't PENDING_OPEN/OPENING here: {1588230740 state=OFFLINE, ts=1434337357591, server=null} 2015-06-15 03:02:37,722 DEBUG [main] zookeeper.ZKAssign(497): mockedServer-0x14df52ac8120003, quorum=localhost:59952, baseZNode=/hbase Deleting any existing unassigned nodes 2015-06-15 03:02:37,732 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer-0x14df52ac8120003, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/1588230740 2015-06-15 03:02:37,733 ERROR [main-EventThread] zookeeper.ClientCnxn$EventThread(524): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1d2bd198 rejected from java.util.concurrent.ThreadPoolExecutor@27cd611e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110) at org.apache.hadoop.hbase.master.AssignmentManager.zkEventWorkersSubmit(AssignmentManager.java:1291) at org.apache.hadoop.hbase.master.AssignmentManager.nodeDeleted(AssignmentManager.java:1322) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:532) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) 2015-06-15 03:02:37,777 INFO [main] hbase.ResourceChecker(171): after: master.TestAssignmentManager#testAssignmentEventIgnoredIfNotExpected Thread=12 (was 11) Potentially hanging thread: main-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) - Thread LEAK? -, OpenFileDescriptor=203 (was 205), MaxFileDescriptor=32768 (was 32768), SystemLoadAverage=635 (was 635), ProcessCount=101 (was 101), AvailableMemoryMB=5582 (was 5608), ConnectionCount=1 (was 1) 2015-06-15 03:02:37,810 INFO [main] hbase.ResourceChecker(147): before: master.TestAssignmentManager#testShutdownHandler Thread=11, OpenFileDescriptor=205, MaxFileDescriptor=32768, SystemLoadAverage=635, ProcessCount=101, AvailableMemoryMB=5579, ConnectionCount=1 2015-06-15 03:02:37,811 INFO [main] zookeeper.RecoverableZooKeeper(121): Process identifier=mockedServer connecting to ZooKeeper ensemble=localhost:59952 2015-06-15 03:02:37,835 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer0x0, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-15 03:02:37,839 DEBUG [main] zookeeper.RecoverableZooKeeper(595): Node /hbase already exists 2015-06-15 03:02:37,842 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(592): mockedServer-0x14df52ac8120004 connected 2015-06-15 03:02:37,849 DEBUG [Thread-9] executor.ExecutorService(99): Starting executor service name=MASTER_OPEN_REGION-testShutdownHandler, corePoolSize=3, maxPoolSize=3 2015-06-15 03:02:37,852 DEBUG [Thread-9] executor.ExecutorService(99): Starting executor service name=MASTER_CLOSE_REGION-testShutdownHandler, corePoolSize=3, maxPoolSize=3 2015-06-15 03:02:37,853 DEBUG [Thread-9] executor.ExecutorService(99): Starting executor service name=MASTER_SERVER_OPERATIONS-testShutdownHandler, corePoolSize=3, maxPoolSize=3 2015-06-15 03:02:37,853 DEBUG [Thread-9] executor.ExecutorService(99): Starting executor service name=MASTER_META_SERVER_OPERATIONS-testShutdownHandler, corePoolSize=3, maxPoolSize=3 2015-06-15 03:02:37,864 DEBUG [Thread-9] executor.ExecutorService(99): Starting executor service name=MASTER_OPEN_REGION-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:02:37,865 DEBUG [Thread-9] executor.ExecutorService(99): Starting executor service name=MASTER_CLOSE_REGION-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:02:37,865 DEBUG [Thread-9] executor.ExecutorService(99): Starting executor service name=MASTER_SERVER_OPERATIONS-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:02:37,866 DEBUG [Thread-9] executor.ExecutorService(99): Starting executor service name=MASTER_META_SERVER_OPERATIONS-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:02:37,870 INFO [Thread-9] balancer.StochasticLoadBalancer(135): loading config 2015-06-15 03:02:37,902 INFO [Thread-9] handler.ServerShutdownHandler(217): Skipping log splitting for example.org,1234,5678 2015-06-15 03:02:37,902 INFO [Thread-9] handler.ServerShutdownHandler(238): Reassigning 1 region(s) that example.org,1234,5678 was carrying (and 0 regions(s) that were opening on this server) 2015-06-15 03:02:37,903 INFO [Thread-9] master.AssignmentManager(1611): Assigning 1 region(s) to example.org,1234,5678 2015-06-15 03:02:37,903 WARN [Thread-9] master.AssignmentManager(1947): Assigning a region not in region states: {ENCODED => cae5ea3b1868354349837ee2dd0d4d89, NAME => 't,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89.', STARTKEY => '', ENDKEY => ''} 2015-06-15 03:02:37,904 DEBUG [Thread-9] zookeeper.ZKAssign(174): mockedServer-0x14df52ac8120004, quorum=localhost:59952, baseZNode=/hbase Async create of unassigned node cae5ea3b1868354349837ee2dd0d4d89 with OFFLINE state 2015-06-15 03:02:37,908 DEBUG [main-EventThread] master.OfflineCallback(69): rs={cae5ea3b1868354349837ee2dd0d4d89 state=OFFLINE, ts=1434337357904, server=null}, server=example.org,1234,5678 2015-06-15 03:02:37,910 DEBUG [main-EventThread] master.OfflineCallback$ExistCallback(106): rs={cae5ea3b1868354349837ee2dd0d4d89 state=OFFLINE, ts=1434337357904, server=null}, server=example.org,1234,5678 2015-06-15 03:02:37,910 DEBUG [Thread-9] master.AssignmentManager(1664): example.org,1234,5678 unassigned znodes=1 of total=1; oldCounter=0 2015-06-15 03:02:37,910 INFO [Thread-9] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=OFFLINE, ts=1434337357904, server=null} to {cae5ea3b1868354349837ee2dd0d4d89 state=PENDING_OPEN, ts=1434337357910, server=example.org,1234,5678} 2015-06-15 03:02:58,001 DEBUG [Thread-9] master.AssignmentManager(1804): some user regions are still in transition: [{ENCODED => cae5ea3b1868354349837ee2dd0d4d89, NAME => 't,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89.', STARTKEY => '', ENDKEY => ''}] 2015-06-15 03:02:58,001 DEBUG [Thread-9] master.AssignmentManager(1806): Bulk assigning done for example.org,1234,5678 2015-06-15 03:02:58,002 DEBUG [Thread-9] master.DeadServer(118): Finished processing example.org,1234,5678 2015-06-15 03:02:58,002 INFO [Thread-9] handler.ServerShutdownHandler(347): Finished processing of shutdown of example.org,1234,5678 2015-06-15 03:02:58,002 DEBUG [Thread-9] zookeeper.ZKAssign(497): mockedServer-0x14df52ac8120004, quorum=localhost:59952, baseZNode=/hbase Deleting any existing unassigned nodes 2015-06-15 03:02:58,006 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer-0x14df52ac8120004, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/cae5ea3b1868354349837ee2dd0d4d89 2015-06-15 03:02:58,007 ERROR [main-EventThread] zookeeper.ClientCnxn$EventThread(524): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5ecbdc34 rejected from java.util.concurrent.ThreadPoolExecutor@19e60a8b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110) at org.apache.hadoop.hbase.master.AssignmentManager.zkEventWorkersSubmit(AssignmentManager.java:1291) at org.apache.hadoop.hbase.master.AssignmentManager.nodeDeleted(AssignmentManager.java:1322) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:532) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) 2015-06-15 03:02:58,007 DEBUG [main] zookeeper.ZKAssign(497): mockedServer-0x14df52ac8120004, quorum=localhost:59952, baseZNode=/hbase Deleting any existing unassigned nodes 2015-06-15 03:02:58,055 INFO [main] hbase.ResourceChecker(171): after: master.TestAssignmentManager#testShutdownHandler Thread=11 (was 11), OpenFileDescriptor=203 (was 205), MaxFileDescriptor=32768 (was 32768), SystemLoadAverage=592 (was 635), ProcessCount=102 (was 101) - ProcessCount LEAK? -, AvailableMemoryMB=5065 (was 5579), ConnectionCount=1 (was 1) 2015-06-15 03:02:58,090 INFO [main] hbase.ResourceChecker(147): before: master.TestAssignmentManager#testSSHWhenSplitRegionInProgress Thread=11, OpenFileDescriptor=205, MaxFileDescriptor=32768, SystemLoadAverage=592, ProcessCount=102, AvailableMemoryMB=5057, ConnectionCount=1 2015-06-15 03:02:58,092 INFO [main] zookeeper.RecoverableZooKeeper(121): Process identifier=mockedServer connecting to ZooKeeper ensemble=localhost:59952 2015-06-15 03:02:58,111 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer0x0, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-15 03:02:58,112 DEBUG [main] zookeeper.RecoverableZooKeeper(595): Node /hbase already exists 2015-06-15 03:02:58,116 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(592): mockedServer-0x14df52ac8120005 connected 2015-06-15 03:02:58,131 DEBUG [Thread-10] executor.ExecutorService(99): Starting executor service name=MASTER_OPEN_REGION-testSSHWhenSplitRegionInProgress, corePoolSize=3, maxPoolSize=3 2015-06-15 03:02:58,136 DEBUG [Thread-10] executor.ExecutorService(99): Starting executor service name=MASTER_CLOSE_REGION-testSSHWhenSplitRegionInProgress, corePoolSize=3, maxPoolSize=3 2015-06-15 03:02:58,136 DEBUG [Thread-10] executor.ExecutorService(99): Starting executor service name=MASTER_SERVER_OPERATIONS-testSSHWhenSplitRegionInProgress, corePoolSize=3, maxPoolSize=3 2015-06-15 03:02:58,144 DEBUG [Thread-10] executor.ExecutorService(99): Starting executor service name=MASTER_META_SERVER_OPERATIONS-testSSHWhenSplitRegionInProgress, corePoolSize=3, maxPoolSize=3 2015-06-15 03:02:58,144 DEBUG [Thread-10] zookeeper.ZKAssign(497): mockedServer-0x14df52ac8120005, quorum=localhost:59952, baseZNode=/hbase Deleting any existing unassigned nodes 2015-06-15 03:02:58,149 DEBUG [Thread-10] executor.ExecutorService(99): Starting executor service name=MASTER_OPEN_REGION-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:02:58,150 DEBUG [Thread-10] executor.ExecutorService(99): Starting executor service name=MASTER_CLOSE_REGION-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:02:58,152 DEBUG [Thread-10] executor.ExecutorService(99): Starting executor service name=MASTER_SERVER_OPERATIONS-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:02:58,153 DEBUG [Thread-10] executor.ExecutorService(99): Starting executor service name=MASTER_META_SERVER_OPERATIONS-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:02:58,154 INFO [Thread-10] balancer.StochasticLoadBalancer(135): loading config 2015-06-15 03:02:58,162 INFO [Thread-10] master.RegionStates(1112): Transition null to {cae5ea3b1868354349837ee2dd0d4d89 state=OPEN, ts=1434337378162, server=example.org,1234,5678} 2015-06-15 03:02:58,162 DEBUG [Thread-10] master.RegionStates(460): Onlined cae5ea3b1868354349837ee2dd0d4d89 on example.org,1234,5678 {ENCODED => cae5ea3b1868354349837ee2dd0d4d89, NAME => 't,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89.', STARTKEY => '', ENDKEY => ''} 2015-06-15 03:02:58,162 INFO [Thread-10] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=OPEN, ts=1434337378162, server=example.org,1234,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=SPLITTING, ts=1434337378162, server=example.org,1234,5678} 2015-06-15 03:02:58,163 INFO [Thread-10] zookeeper.ZKTableStateManager(101): Moving table t state from ENABLED to ENABLED 2015-06-15 03:02:58,183 INFO [Thread-10] handler.ServerShutdownHandler(217): Skipping log splitting for example.org,1234,5678 2015-06-15 03:02:58,184 DEBUG [Thread-10] master.RegionStates(634): Offline splitting/merging region {cae5ea3b1868354349837ee2dd0d4d89 state=SPLITTING, ts=1434337378162, server=example.org,1234,5678} 2015-06-15 03:02:58,186 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer-0x14df52ac8120005, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/cae5ea3b1868354349837ee2dd0d4d89 2015-06-15 03:02:58,186 INFO [Thread-10] master.RegionStates(651): Transitioning {cae5ea3b1868354349837ee2dd0d4d89 state=SPLITTING, ts=1434337378162, server=example.org,1234,5678} will be handled by SSH for example.org,1234,5678 2015-06-15 03:02:58,186 INFO [Thread-10] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=SPLITTING, ts=1434337378162, server=example.org,1234,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=OFFLINE, ts=1434337378186, server=example.org,1234,5678} 2015-06-15 03:02:58,186 INFO [Thread-10] handler.ServerShutdownHandler(238): Reassigning 1 region(s) that example.org,1234,5678 was carrying (and 0 regions(s) that were opening on this server) 2015-06-15 03:02:58,186 DEBUG [Thread-10] master.DeadServer(118): Finished processing example.org,1234,5678 2015-06-15 03:02:58,193 INFO [Thread-10] handler.ServerShutdownHandler(347): Finished processing of shutdown of example.org,1234,5678 2015-06-15 03:02:58,194 DEBUG [Thread-10] zookeeper.ZKAssign(497): mockedServer-0x14df52ac8120005, quorum=localhost:59952, baseZNode=/hbase Deleting any existing unassigned nodes 2015-06-15 03:02:58,195 DEBUG [Thread-10] executor.ExecutorService(99): Starting executor service name=MASTER_OPEN_REGION-testSSHWhenSplitRegionInProgress, corePoolSize=3, maxPoolSize=3 2015-06-15 03:02:58,195 DEBUG [Thread-10] executor.ExecutorService(99): Starting executor service name=MASTER_CLOSE_REGION-testSSHWhenSplitRegionInProgress, corePoolSize=3, maxPoolSize=3 2015-06-15 03:02:58,195 DEBUG [Thread-10] executor.ExecutorService(99): Starting executor service name=MASTER_SERVER_OPERATIONS-testSSHWhenSplitRegionInProgress, corePoolSize=3, maxPoolSize=3 2015-06-15 03:02:58,195 DEBUG [Thread-10] executor.ExecutorService(99): Starting executor service name=MASTER_META_SERVER_OPERATIONS-testSSHWhenSplitRegionInProgress, corePoolSize=3, maxPoolSize=3 2015-06-15 03:02:58,195 DEBUG [Thread-10] zookeeper.ZKAssign(497): mockedServer-0x14df52ac8120005, quorum=localhost:59952, baseZNode=/hbase Deleting any existing unassigned nodes 2015-06-15 03:02:58,199 DEBUG [Thread-10] executor.ExecutorService(99): Starting executor service name=MASTER_OPEN_REGION-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:02:58,199 DEBUG [Thread-10] executor.ExecutorService(99): Starting executor service name=MASTER_CLOSE_REGION-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:02:58,199 DEBUG [Thread-10] executor.ExecutorService(99): Starting executor service name=MASTER_SERVER_OPERATIONS-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:02:58,200 DEBUG [Thread-10] executor.ExecutorService(99): Starting executor service name=MASTER_META_SERVER_OPERATIONS-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:02:58,203 INFO [Thread-10] balancer.StochasticLoadBalancer(135): loading config 2015-06-15 03:02:58,205 INFO [Thread-10] master.RegionStates(1112): Transition null to {cae5ea3b1868354349837ee2dd0d4d89 state=OPEN, ts=1434337378205, server=example.org,1234,5678} 2015-06-15 03:02:58,206 DEBUG [Thread-10] master.RegionStates(460): Onlined cae5ea3b1868354349837ee2dd0d4d89 on example.org,1234,5678 {ENCODED => cae5ea3b1868354349837ee2dd0d4d89, NAME => 't,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89.', STARTKEY => '', ENDKEY => ''} 2015-06-15 03:02:58,206 INFO [Thread-10] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=OPEN, ts=1434337378205, server=example.org,1234,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=SPLITTING, ts=1434337378206, server=example.org,1234,5678} 2015-06-15 03:02:58,206 INFO [Thread-10] zookeeper.ZKTableStateManager(101): Moving table t state from ENABLED to ENABLED 2015-06-15 03:02:58,229 INFO [Thread-10] handler.ServerShutdownHandler(217): Skipping log splitting for example.org,1234,5678 2015-06-15 03:02:58,229 DEBUG [Thread-10] master.RegionStates(634): Offline splitting/merging region {cae5ea3b1868354349837ee2dd0d4d89 state=SPLITTING, ts=1434337378206, server=example.org,1234,5678} 2015-06-15 03:02:58,232 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer-0x14df52ac8120005, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/cae5ea3b1868354349837ee2dd0d4d89 2015-06-15 03:02:58,232 INFO [Thread-10] master.RegionStates(651): Transitioning {cae5ea3b1868354349837ee2dd0d4d89 state=SPLITTING, ts=1434337378206, server=example.org,1234,5678} will be handled by SSH for example.org,1234,5678 2015-06-15 03:02:58,232 INFO [Thread-10] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=SPLITTING, ts=1434337378206, server=example.org,1234,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=OFFLINE, ts=1434337378232, server=example.org,1234,5678} 2015-06-15 03:02:58,232 INFO [Thread-10] handler.ServerShutdownHandler(238): Reassigning 1 region(s) that example.org,1234,5678 was carrying (and 0 regions(s) that were opening on this server) 2015-06-15 03:02:58,235 ERROR [main-EventThread] zookeeper.ClientCnxn$EventThread(524): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@fe43069 rejected from java.util.concurrent.ThreadPoolExecutor@4e2645a5[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110) at org.apache.hadoop.hbase.master.AssignmentManager.zkEventWorkersSubmit(AssignmentManager.java:1291) at org.apache.hadoop.hbase.master.AssignmentManager.nodeDeleted(AssignmentManager.java:1322) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:532) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) 2015-06-15 03:02:58,326 INFO [Thread-10] master.AssignmentManager(1611): Assigning 1 region(s) to example.org,0,5678 2015-06-15 03:02:58,326 WARN [Thread-10] master.RegionStates(856): Couldn't reach online server example.org,1234,5678 2015-06-15 03:02:58,330 WARN [Thread-10] master.RegionStates(856): Couldn't reach online server example.org,1234,5678 2015-06-15 03:02:58,330 INFO [Thread-10] master.AssignmentManager(1633): Skip assigning t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89., it's host example.org,1234,5678 is dead but not processed yet 2015-06-15 03:03:18,416 DEBUG [Thread-10] master.AssignmentManager(1804): some user regions are still in transition: [{ENCODED => cae5ea3b1868354349837ee2dd0d4d89, NAME => 't,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89.', STARTKEY => '', ENDKEY => ''}] 2015-06-15 03:03:18,417 DEBUG [Thread-10] master.AssignmentManager(1806): Bulk assigning done for example.org,0,5678 2015-06-15 03:03:18,417 DEBUG [Thread-10] master.DeadServer(118): Finished processing example.org,1234,5678 2015-06-15 03:03:18,417 INFO [Thread-10] handler.ServerShutdownHandler(347): Finished processing of shutdown of example.org,1234,5678 2015-06-15 03:03:18,418 DEBUG [Thread-10] zookeeper.ZKAssign(497): mockedServer-0x14df52ac8120005, quorum=localhost:59952, baseZNode=/hbase Deleting any existing unassigned nodes 2015-06-15 03:03:18,420 DEBUG [main] zookeeper.ZKAssign(497): mockedServer-0x14df52ac8120005, quorum=localhost:59952, baseZNode=/hbase Deleting any existing unassigned nodes 2015-06-15 03:03:18,450 INFO [main] hbase.ResourceChecker(171): after: master.TestAssignmentManager#testSSHWhenSplitRegionInProgress Thread=11 (was 11), OpenFileDescriptor=203 (was 205), MaxFileDescriptor=32768 (was 32768), SystemLoadAverage=636 (was 592) - SystemLoadAverage LEAK? -, ProcessCount=102 (was 102), AvailableMemoryMB=4996 (was 5057), ConnectionCount=1 (was 1) 2015-06-15 03:03:18,501 INFO [main] hbase.ResourceChecker(147): before: master.TestAssignmentManager#testRegionPlanIsUpdatedWhenRegionFailsToOpen Thread=11, OpenFileDescriptor=205, MaxFileDescriptor=32768, SystemLoadAverage=636, ProcessCount=102, AvailableMemoryMB=4991, ConnectionCount=1 2015-06-15 03:03:18,502 INFO [main] zookeeper.RecoverableZooKeeper(121): Process identifier=mockedServer connecting to ZooKeeper ensemble=localhost:59952 2015-06-15 03:03:18,519 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer0x0, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-15 03:03:18,519 DEBUG [main] zookeeper.RecoverableZooKeeper(595): Node /hbase already exists 2015-06-15 03:03:18,521 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(592): mockedServer-0x14df52ac8120006 connected 2015-06-15 03:03:18,554 DEBUG [Thread-11] executor.ExecutorService(99): Starting executor service name=MASTER_OPEN_REGION-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:03:18,561 DEBUG [Thread-11] executor.ExecutorService(99): Starting executor service name=MASTER_CLOSE_REGION-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:03:18,561 DEBUG [Thread-11] executor.ExecutorService(99): Starting executor service name=MASTER_SERVER_OPERATIONS-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:03:18,561 DEBUG [Thread-11] executor.ExecutorService(99): Starting executor service name=MASTER_META_SERVER_OPERATIONS-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:03:18,584 DEBUG [Thread-11] zookeeper.ZKAssign(147): mockedServer-0x14df52ac8120006, quorum=localhost:59952, baseZNode=/hbase Creating unassigned node cae5ea3b1868354349837ee2dd0d4d89 in OFFLINE state 2015-06-15 03:03:18,588 DEBUG [Thread-11] zookeeper.ZKAssign(805): mockedServer-0x14df52ac8120006, quorum=localhost:59952, baseZNode=/hbase Transitioning cae5ea3b1868354349837ee2dd0d4d89 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_FAILED_OPEN 2015-06-15 03:03:18,591 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer-0x14df52ac8120006, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/cae5ea3b1868354349837ee2dd0d4d89 2015-06-15 03:03:18,591 DEBUG [Thread-11] zookeeper.ZKAssign(880): mockedServer-0x14df52ac8120006, quorum=localhost:59952, baseZNode=/hbase Transitioned node cae5ea3b1868354349837ee2dd0d4d89 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_FAILED_OPEN 2015-06-15 03:03:18,591 INFO [Thread-11] master.RegionStates(1112): Transition null to {cae5ea3b1868354349837ee2dd0d4d89 state=OPENING, ts=1434337398591, server=example.org,1234,5678} 2015-06-15 03:03:18,605 DEBUG [AM.ZK.Worker-pool23-t1] master.AssignmentManager(954): Handling RS_ZK_REGION_FAILED_OPEN, server=example.org,1234,5678, region=cae5ea3b1868354349837ee2dd0d4d89, current_state={cae5ea3b1868354349837ee2dd0d4d89 state=OPENING, ts=1434337398591, server=example.org,1234,5678} 2015-06-15 03:03:18,606 INFO [AM.ZK.Worker-pool23-t1] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=OPENING, ts=1434337398591, server=example.org,1234,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=CLOSED, ts=1434337398606, server=example.org,1234,5678} 2015-06-15 03:03:18,606 DEBUG [AM.ZK.Worker-pool23-t1] master.AssignmentManager(2381): Found an existing plan for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. destination server is example.org,1234,5678 accepted as a dest server = false 2015-06-15 03:03:18,606 DEBUG [AM.ZK.Worker-pool23-t1] master.AssignmentManager(2413): No previous transition plan found (or ignoring an existing plan) for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89.; generated random plan=hri=t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89., src=, dest=example.org,0,5678; 1 (online=2) available servers, forceNewPlan=true 2015-06-15 03:03:18,610 DEBUG [AM.ZK.Worker-pool23-t1] handler.ClosedRegionHandler(93): Handling CLOSED event for cae5ea3b1868354349837ee2dd0d4d89 2015-06-15 03:03:18,613 DEBUG [AM.ZK.Worker-pool23-t1] master.AssignmentManager(2381): Found an existing plan for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. destination server is example.org,0,5678 accepted as a dest server = true 2015-06-15 03:03:18,613 DEBUG [AM.ZK.Worker-pool23-t1] master.AssignmentManager(2419): Using pre-existing plan for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89.; plan=hri=t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89., src=, dest=example.org,0,5678 2015-06-15 03:03:18,613 INFO [AM.ZK.Worker-pool23-t1] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=CLOSED, ts=1434337398612, server=example.org,1234,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=OFFLINE, ts=1434337398613, server=example.org,1234,5678} 2015-06-15 03:03:18,613 DEBUG [AM.ZK.Worker-pool23-t1] zookeeper.ZKAssign(205): mockedServer-0x14df52ac8120006, quorum=localhost:59952, baseZNode=/hbase Creating (or updating) unassigned node cae5ea3b1868354349837ee2dd0d4d89 with OFFLINE state 2015-06-15 03:03:18,619 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer-0x14df52ac8120006, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/cae5ea3b1868354349837ee2dd0d4d89 2015-06-15 03:03:18,626 INFO [AM.ZK.Worker-pool23-t1] master.AssignmentManager(2122): Assigning t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. to example.org,0,5678 2015-06-15 03:03:18,626 INFO [AM.ZK.Worker-pool23-t1] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=OFFLINE, ts=1434337398613, server=example.org,1234,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=PENDING_OPEN, ts=1434337398626, server=example.org,0,5678} 2015-06-15 03:03:18,627 DEBUG [main] zookeeper.ZKAssign(497): mockedServer-0x14df52ac8120006, quorum=localhost:59952, baseZNode=/hbase Deleting any existing unassigned nodes 2015-06-15 03:03:18,630 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer-0x14df52ac8120006, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/cae5ea3b1868354349837ee2dd0d4d89 2015-06-15 03:03:18,679 INFO [main] hbase.ResourceChecker(171): after: master.TestAssignmentManager#testRegionPlanIsUpdatedWhenRegionFailsToOpen Thread=11 (was 11), OpenFileDescriptor=203 (was 205), MaxFileDescriptor=32768 (was 32768), SystemLoadAverage=636 (was 636), ProcessCount=102 (was 102), AvailableMemoryMB=4973 (was 4991), ConnectionCount=1 (was 1) 2015-06-15 03:03:18,704 INFO [main] hbase.ResourceChecker(147): before: master.TestAssignmentManager#testBalance Thread=11, OpenFileDescriptor=203, MaxFileDescriptor=32768, SystemLoadAverage=636, ProcessCount=102, AvailableMemoryMB=4971, ConnectionCount=1 2015-06-15 03:03:18,705 INFO [main] zookeeper.RecoverableZooKeeper(121): Process identifier=mockedServer connecting to ZooKeeper ensemble=localhost:59952 2015-06-15 03:03:18,726 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer0x0, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-15 03:03:18,727 DEBUG [main] zookeeper.RecoverableZooKeeper(595): Node /hbase already exists 2015-06-15 03:03:18,731 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(592): mockedServer-0x14df52ac8120007 connected 2015-06-15 03:03:18,743 DEBUG [Thread-12] executor.ExecutorService(99): Starting executor service name=MASTER_OPEN_REGION-testBalanceExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:03:18,744 DEBUG [Thread-12] executor.ExecutorService(99): Starting executor service name=MASTER_CLOSE_REGION-testBalanceExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:03:18,744 DEBUG [Thread-12] executor.ExecutorService(99): Starting executor service name=MASTER_SERVER_OPERATIONS-testBalanceExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:03:18,744 DEBUG [Thread-12] executor.ExecutorService(99): Starting executor service name=MASTER_META_SERVER_OPERATIONS-testBalanceExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:03:18,749 INFO [Thread-12] master.RegionStates(1112): Transition null to {cae5ea3b1868354349837ee2dd0d4d89 state=OPEN, ts=1434337398749, server=example.org,1234,5678} 2015-06-15 03:03:18,751 DEBUG [Thread-12] master.RegionStates(460): Onlined cae5ea3b1868354349837ee2dd0d4d89 on example.org,1234,5678 {ENCODED => cae5ea3b1868354349837ee2dd0d4d89, NAME => 't,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89.', STARTKEY => '', ENDKEY => ''} 2015-06-15 03:03:18,751 DEBUG [Thread-12] master.AssignmentManager(2470): Starting unassign of t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. (offlining), current state: {cae5ea3b1868354349837ee2dd0d4d89 state=OPEN, ts=1434337398749, server=example.org,1234,5678} 2015-06-15 03:03:18,752 DEBUG [Thread-12] zookeeper.ZKAssign(527): mockedServer-0x14df52ac8120007, quorum=localhost:59952, baseZNode=/hbase Creating unassigned node cae5ea3b1868354349837ee2dd0d4d89 in a CLOSING state 2015-06-15 03:03:18,756 INFO [Thread-12] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=OPEN, ts=1434337398749, server=example.org,1234,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=PENDING_CLOSE, ts=1434337398756, server=example.org,1234,5678} 2015-06-15 03:03:18,756 WARN [Thread-12] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:18,756 WARN [Thread-12] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:18,761 WARN [Thread-12] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:18,761 WARN [Thread-12] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:18,762 WARN [Thread-12] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:18,762 WARN [Thread-12] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:18,819 WARN [Thread-12] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:18,819 WARN [Thread-12] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:18,820 WARN [Thread-12] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:18,820 WARN [Thread-12] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:18,820 WARN [Thread-12] master.RegionStates(1103): Failed to open/close cae5ea3b1868354349837ee2dd0d4d89 on example.org,1234,5678, set to FAILED_CLOSE 2015-06-15 03:03:18,820 INFO [Thread-12] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=PENDING_CLOSE, ts=1434337398756, server=example.org,1234,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=FAILED_CLOSE, ts=1434337398820, server=example.org,1234,5678} 2015-06-15 03:03:18,821 INFO [Thread-12] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=FAILED_CLOSE, ts=1434337398820, server=example.org,1234,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=PENDING_CLOSE, ts=1434337398820, server=example.org,1234,5678} 2015-06-15 03:03:18,821 DEBUG [Thread-12] zookeeper.ZKAssign(805): mockedServer-0x14df52ac8120007, quorum=localhost:59952, baseZNode=/hbase Transitioning cae5ea3b1868354349837ee2dd0d4d89 from M_ZK_REGION_CLOSING to RS_ZK_REGION_CLOSED 2015-06-15 03:03:18,845 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer-0x14df52ac8120007, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/cae5ea3b1868354349837ee2dd0d4d89 2015-06-15 03:03:18,845 DEBUG [Thread-12] zookeeper.ZKAssign(880): mockedServer-0x14df52ac8120007, quorum=localhost:59952, baseZNode=/hbase Transitioned node cae5ea3b1868354349837ee2dd0d4d89 from M_ZK_REGION_CLOSING to RS_ZK_REGION_CLOSED 2015-06-15 03:03:18,860 DEBUG [AM.ZK.Worker-pool25-t1] master.AssignmentManager(954): Handling RS_ZK_REGION_CLOSED, server=example.org,1234,5678, region=cae5ea3b1868354349837ee2dd0d4d89, current_state={cae5ea3b1868354349837ee2dd0d4d89 state=PENDING_CLOSE, ts=1434337398820, server=example.org,1234,5678} 2015-06-15 03:03:18,861 DEBUG [AM.ZK.Worker-pool25-t1] handler.ClosedRegionHandler(93): Handling CLOSED event for cae5ea3b1868354349837ee2dd0d4d89 2015-06-15 03:03:18,862 INFO [AM.ZK.Worker-pool25-t1] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=PENDING_CLOSE, ts=1434337398820, server=example.org,1234,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=CLOSED, ts=1434337398862, server=example.org,1234,5678} 2015-06-15 03:03:18,863 DEBUG [AM.ZK.Worker-pool25-t1] master.AssignmentManager(2381): Found an existing plan for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. destination server is example.org,0,5678 accepted as a dest server = true 2015-06-15 03:03:18,863 DEBUG [AM.ZK.Worker-pool25-t1] master.AssignmentManager(2419): Using pre-existing plan for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89.; plan=hri=t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89., src=example.org,1234,5678, dest=example.org,0,5678 2015-06-15 03:03:18,863 INFO [AM.ZK.Worker-pool25-t1] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=CLOSED, ts=1434337398862, server=example.org,1234,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=OFFLINE, ts=1434337398863, server=example.org,1234,5678} 2015-06-15 03:03:18,863 DEBUG [AM.ZK.Worker-pool25-t1] zookeeper.ZKAssign(205): mockedServer-0x14df52ac8120007, quorum=localhost:59952, baseZNode=/hbase Creating (or updating) unassigned node cae5ea3b1868354349837ee2dd0d4d89 with OFFLINE state 2015-06-15 03:03:18,871 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer-0x14df52ac8120007, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/cae5ea3b1868354349837ee2dd0d4d89 2015-06-15 03:03:18,872 INFO [AM.ZK.Worker-pool25-t1] master.AssignmentManager(2122): Assigning t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. to example.org,0,5678 2015-06-15 03:03:18,872 INFO [AM.ZK.Worker-pool25-t1] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=OFFLINE, ts=1434337398863, server=example.org,1234,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=PENDING_OPEN, ts=1434337398872, server=example.org,0,5678} 2015-06-15 03:03:18,874 DEBUG [Thread-12] zookeeper.ZKAssign(805): mockedServer-0x14df52ac8120007, quorum=localhost:59952, baseZNode=/hbase Transitioning cae5ea3b1868354349837ee2dd0d4d89 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-06-15 03:03:18,877 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer-0x14df52ac8120007, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/cae5ea3b1868354349837ee2dd0d4d89 2015-06-15 03:03:18,877 DEBUG [Thread-12] zookeeper.ZKAssign(880): mockedServer-0x14df52ac8120007, quorum=localhost:59952, baseZNode=/hbase Transitioned node cae5ea3b1868354349837ee2dd0d4d89 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-06-15 03:03:18,877 DEBUG [Thread-12] zookeeper.ZKAssign(805): mockedServer-0x14df52ac8120007, quorum=localhost:59952, baseZNode=/hbase Transitioning cae5ea3b1868354349837ee2dd0d4d89 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2015-06-15 03:03:18,880 DEBUG [Thread-12] zookeeper.ZKAssign(880): mockedServer-0x14df52ac8120007, quorum=localhost:59952, baseZNode=/hbase Transitioned node cae5ea3b1868354349837ee2dd0d4d89 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2015-06-15 03:03:18,888 DEBUG [AM.ZK.Worker-pool25-t2] master.AssignmentManager(954): Handling RS_ZK_REGION_OPENED, server=example.org,0,5678, region=cae5ea3b1868354349837ee2dd0d4d89, current_state={cae5ea3b1868354349837ee2dd0d4d89 state=PENDING_OPEN, ts=1434337398872, server=example.org,0,5678} 2015-06-15 03:03:18,888 INFO [AM.ZK.Worker-pool25-t2] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=PENDING_OPEN, ts=1434337398872, server=example.org,0,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=OPEN, ts=1434337398888, server=example.org,0,5678} 2015-06-15 03:03:18,888 DEBUG [AM.ZK.Worker-pool25-t2] coordination.ZkOpenRegionCoordination(350): Handling OPENED of cae5ea3b1868354349837ee2dd0d4d89 from master,1,1; deleting unassigned node 2015-06-15 03:03:18,893 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer-0x14df52ac8120007, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/cae5ea3b1868354349837ee2dd0d4d89 2015-06-15 03:03:18,893 DEBUG [AM.ZK.Worker-pool25-t2] zookeeper.ZKAssign(479): mockedServer-0x14df52ac8120007, quorum=localhost:59952, baseZNode=/hbase Deleted unassigned node cae5ea3b1868354349837ee2dd0d4d89 in expected state RS_ZK_REGION_OPENED 2015-06-15 03:03:18,910 DEBUG [AM.ZK.Worker-pool25-t4] master.AssignmentManager$4(1343): Znode t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. deleted, state: {cae5ea3b1868354349837ee2dd0d4d89 state=OPEN, ts=1434337398888, server=example.org,0,5678} 2015-06-15 03:03:18,911 DEBUG [AM.ZK.Worker-pool25-t4] master.RegionStates(460): Onlined cae5ea3b1868354349837ee2dd0d4d89 on example.org,0,5678 {ENCODED => cae5ea3b1868354349837ee2dd0d4d89, NAME => 't,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89.', STARTKEY => '', ENDKEY => ''} 2015-06-15 03:03:18,911 INFO [AM.ZK.Worker-pool25-t4] master.RegionStates(472): Offlined cae5ea3b1868354349837ee2dd0d4d89 from example.org,1234,5678 2015-06-15 03:03:18,912 DEBUG [Thread-12] zookeeper.ZKAssign(497): mockedServer-0x14df52ac8120007, quorum=localhost:59952, baseZNode=/hbase Deleting any existing unassigned nodes 2015-06-15 03:03:18,914 DEBUG [main] zookeeper.ZKAssign(497): mockedServer-0x14df52ac8120007, quorum=localhost:59952, baseZNode=/hbase Deleting any existing unassigned nodes 2015-06-15 03:03:18,958 INFO [main] hbase.ResourceChecker(171): after: master.TestAssignmentManager#testBalance Thread=11 (was 11), OpenFileDescriptor=203 (was 203), MaxFileDescriptor=32768 (was 32768), SystemLoadAverage=636 (was 636), ProcessCount=102 (was 102), AvailableMemoryMB=4952 (was 4971), ConnectionCount=1 (was 1) 2015-06-15 03:03:18,987 INFO [main] hbase.ResourceChecker(147): before: master.TestAssignmentManager#testForceAssignMergingRegion Thread=11, OpenFileDescriptor=205, MaxFileDescriptor=32768, SystemLoadAverage=636, ProcessCount=102, AvailableMemoryMB=4947, ConnectionCount=1 2015-06-15 03:03:18,987 INFO [main] zookeeper.RecoverableZooKeeper(121): Process identifier=mockedServer connecting to ZooKeeper ensemble=localhost:59952 2015-06-15 03:03:19,009 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer0x0, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-15 03:03:19,013 DEBUG [main] zookeeper.RecoverableZooKeeper(595): Node /hbase already exists 2015-06-15 03:03:19,017 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(592): mockedServer-0x14df52ac8120008 connected 2015-06-15 03:03:19,045 INFO [Thread-13] master.RegionStates(1112): Transition null to {1588230740 state=MERGING, ts=1434337399045, server=null} 2015-06-15 03:03:19,045 DEBUG [Thread-13] master.AssignmentManager(1953): Force region state offline {1588230740 state=MERGING, ts=1434337399045, server=null} 2015-06-15 03:03:19,045 ERROR [Thread-13] master.AssignmentManager(2001): Trying to assign region {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, which is {1588230740 state=MERGING, ts=1434337399045, server=null} 2015-06-15 03:03:19,045 DEBUG [main] zookeeper.ZKAssign(497): mockedServer-0x14df52ac8120008, quorum=localhost:59952, baseZNode=/hbase Deleting any existing unassigned nodes 2015-06-15 03:03:19,087 INFO [main] hbase.ResourceChecker(171): after: master.TestAssignmentManager#testForceAssignMergingRegion Thread=11 (was 11), OpenFileDescriptor=203 (was 205), MaxFileDescriptor=32768 (was 32768), SystemLoadAverage=636 (was 636), ProcessCount=102 (was 102), AvailableMemoryMB=4947 (was 4947), ConnectionCount=1 (was 1) 2015-06-15 03:03:19,125 INFO [main] hbase.ResourceChecker(147): before: master.TestAssignmentManager#testMasterRestartWhenTableInEnabling Thread=11, OpenFileDescriptor=205, MaxFileDescriptor=32768, SystemLoadAverage=636, ProcessCount=102, AvailableMemoryMB=4944, ConnectionCount=1 2015-06-15 03:03:19,126 INFO [main] zookeeper.RecoverableZooKeeper(121): Process identifier=mockedServer connecting to ZooKeeper ensemble=localhost:59952 2015-06-15 03:03:19,156 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer0x0, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-15 03:03:19,157 DEBUG [main] zookeeper.RecoverableZooKeeper(595): Node /hbase already exists 2015-06-15 03:03:19,179 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(592): mockedServer-0x14df52ac8120009 connected 2015-06-15 03:03:19,673 INFO [Thread-14] client.ConnectionUtils(103): master/ndimiduk-hbase-1-1-1RC0-1.openstacklocal/172.22.113.182:0 server-side HConnection retries=350 2015-06-15 03:03:20,056 INFO [Thread-14] ipc.SimpleRpcScheduler(128): Using deadline as user call queue, count=1 2015-06-15 03:03:20,070 INFO [Thread-14] ipc.RpcServer$Listener(595): master/ndimiduk-hbase-1-1-1RC0-1.openstacklocal/172.22.113.182:0: started 10 reader(s). 2015-06-15 03:03:20,087 DEBUG [Thread-14] fs.HFileSystem(221): The file system is not a DistributedFileSystem. Skipping on block location reordering 2015-06-15 03:03:20,217 INFO [Thread-14] zookeeper.RecoverableZooKeeper(121): Process identifier=master:60005 connecting to ZooKeeper ensemble=localhost:59952 2015-06-15 03:03:20,224 DEBUG [Thread-14-EventThread] zookeeper.ZooKeeperWatcher(508): master:600050x0, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-15 03:03:20,226 DEBUG [Thread-14] zookeeper.RecoverableZooKeeper(595): Node /hbase already exists 2015-06-15 03:03:20,226 DEBUG [Thread-14-EventThread] zookeeper.ZooKeeperWatcher(592): master:60005-0x14df52ac812000a connected 2015-06-15 03:03:20,241 DEBUG [Thread-14] zookeeper.ZKUtil(492): master:60005-0x14df52ac812000a, quorum=localhost:59952, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2015-06-15 03:03:20,242 DEBUG [Thread-14] zookeeper.ZKUtil(492): master:60005-0x14df52ac812000a, quorum=localhost:59952, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2015-06-15 03:03:20,249 INFO [RpcServer.responder] ipc.RpcServer$Responder(914): RpcServer.responder: starting 2015-06-15 03:03:20,250 DEBUG [Thread-14] ipc.RpcExecutor(114): B.default Start Handler index=0 queue=0 2015-06-15 03:03:20,251 DEBUG [Thread-14] ipc.RpcExecutor(114): B.default Start Handler index=1 queue=0 2015-06-15 03:03:20,252 DEBUG [Thread-14] ipc.RpcExecutor(114): B.default Start Handler index=2 queue=0 2015-06-15 03:03:20,252 DEBUG [Thread-14] ipc.RpcExecutor(114): B.default Start Handler index=3 queue=0 2015-06-15 03:03:20,253 DEBUG [Thread-14] ipc.RpcExecutor(114): B.default Start Handler index=4 queue=0 2015-06-15 03:03:20,253 DEBUG [Thread-14] ipc.RpcExecutor(114): Priority Start Handler index=0 queue=0 2015-06-15 03:03:20,254 DEBUG [Thread-14] ipc.RpcExecutor(114): Priority Start Handler index=1 queue=1 2015-06-15 03:03:20,254 DEBUG [Thread-14] ipc.RpcExecutor(114): Priority Start Handler index=2 queue=0 2015-06-15 03:03:20,260 DEBUG [Thread-14] ipc.RpcExecutor(114): Priority Start Handler index=3 queue=1 2015-06-15 03:03:20,262 DEBUG [Thread-14] ipc.RpcExecutor(114): Priority Start Handler index=4 queue=0 2015-06-15 03:03:20,262 INFO [RpcServer.listener,port=60005] ipc.RpcServer$Listener(726): RpcServer.listener,port=60005: starting 2015-06-15 03:03:20,267 DEBUG [Thread-14] ipc.RpcExecutor(114): Priority Start Handler index=5 queue=1 2015-06-15 03:03:20,273 DEBUG [Thread-14] ipc.RpcExecutor(114): Priority Start Handler index=6 queue=0 2015-06-15 03:03:20,275 DEBUG [Thread-14] ipc.RpcExecutor(114): Priority Start Handler index=7 queue=1 2015-06-15 03:03:20,286 DEBUG [Thread-14] ipc.RpcExecutor(114): Priority Start Handler index=8 queue=0 2015-06-15 03:03:20,289 DEBUG [Thread-14] ipc.RpcExecutor(114): Priority Start Handler index=9 queue=1 2015-06-15 03:03:20,298 DEBUG [Thread-14] ipc.RpcExecutor(114): Priority Start Handler index=10 queue=0 2015-06-15 03:03:20,298 DEBUG [Thread-14] ipc.RpcExecutor(114): Priority Start Handler index=11 queue=1 2015-06-15 03:03:20,301 DEBUG [Thread-14] ipc.RpcExecutor(114): Priority Start Handler index=12 queue=0 2015-06-15 03:03:20,308 DEBUG [Thread-14] ipc.RpcExecutor(114): Priority Start Handler index=13 queue=1 2015-06-15 03:03:20,319 DEBUG [Thread-14] ipc.RpcExecutor(114): Priority Start Handler index=14 queue=0 2015-06-15 03:03:20,326 DEBUG [Thread-14] ipc.RpcExecutor(114): Priority Start Handler index=15 queue=1 2015-06-15 03:03:20,332 DEBUG [Thread-14] ipc.RpcExecutor(114): Priority Start Handler index=16 queue=0 2015-06-15 03:03:20,345 DEBUG [Thread-14] ipc.RpcExecutor(114): Priority Start Handler index=17 queue=1 2015-06-15 03:03:20,352 DEBUG [Thread-14] ipc.RpcExecutor(114): Priority Start Handler index=18 queue=0 2015-06-15 03:03:20,354 DEBUG [Thread-14] ipc.RpcExecutor(114): Priority Start Handler index=19 queue=1 2015-06-15 03:03:20,358 DEBUG [Thread-14] ipc.RpcExecutor(114): Replication Start Handler index=0 queue=0 2015-06-15 03:03:20,365 DEBUG [Thread-14] ipc.RpcExecutor(114): Replication Start Handler index=1 queue=0 2015-06-15 03:03:20,372 DEBUG [Thread-14] ipc.RpcExecutor(114): Replication Start Handler index=2 queue=0 2015-06-15 03:03:20,377 INFO [Thread-14] master.HMaster(368): hbase.rootdir=file:/tmp/hbase-root/hbase, hbase.cluster.distributed=false 2015-06-15 03:03:20,401 INFO [Thread-14] master.HMaster(1616): Adding backup master ZNode /hbase/backup-masters/ndimiduk-hbase-1-1-1RC0-1.openstacklocal,60005,1434337400084 2015-06-15 03:03:20,409 DEBUG [Thread-14] zookeeper.ZKUtil(490): master:60005-0x14df52ac812000a, quorum=localhost:59952, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/ndimiduk-hbase-1-1-1RC0-1.openstacklocal,60005,1434337400084 2015-06-15 03:03:20,432 DEBUG [Thread-14-EventThread] zookeeper.ZooKeeperWatcher(508): master:60005-0x14df52ac812000a, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2015-06-15 03:03:20,432 DEBUG [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] zookeeper.ZKUtil(490): master:60005-0x14df52ac812000a, quorum=localhost:59952, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2015-06-15 03:03:20,436 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ActiveMasterManager(170): Deleting ZNode for /hbase/backup-masters/ndimiduk-hbase-1-1-1RC0-1.openstacklocal,60005,1434337400084 from backup master directory 2015-06-15 03:03:20,444 DEBUG [Thread-14-EventThread] zookeeper.ZKUtil(490): master:60005-0x14df52ac812000a, quorum=localhost:59952, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2015-06-15 03:03:20,445 WARN [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] hbase.ZNodeClearer(58): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2015-06-15 03:03:20,445 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ActiveMasterManager(179): Registered Active Master=ndimiduk-hbase-1-1-1RC0-1.openstacklocal,60005,1434337400084 2015-06-15 03:03:20,447 DEBUG [Thread-14] executor.ExecutorService(99): Starting executor service name=MASTER_OPEN_REGION-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:03:20,448 DEBUG [Thread-14] executor.ExecutorService(99): Starting executor service name=MASTER_CLOSE_REGION-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:03:20,448 DEBUG [Thread-14] executor.ExecutorService(99): Starting executor service name=MASTER_SERVER_OPERATIONS-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:03:20,448 DEBUG [Thread-14] executor.ExecutorService(99): Starting executor service name=MASTER_META_SERVER_OPERATIONS-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:03:20,444 DEBUG [Thread-14-EventThread] master.ActiveMasterManager(126): A master is now available 2015-06-15 03:03:20,454 DEBUG [Thread-14-EventThread] zookeeper.ZooKeeperWatcher(508): master:60005-0x14df52ac812000a, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/ndimiduk-hbase-1-1-1RC0-1.openstacklocal,60005,1434337400084 2015-06-15 03:03:20,471 INFO [Thread-14] zookeeper.ZKTableStateManager(101): Moving table t state from ENABLED to ENABLING 2015-06-15 03:03:20,511 INFO [Thread-14] handler.EnableTableHandler(150): Attempting to enable the table t 2015-06-15 03:03:20,511 INFO [Thread-14] zookeeper.ZKTableStateManager(101): Moving table t state from ENABLING to ENABLING 2015-06-15 03:03:20,532 INFO [Thread-14] handler.EnableTableHandler(224): Table 't' has 1 regions, of which 1 are offline. 2015-06-15 03:03:20,532 INFO [Thread-14] handler.EnableTableHandler(230): Bulk assigning 1 region(s) across 1 server(s), retainAssignment=true 2015-06-15 03:03:20,540 DEBUG [Thread-14] master.GeneralBulkAssigner(152): Timeout-on-RIT=91000 2015-06-15 03:03:20,552 INFO [ndimiduk-hbase-1-1-1RC0-1.openstacklocal,60005,1434337400084-GeneralBulkAssigner-0] master.RegionStates(1112): Transition null to {cae5ea3b1868354349837ee2dd0d4d89 state=OPEN, ts=1434337400552, server=example.org,1234,5678} 2015-06-15 03:03:20,553 DEBUG [ndimiduk-hbase-1-1-1RC0-1.openstacklocal,60005,1434337400084-GeneralBulkAssigner-0] master.RegionStates(460): Onlined cae5ea3b1868354349837ee2dd0d4d89 on example.org,1234,5678 {ENCODED => cae5ea3b1868354349837ee2dd0d4d89, NAME => 't,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89.', STARTKEY => '', ENDKEY => ''} 2015-06-15 03:03:20,553 DEBUG [Thread-14] master.GeneralBulkAssigner(128): bulk assigning total 1 regions to 1 servers, took 13ms, successfully 2015-06-15 03:03:20,553 INFO [Thread-14] zookeeper.ZKTableStateManager(101): Moving table t state from ENABLING to ENABLED 2015-06-15 03:03:20,558 INFO [Thread-14] handler.EnableTableHandler(254): Table 't' was successfully enabled. Status: done=true 2015-06-15 03:03:20,559 INFO [Thread-14] zookeeper.ZKTableStateManager(101): Moving table t state from ENABLED to ENABLED 2015-06-15 03:03:20,561 DEBUG [Thread-14] zookeeper.ZKAssign(497): mockedServer-0x14df52ac8120009, quorum=localhost:59952, baseZNode=/hbase Deleting any existing unassigned nodes 2015-06-15 03:03:20,562 DEBUG [main] zookeeper.ZKAssign(497): mockedServer-0x14df52ac8120009, quorum=localhost:59952, baseZNode=/hbase Deleting any existing unassigned nodes 2015-06-15 03:03:20,581 DEBUG [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] util.FSTableDescriptors(727): Current tableInfoPath = file:/tmp/hbase-root/hbase/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2015-06-15 03:03:20,594 INFO [main] hbase.ResourceChecker(171): after: master.TestAssignmentManager#testMasterRestartWhenTableInEnabling Thread=56 (was 11) Potentially hanging thread: RpcServer.reader=8,bindAddress=ndimiduk-hbase-1-1-1RC0-1.openstacklocal,port=60005 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:627) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:614) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PriorityRpcServer.handler=12,queue=0,port=60005 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PriorityRpcServer.handler=10,queue=0,port=60005 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RpcServer.reader=9,bindAddress=ndimiduk-hbase-1-1-1RC0-1.openstacklocal,port=60005 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:627) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:614) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: B.defaultRpcServer.handler=4,queue=0,port=60005 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) org.apache.hadoop.hbase.util.BoundedPriorityBlockingQueue.take(BoundedPriorityBlockingQueue.java:214) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: ReplicationRpcServer.handler=1,queue=0,port=60005 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PriorityRpcServer.handler=19,queue=1,port=60005 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PriorityRpcServer.handler=7,queue=1,port=60005 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PriorityRpcServer.handler=2,queue=0,port=60005 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PriorityRpcServer.handler=18,queue=0,port=60005 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PriorityRpcServer.handler=17,queue=1,port=60005 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PriorityRpcServer.handler=9,queue=1,port=60005 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager org.apache.hadoop.hbase.util.FSUtils.metaRegionExists(FSUtils.java:1027) org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:477) org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:146) org.apache.hadoop.hbase.master.MasterFileSystem.(MasterFileSystem.java:126) org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:649) org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:182) org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1646) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PriorityRpcServer.handler=13,queue=1,port=60005 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RpcServer.reader=1,bindAddress=ndimiduk-hbase-1-1-1RC0-1.openstacklocal,port=60005 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:627) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:614) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: Thread-14-SendThread(localhost:59952) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) Potentially hanging thread: PriorityRpcServer.handler=1,queue=1,port=60005 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: B.defaultRpcServer.handler=1,queue=0,port=60005 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) org.apache.hadoop.hbase.util.BoundedPriorityBlockingQueue.take(BoundedPriorityBlockingQueue.java:214) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PriorityRpcServer.handler=5,queue=1,port=60005 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RpcServer.reader=0,bindAddress=ndimiduk-hbase-1-1-1RC0-1.openstacklocal,port=60005 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:627) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:614) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RpcServer.listener,port=60005 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) org.apache.hadoop.hbase.ipc.RpcServer$Listener.run(RpcServer.java:730) Potentially hanging thread: PriorityRpcServer.handler=16,queue=0,port=60005 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RpcServer.reader=4,bindAddress=ndimiduk-hbase-1-1-1RC0-1.openstacklocal,port=60005 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:627) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:614) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RpcServer.reader=2,bindAddress=ndimiduk-hbase-1-1-1RC0-1.openstacklocal,port=60005 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:627) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:614) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: ReplicationRpcServer.handler=0,queue=0,port=60005 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PriorityRpcServer.handler=14,queue=0,port=60005 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PriorityRpcServer.handler=15,queue=1,port=60005 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RpcServer.reader=6,bindAddress=ndimiduk-hbase-1-1-1RC0-1.openstacklocal,port=60005 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:627) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:614) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: B.defaultRpcServer.handler=3,queue=0,port=60005 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) org.apache.hadoop.hbase.util.BoundedPriorityBlockingQueue.take(BoundedPriorityBlockingQueue.java:214) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: B.defaultRpcServer.handler=0,queue=0,port=60005 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) org.apache.hadoop.hbase.util.BoundedPriorityBlockingQueue.take(BoundedPriorityBlockingQueue.java:214) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PriorityRpcServer.handler=6,queue=0,port=60005 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: ReplicationRpcServer.handler=2,queue=0,port=60005 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: main-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) Potentially hanging thread: Thread-46 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.master.HMaster$InitializationMonitor.run(HMaster.java:218) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PriorityRpcServer.handler=0,queue=0,port=60005 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RpcServer.reader=7,bindAddress=ndimiduk-hbase-1-1-1RC0-1.openstacklocal,port=60005 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:627) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:614) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RpcServer.responder sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.hadoop.hbase.ipc.RpcServer$Responder.doRunLoop(RpcServer.java:969) org.apache.hadoop.hbase.ipc.RpcServer$Responder.run(RpcServer.java:916) Potentially hanging thread: PriorityRpcServer.handler=8,queue=0,port=60005 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PriorityRpcServer.handler=4,queue=0,port=60005 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PriorityRpcServer.handler=3,queue=1,port=60005 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RpcServer.reader=5,bindAddress=ndimiduk-hbase-1-1-1RC0-1.openstacklocal,port=60005 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:627) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:614) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RpcServer.reader=3,bindAddress=ndimiduk-hbase-1-1-1RC0-1.openstacklocal,port=60005 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:627) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:614) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PriorityRpcServer.handler=11,queue=1,port=60005 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: B.defaultRpcServer.handler=2,queue=0,port=60005 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) org.apache.hadoop.hbase.util.BoundedPriorityBlockingQueue.take(BoundedPriorityBlockingQueue.java:214) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: Thread-14-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) - Thread LEAK? -, OpenFileDescriptor=245 (was 205) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=32768 (was 32768), SystemLoadAverage=636 (was 636), ProcessCount=100 (was 102), AvailableMemoryMB=5076 (was 4944) - AvailableMemoryMB LEAK? -, ConnectionCount=2 (was 1) - ConnectionCount LEAK? - 2015-06-15 03:03:20,602 DEBUG [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] util.FSTableDescriptors(731): TableInfo already exists.. Skipping creation 2015-06-15 03:03:20,615 DEBUG [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] fs.HFileSystem(221): The file system is not a DistributedFileSystem. Skipping on block location reordering 2015-06-15 03:03:20,618 DEBUG [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] coordination.ZKSplitLogManagerCoordination(886): Distributed log replay=false 2015-06-15 03:03:20,621 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] coordination.ZKSplitLogManagerCoordination(599): Found 0 orphan tasks and 0 rescan nodes 2015-06-15 03:03:20,622 INFO [main] hbase.ResourceChecker(147): before: master.TestAssignmentManager#testClosingFailureDuringRecovery Thread=55, OpenFileDescriptor=247, MaxFileDescriptor=32768, SystemLoadAverage=636, ProcessCount=100, AvailableMemoryMB=5076, ConnectionCount=2 2015-06-15 03:03:20,629 INFO [main] zookeeper.RecoverableZooKeeper(121): Process identifier=mockedServer connecting to ZooKeeper ensemble=localhost:59952 2015-06-15 03:03:20,625 DEBUG [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] util.FSTableDescriptors(204): Fetching table descriptors from the filesystem. 2015-06-15 03:03:20,640 DEBUG [main] zookeeper.RecoverableZooKeeper(595): Node /hbase already exists 2015-06-15 03:03:20,638 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer0x0, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-15 03:03:20,644 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(592): mockedServer-0x14df52ac812000b connected 2015-06-15 03:03:20,649 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] zookeeper.RecoverableZooKeeper(121): Process identifier=hconnection-0x6cec66ea connecting to ZooKeeper ensemble=localhost:59952 2015-06-15 03:03:20,653 DEBUG [Thread-47] executor.ExecutorService(99): Starting executor service name=MASTER_OPEN_REGION-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:03:20,653 DEBUG [Thread-47] executor.ExecutorService(99): Starting executor service name=MASTER_CLOSE_REGION-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:03:20,656 DEBUG [Thread-47] executor.ExecutorService(99): Starting executor service name=MASTER_SERVER_OPERATIONS-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:03:20,657 DEBUG [Thread-47] executor.ExecutorService(99): Starting executor service name=MASTER_META_SERVER_OPERATIONS-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:03:20,679 DEBUG [Thread-47] zookeeper.ZKAssign(527): mockedServer-0x14df52ac812000b, quorum=localhost:59952, baseZNode=/hbase Creating unassigned node cae5ea3b1868354349837ee2dd0d4d89 in a CLOSING state 2015-06-15 03:03:20,681 DEBUG [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(508): hconnection-0x6cec66ea0x0, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-15 03:03:20,687 INFO [Thread-47] master.AssignmentManager(767): Processing cae5ea3b1868354349837ee2dd0d4d89 in state: M_ZK_REGION_CLOSING 2015-06-15 03:03:20,687 INFO [Thread-47] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=OFFLINE, ts=1434337400686, server=null} to {cae5ea3b1868354349837ee2dd0d4d89 state=CLOSING, ts=1434337400687, server=example.org,1234,5678} 2015-06-15 03:03:20,688 DEBUG [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(592): hconnection-0x6cec66ea-0x14df52ac812000c connected 2015-06-15 03:03:20,688 INFO [Thread-47] master.AssignmentManager(898): Processed region cae5ea3b1868354349837ee2dd0d4d89 in state M_ZK_REGION_CLOSING, on server: example.org,1234,5678 2015-06-15 03:03:20,689 DEBUG [main] zookeeper.ZKAssign(497): mockedServer-0x14df52ac812000b, quorum=localhost:59952, baseZNode=/hbase Deleting any existing unassigned nodes 2015-06-15 03:03:20,694 DEBUG [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] ipc.AbstractRpcClient(105): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@580416e4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2015-06-15 03:03:20,696 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer-0x14df52ac812000b, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/cae5ea3b1868354349837ee2dd0d4d89 2015-06-15 03:03:20,701 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:20,703 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:20,704 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:20,706 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:20,707 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:20,707 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:20,708 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:20,708 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:20,709 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:20,709 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:20,710 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.RegionStates(1103): Failed to open/close cae5ea3b1868354349837ee2dd0d4d89 on example.org,1234,5678, set to FAILED_CLOSE 2015-06-15 03:03:20,711 INFO [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=CLOSING, ts=1434337400687, server=example.org,1234,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=FAILED_CLOSE, ts=1434337400711, server=example.org,1234,5678} 2015-06-15 03:03:20,712 DEBUG [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] zookeeper.ZKUtil(492): master:60005-0x14df52ac812000a, quorum=localhost:59952, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2015-06-15 03:03:20,732 INFO [main] hbase.ResourceChecker(171): after: master.TestAssignmentManager#testClosingFailureDuringRecovery Thread=59 (was 55) Potentially hanging thread: MASTER_SERVER_OPERATIONS-mockedAMExecutor-0 org.apache.hadoop.hbase.master.AssignmentManager$1.process(AssignmentManager.java:817) org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) Potentially hanging thread: ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager-SendThread(localhost:59952) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) Potentially hanging thread: ndimiduk-hbase-1-1-1RC0-1.openstacklocal,60005,1434337400084_splitLogManager__ChoreService_1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) - Thread LEAK? -, OpenFileDescriptor=250 (was 247) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=32768 (was 32768), SystemLoadAverage=636 (was 636), ProcessCount=100 (was 100), AvailableMemoryMB=5071 (was 5076), ConnectionCount=2 (was 2) 2015-06-15 03:03:20,745 DEBUG [Thread-14-EventThread] zookeeper.ZooKeeperWatcher(508): master:60005-0x14df52ac812000a, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2015-06-15 03:03:20,747 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.HMaster(598): Server active/primary master=ndimiduk-hbase-1-1-1RC0-1.openstacklocal,60005,1434337400084, sessionid=0x14df52ac812000a, setting cluster-up flag (Was=false) 2015-06-15 03:03:20,783 INFO [main] hbase.ResourceChecker(147): before: master.TestAssignmentManager#testRegionInOpeningStateOnDeadRSWhileMasterFailover Thread=59, OpenFileDescriptor=252, MaxFileDescriptor=32768, SystemLoadAverage=636, ProcessCount=101, AvailableMemoryMB=5070, ConnectionCount=2 2015-06-15 03:03:20,784 INFO [main] zookeeper.RecoverableZooKeeper(121): Process identifier=mockedServer connecting to ZooKeeper ensemble=localhost:59952 2015-06-15 03:03:20,802 DEBUG [main] zookeeper.RecoverableZooKeeper(595): Node /hbase already exists 2015-06-15 03:03:20,809 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer0x0, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-15 03:03:20,812 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2015-06-15 03:03:20,814 DEBUG [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] procedure.ZKProcedureCoordinatorRpcs(238): Starting the controller for procedure member:ndimiduk-hbase-1-1-1RC0-1.openstacklocal,60005,1434337400084 2015-06-15 03:03:20,818 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(592): mockedServer-0x14df52ac812000d connected 2015-06-15 03:03:20,842 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2015-06-15 03:03:20,844 DEBUG [Thread-48] executor.ExecutorService(99): Starting executor service name=MASTER_OPEN_REGION-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:03:20,844 DEBUG [Thread-48] executor.ExecutorService(99): Starting executor service name=MASTER_CLOSE_REGION-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:03:20,844 DEBUG [Thread-48] executor.ExecutorService(99): Starting executor service name=MASTER_SERVER_OPERATIONS-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:03:20,845 DEBUG [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] procedure.ZKProcedureCoordinatorRpcs(238): Starting the controller for procedure member:ndimiduk-hbase-1-1-1RC0-1.openstacklocal,60005,1434337400084 2015-06-15 03:03:20,845 DEBUG [Thread-48] executor.ExecutorService(99): Starting executor service name=MASTER_META_SERVER_OPERATIONS-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:03:20,864 DEBUG [Thread-48] zookeeper.ZKAssign(147): mockedServer-0x14df52ac812000d, quorum=localhost:59952, baseZNode=/hbase Creating unassigned node cae5ea3b1868354349837ee2dd0d4d89 in OFFLINE state 2015-06-15 03:03:20,871 DEBUG [Thread-48] zookeeper.ZKAssign(805): mockedServer-0x14df52ac812000d, quorum=localhost:59952, baseZNode=/hbase Transitioning cae5ea3b1868354349837ee2dd0d4d89 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-06-15 03:03:20,880 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer-0x14df52ac812000d, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/cae5ea3b1868354349837ee2dd0d4d89 2015-06-15 03:03:20,881 DEBUG [Thread-48] zookeeper.ZKAssign(880): mockedServer-0x14df52ac812000d, quorum=localhost:59952, baseZNode=/hbase Transitioned node cae5ea3b1868354349837ee2dd0d4d89 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-06-15 03:03:20,882 DEBUG [Thread-48] master.RegionStates(532): Adding to processed servers example.org,1234,5678 2015-06-15 03:03:20,882 INFO [Thread-48] master.AssignmentManager(767): Processing cae5ea3b1868354349837ee2dd0d4d89 in state: RS_ZK_REGION_OPENING 2015-06-15 03:03:20,882 DEBUG [Thread-48] master.AssignmentManager(781): RIT cae5ea3b1868354349837ee2dd0d4d89 in state=RS_ZK_REGION_OPENING was on deadserver; forcing offline 2015-06-15 03:03:20,882 INFO [Thread-48] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=OFFLINE, ts=1434337400882, server=null} to {cae5ea3b1868354349837ee2dd0d4d89 state=OFFLINE, ts=1434337400882, server=example.org,1234,5678} 2015-06-15 03:03:20,883 INFO [Thread-48] zookeeper.ZKTableStateManager(101): Moving table t state from ENABLED to ENABLED 2015-06-15 03:03:20,941 INFO [Thread-48] handler.ServerShutdownHandler(217): Skipping log splitting for example.org,1234,5678 2015-06-15 03:03:20,941 INFO [Thread-48] master.RegionStates(661): Found region in {cae5ea3b1868354349837ee2dd0d4d89 state=OFFLINE, ts=1434337400882, server=example.org,1234,5678} to be reassigned by SSH for example.org,1234,5678 2015-06-15 03:03:20,944 INFO [Thread-48] handler.ServerShutdownHandler(238): Reassigning 1 region(s) that example.org,1234,5678 was carrying (and 1 regions(s) that were opening on this server) 2015-06-15 03:03:20,944 INFO [Thread-48] master.AssignmentManager(1611): Assigning 1 region(s) to example.org,1234,5678 2015-06-15 03:03:20,945 DEBUG [Thread-48] zookeeper.ZKAssign(174): mockedServer-0x14df52ac812000d, quorum=localhost:59952, baseZNode=/hbase Async create of unassigned node cae5ea3b1868354349837ee2dd0d4d89 with OFFLINE state 2015-06-15 03:03:20,947 DEBUG [main-EventThread] master.OfflineCallback(69): rs={cae5ea3b1868354349837ee2dd0d4d89 state=OFFLINE, ts=1434337400944, server=example.org,1234,5678}, server=example.org,1234,5678 2015-06-15 03:03:20,947 DEBUG [main-EventThread] master.OfflineCallback$ExistCallback(106): rs={cae5ea3b1868354349837ee2dd0d4d89 state=OFFLINE, ts=1434337400944, server=example.org,1234,5678}, server=example.org,1234,5678 2015-06-15 03:03:20,951 DEBUG [Thread-48] master.AssignmentManager(1664): example.org,1234,5678 unassigned znodes=1 of total=1; oldCounter=0 2015-06-15 03:03:20,951 INFO [Thread-48] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=OFFLINE, ts=1434337400945, server=example.org,1234,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=PENDING_OPEN, ts=1434337400951, server=example.org,1234,5678} 2015-06-15 03:03:20,974 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.MasterCoprocessorHost(82): System coprocessor loading is enabled 2015-06-15 03:03:20,975 DEBUG [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] executor.ExecutorService(99): Starting executor service name=MASTER_OPEN_REGION-ndimiduk-hbase-1-1-1RC0-1:60005, corePoolSize=5, maxPoolSize=5 2015-06-15 03:03:20,978 DEBUG [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] executor.ExecutorService(99): Starting executor service name=MASTER_CLOSE_REGION-ndimiduk-hbase-1-1-1RC0-1:60005, corePoolSize=5, maxPoolSize=5 2015-06-15 03:03:20,978 DEBUG [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] executor.ExecutorService(99): Starting executor service name=MASTER_SERVER_OPERATIONS-ndimiduk-hbase-1-1-1RC0-1:60005, corePoolSize=5, maxPoolSize=5 2015-06-15 03:03:20,979 DEBUG [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] executor.ExecutorService(99): Starting executor service name=MASTER_META_SERVER_OPERATIONS-ndimiduk-hbase-1-1-1RC0-1:60005, corePoolSize=5, maxPoolSize=5 2015-06-15 03:03:20,979 DEBUG [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] executor.ExecutorService(99): Starting executor service name=M_LOG_REPLAY_OPS-ndimiduk-hbase-1-1-1RC0-1:60005, corePoolSize=10, maxPoolSize=10 2015-06-15 03:03:20,979 DEBUG [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] executor.ExecutorService(99): Starting executor service name=MASTER_TABLE_OPERATIONS-ndimiduk-hbase-1-1-1RC0-1:60005, corePoolSize=1, maxPoolSize=1 2015-06-15 03:03:21,015 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] procedure2.ProcedureExecutor(406): Starting procedure executor threads=5 2015-06-15 03:03:21,017 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] wal.WALProcedureStore(214): Starting WAL Procedure Store lease recovery 2015-06-15 03:03:21,026 WARN [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] wal.WALProcedureStore(739): Remove uninitialized log: DeprecatedRawLocalFileStatus{path=file:/tmp/hbase-root/hbase/MasterProcWALs/state-00000000000000000002.log; isDirectory=false; length=0; replication=1; blocksize=33554432; modification_time=1434325506000; access_time=0; owner=; group=; permission=rw-rw-rw-; isSymlink=false} 2015-06-15 03:03:21,073 DEBUG [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] wal.WALProcedureStore(589): Roll new state log: 3 2015-06-15 03:03:21,073 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] wal.WALProcedureStore(238): Lease acquired for flushLogId: 3 2015-06-15 03:03:21,073 DEBUG [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] wal.WALProcedureStore(252): No state logs to replay. 2015-06-15 03:03:21,107 DEBUG [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2015-06-15 03:03:21,112 DEBUG [ProcedureExecutorTimeoutThread] procedure2.ProcedureExecutor$CompletedProcedureCleaner(152): No completed procedures to cleanup. 2015-06-15 03:03:21,113 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] zookeeper.RecoverableZooKeeper(121): Process identifier=replicationLogCleaner connecting to ZooKeeper ensemble=localhost:59952 2015-06-15 03:03:21,134 DEBUG [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(508): replicationLogCleaner0x0, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-15 03:03:21,137 DEBUG [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(592): replicationLogCleaner-0x14df52ac812000e connected 2015-06-15 03:03:21,149 DEBUG [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2015-06-15 03:03:21,163 DEBUG [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotLogCleaner 2015-06-15 03:03:21,168 DEBUG [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2015-06-15 03:03:21,173 DEBUG [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2015-06-15 03:03:21,174 DEBUG [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2015-06-15 03:03:21,174 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 0 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:03:22,678 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 1503 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:03:24,187 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 3013 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:03:25,693 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 4519 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:03:27,198 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 6024 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:03:28,703 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 7529 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:03:30,249 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 9075 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:03:31,758 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 10584 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:03:33,263 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 12089 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:03:34,771 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 13597 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:03:36,282 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 15108 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:03:37,791 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 16617 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:03:39,295 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 18121 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:03:40,801 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 19627 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:03:41,017 DEBUG [Thread-48] master.AssignmentManager(1804): some user regions are still in transition: [{ENCODED => cae5ea3b1868354349837ee2dd0d4d89, NAME => 't,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89.', STARTKEY => '', ENDKEY => ''}] 2015-06-15 03:03:41,017 DEBUG [Thread-48] master.AssignmentManager(1806): Bulk assigning done for example.org,1234,5678 2015-06-15 03:03:41,018 DEBUG [Thread-48] master.DeadServer(118): Finished processing example.org,1234,5678 2015-06-15 03:03:41,018 INFO [Thread-48] handler.ServerShutdownHandler(347): Finished processing of shutdown of example.org,1234,5678 2015-06-15 03:03:41,018 DEBUG [main] zookeeper.ZKAssign(497): mockedServer-0x14df52ac812000d, quorum=localhost:59952, baseZNode=/hbase Deleting any existing unassigned nodes 2015-06-15 03:03:41,023 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer-0x14df52ac812000d, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/cae5ea3b1868354349837ee2dd0d4d89 2015-06-15 03:03:41,023 ERROR [main-EventThread] zookeeper.ClientCnxn$EventThread(524): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1c9374d8 rejected from java.util.concurrent.ThreadPoolExecutor@14dc1f90[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110) at org.apache.hadoop.hbase.master.AssignmentManager.zkEventWorkersSubmit(AssignmentManager.java:1291) at org.apache.hadoop.hbase.master.AssignmentManager.nodeDeleted(AssignmentManager.java:1322) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:532) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) 2015-06-15 03:03:41,044 INFO [main] hbase.ResourceChecker(171): after: master.TestAssignmentManager#testRegionInOpeningStateOnDeadRSWhileMasterFailover Thread=70 (was 59) Potentially hanging thread: ProcedureExecutorThread-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) org.apache.hadoop.hbase.master.procedure.MasterProcedureQueue.poll(MasterProcedureQueue.java:117) org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:620) org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$200(ProcedureExecutor.java:70) org.apache.hadoop.hbase.procedure2.ProcedureExecutor$1.run(ProcedureExecutor.java:413) Potentially hanging thread: ProcedureExecutorThread-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) org.apache.hadoop.hbase.master.procedure.MasterProcedureQueue.poll(MasterProcedureQueue.java:117) org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:620) org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$200(ProcedureExecutor.java:70) org.apache.hadoop.hbase.procedure2.ProcedureExecutor$1.run(ProcedureExecutor.java:413) Potentially hanging thread: snapshot-hfile-cleaner-cache-refresher java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: WALProcedureStoreSyncThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.syncLoop(WALProcedureStore.java:461) org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.access$100(WALProcedureStore.java:61) org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore$1.run(WALProcedureStore.java:142) Potentially hanging thread: ndimiduk-hbase-1-1-1RC0-1.openstacklocal,60005,1434337400084_ChoreService_1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: ProcedureExecutorTimeoutThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) org.apache.hadoop.hbase.procedure2.util.TimeoutBlockingQueue.poll(TimeoutBlockingQueue.java:106) org.apache.hadoop.hbase.procedure2.ProcedureExecutor.timeoutLoop(ProcedureExecutor.java:700) org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$300(ProcedureExecutor.java:70) org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:422) Potentially hanging thread: ProcedureExecutorThread-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) org.apache.hadoop.hbase.master.procedure.MasterProcedureQueue.poll(MasterProcedureQueue.java:117) org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:620) org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$200(ProcedureExecutor.java:70) org.apache.hadoop.hbase.procedure2.ProcedureExecutor$1.run(ProcedureExecutor.java:413) Potentially hanging thread: ProcedureExecutorThread-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) org.apache.hadoop.hbase.master.procedure.MasterProcedureQueue.poll(MasterProcedureQueue.java:117) org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:620) org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$200(ProcedureExecutor.java:70) org.apache.hadoop.hbase.procedure2.ProcedureExecutor$1.run(ProcedureExecutor.java:413) Potentially hanging thread: snapshot-log-cleaner-cache-refresher java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) - Thread LEAK? -, OpenFileDescriptor=256 (was 252) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=32768 (was 32768), SystemLoadAverage=604 (was 636), ProcessCount=100 (was 101), AvailableMemoryMB=5050 (was 5070), ConnectionCount=2 (was 2) 2015-06-15 03:03:41,057 INFO [main] hbase.ResourceChecker(147): before: master.TestAssignmentManager#testProcessDeadServersAndRegionsInTransitionShouldNotFailWithNPE Thread=70, OpenFileDescriptor=258, MaxFileDescriptor=32768, SystemLoadAverage=604, ProcessCount=100, AvailableMemoryMB=5049, ConnectionCount=2 2015-06-15 03:03:41,058 INFO [main] zookeeper.RecoverableZooKeeper(121): Process identifier=mockedServer connecting to ZooKeeper ensemble=localhost:59952 2015-06-15 03:03:41,096 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer0x0, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-15 03:03:41,098 DEBUG [main] zookeeper.RecoverableZooKeeper(595): Node /hbase already exists 2015-06-15 03:03:41,098 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(592): mockedServer-0x14df52ac812000f connected 2015-06-15 03:03:41,197 DEBUG [Thread-53] executor.ExecutorService(99): Starting executor service name=MASTER_OPEN_REGION-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:03:41,197 DEBUG [Thread-53] executor.ExecutorService(99): Starting executor service name=MASTER_CLOSE_REGION-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:03:41,198 DEBUG [Thread-53] executor.ExecutorService(99): Starting executor service name=MASTER_SERVER_OPERATIONS-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:03:41,198 DEBUG [Thread-53] executor.ExecutorService(99): Starting executor service name=MASTER_META_SERVER_OPERATIONS-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:03:41,364 INFO [Thread-53] zookeeper.RecoverableZooKeeper(121): Process identifier=unittest connecting to ZooKeeper ensemble=localhost:21818 2015-06-15 03:03:41,371 DEBUG [Thread-53] zookeeper.ZooKeeperWatcher(664): unittest0x0, quorum=localhost:21818, baseZNode=/hbase Received InterruptedException, doing nothing here java.lang.InterruptedException at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenNoWatch(ZKUtil.java:635) at org.apache.hadoop.hbase.master.AssignmentManager.processDeadServersAndRegionsInTransition(AssignmentManager.java:517) at org.apache.hadoop.hbase.master.TestAssignmentManager.testProcessDeadServersAndRegionsInTransitionShouldNotFailWithNPE(TestAssignmentManager.java:779) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) 2015-06-15 03:03:41,382 DEBUG [main] zookeeper.ZKAssign(497): mockedServer-0x14df52ac812000f, quorum=localhost:59952, baseZNode=/hbase Deleting any existing unassigned nodes 2015-06-15 03:03:41,454 INFO [main] hbase.ResourceChecker(171): after: master.TestAssignmentManager#testProcessDeadServersAndRegionsInTransitionShouldNotFailWithNPE Thread=72 (was 70) Potentially hanging thread: Thread-53-SendThread(localhost:21818) java.lang.Thread.sleep(Native Method) org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:230) org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1185) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1110) Potentially hanging thread: Thread-53-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) - Thread LEAK? -, OpenFileDescriptor=260 (was 258) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=32768 (was 32768), SystemLoadAverage=604 (was 604), ProcessCount=100 (was 100), AvailableMemoryMB=5044 (was 5049), ConnectionCount=2 (was 2) 2015-06-15 03:03:41,514 INFO [main] hbase.ResourceChecker(147): before: master.TestAssignmentManager#testUnassignWithSplitAtSameTime Thread=72, OpenFileDescriptor=262, MaxFileDescriptor=32768, SystemLoadAverage=620, ProcessCount=100, AvailableMemoryMB=5043, ConnectionCount=2 2015-06-15 03:03:41,515 INFO [main] zookeeper.RecoverableZooKeeper(121): Process identifier=mockedServer connecting to ZooKeeper ensemble=localhost:59952 2015-06-15 03:03:41,549 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer0x0, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-15 03:03:41,550 DEBUG [main] zookeeper.RecoverableZooKeeper(595): Node /hbase already exists 2015-06-15 03:03:41,563 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(592): mockedServer-0x14df52ac8120010 connected 2015-06-15 03:03:41,567 INFO [Thread-54] master.RegionStates(1112): Transition null to {1588230740 state=OPEN, ts=1434337421567, server=example.org,1234,5678} 2015-06-15 03:03:41,567 DEBUG [Thread-54] master.RegionStates(460): Onlined 1588230740 on example.org,1234,5678 {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2015-06-15 03:03:41,567 DEBUG [Thread-54] master.AssignmentManager(2470): Starting unassign of hbase:meta,,1.1588230740 (offlining), current state: {1588230740 state=OPEN, ts=1434337421567, server=example.org,1234,5678} 2015-06-15 03:03:41,568 DEBUG [Thread-54] zookeeper.ZKAssign(527): mockedServer-0x14df52ac8120010, quorum=localhost:59952, baseZNode=/hbase Creating unassigned node 1588230740 in a CLOSING state 2015-06-15 03:03:41,570 INFO [Thread-54] master.RegionStates(1112): Transition {1588230740 state=OPEN, ts=1434337421567, server=example.org,1234,5678} to {1588230740 state=PENDING_CLOSE, ts=1434337421570, server=example.org,1234,5678} 2015-06-15 03:03:41,571 WARN [Thread-54] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for hbase:meta,,1.1588230740 2015-06-15 03:03:41,572 WARN [Thread-54] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for hbase:meta,,1.1588230740 2015-06-15 03:03:41,572 WARN [Thread-54] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for hbase:meta,,1.1588230740 2015-06-15 03:03:41,577 WARN [Thread-54] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for hbase:meta,,1.1588230740 2015-06-15 03:03:41,579 WARN [Thread-54] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for hbase:meta,,1.1588230740 2015-06-15 03:03:41,580 WARN [Thread-54] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for hbase:meta,,1.1588230740 2015-06-15 03:03:41,580 WARN [Thread-54] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for hbase:meta,,1.1588230740 2015-06-15 03:03:41,580 WARN [Thread-54] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for hbase:meta,,1.1588230740 2015-06-15 03:03:41,581 WARN [Thread-54] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for hbase:meta,,1.1588230740 2015-06-15 03:03:41,581 WARN [Thread-54] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for hbase:meta,,1.1588230740 2015-06-15 03:03:41,581 WARN [Thread-54] master.RegionStates(1103): Failed to open/close 1588230740 on example.org,1234,5678, set to FAILED_CLOSE 2015-06-15 03:03:41,581 INFO [Thread-54] master.RegionStates(1112): Transition {1588230740 state=PENDING_CLOSE, ts=1434337421570, server=example.org,1234,5678} to {1588230740 state=FAILED_CLOSE, ts=1434337421581, server=example.org,1234,5678} 2015-06-15 03:03:41,591 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer-0x14df52ac8120010, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/1588230740 2015-06-15 03:03:41,593 DEBUG [Thread-54] zookeeper.ZKAssign(479): mockedServer-0x14df52ac8120010, quorum=localhost:59952, baseZNode=/hbase Deleted unassigned node 1588230740 in expected state M_ZK_REGION_CLOSING 2015-06-15 03:03:41,597 DEBUG [Thread-54] zookeeper.ZKUtil(490): mockedServer-0x14df52ac8120010, quorum=localhost:59952, baseZNode=/hbase Set watcher on existing znode=/hbase/region-in-transition/1588230740 2015-06-15 03:03:41,608 DEBUG [Thread-54] zookeeper.ZKAssign(805): mockedServer-0x14df52ac8120010, quorum=localhost:59952, baseZNode=/hbase Transitioning 1588230740 from RS_ZK_REGION_SPLITTING to RS_ZK_REGION_SPLITTING 2015-06-15 03:03:41,612 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer-0x14df52ac8120010, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/1588230740 2015-06-15 03:03:41,612 DEBUG [Thread-54] zookeeper.ZKAssign(880): mockedServer-0x14df52ac8120010, quorum=localhost:59952, baseZNode=/hbase Transitioned node 1588230740 from RS_ZK_REGION_SPLITTING to RS_ZK_REGION_SPLITTING 2015-06-15 03:03:41,612 INFO [Thread-54] master.RegionStates(1112): Transition {1588230740 state=FAILED_CLOSE, ts=1434337421581, server=example.org,1234,5678} to {1588230740 state=OPEN, ts=1434337421612, server=example.org,1234,5678} 2015-06-15 03:03:41,613 DEBUG [Thread-54] master.AssignmentManager(2470): Starting unassign of hbase:meta,,1.1588230740 (offlining), current state: {1588230740 state=OPEN, ts=1434337421612, server=example.org,1234,5678} 2015-06-15 03:03:41,613 DEBUG [Thread-54] zookeeper.ZKAssign(527): mockedServer-0x14df52ac8120010, quorum=localhost:59952, baseZNode=/hbase Creating unassigned node 1588230740 in a CLOSING state 2015-06-15 03:03:41,619 DEBUG [Thread-54] zookeeper.RecoverableZooKeeper(595): Node /hbase/region-in-transition/1588230740 already exists 2015-06-15 03:03:41,620 DEBUG [Thread-54] master.AssignmentManager(2522): /hbase/region-in-transition/1588230740 is SPLIT or SPLITTING or MERGED or MERGING; skipping unassign because region no longer exists -- its split or merge 2015-06-15 03:03:41,620 DEBUG [Thread-54] zookeeper.ZKAssign(805): mockedServer-0x14df52ac8120010, quorum=localhost:59952, baseZNode=/hbase Transitioning 1588230740 from RS_ZK_REGION_SPLITTING to RS_ZK_REGION_SPLITTING 2015-06-15 03:03:41,623 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer-0x14df52ac8120010, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/1588230740 2015-06-15 03:03:41,624 DEBUG [Thread-54] zookeeper.ZKAssign(880): mockedServer-0x14df52ac8120010, quorum=localhost:59952, baseZNode=/hbase Transitioned node 1588230740 from RS_ZK_REGION_SPLITTING to RS_ZK_REGION_SPLITTING 2015-06-15 03:03:41,624 DEBUG [main] zookeeper.ZKAssign(497): mockedServer-0x14df52ac8120010, quorum=localhost:59952, baseZNode=/hbase Deleting any existing unassigned nodes 2015-06-15 03:03:41,648 INFO [main] hbase.ResourceChecker(171): after: master.TestAssignmentManager#testUnassignWithSplitAtSameTime Thread=72 (was 72), OpenFileDescriptor=260 (was 262), MaxFileDescriptor=32768 (was 32768), SystemLoadAverage=620 (was 620), ProcessCount=100 (was 100), AvailableMemoryMB=5041 (was 5043), ConnectionCount=2 (was 2) 2015-06-15 03:03:41,676 INFO [main] hbase.ResourceChecker(147): before: master.TestAssignmentManager#testDisablingTableRegionsAssignmentDuringCleanClusterStartup Thread=72, OpenFileDescriptor=262, MaxFileDescriptor=32768, SystemLoadAverage=620, ProcessCount=100, AvailableMemoryMB=5039, ConnectionCount=2 2015-06-15 03:03:41,677 INFO [main] zookeeper.RecoverableZooKeeper(121): Process identifier=mockedServer connecting to ZooKeeper ensemble=localhost:59952 2015-06-15 03:03:41,688 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer0x0, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-15 03:03:41,689 DEBUG [main] zookeeper.RecoverableZooKeeper(595): Node /hbase already exists 2015-06-15 03:03:41,706 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(592): mockedServer-0x14df52ac8120011 connected 2015-06-15 03:03:41,717 INFO [Thread-55] client.ConnectionUtils(103): master/ndimiduk-hbase-1-1-1RC0-1.openstacklocal/172.22.113.182:0 server-side HConnection retries=3500 2015-06-15 03:03:41,718 INFO [Thread-55] ipc.SimpleRpcScheduler(128): Using deadline as user call queue, count=1 2015-06-15 03:03:41,821 INFO [Thread-55] ipc.RpcServer$Listener(595): master/ndimiduk-hbase-1-1-1RC0-1.openstacklocal/172.22.113.182:0: started 10 reader(s). 2015-06-15 03:03:41,823 DEBUG [Thread-55] fs.HFileSystem(221): The file system is not a DistributedFileSystem. Skipping on block location reordering 2015-06-15 03:03:41,824 INFO [Thread-55] zookeeper.RecoverableZooKeeper(121): Process identifier=master:55682 connecting to ZooKeeper ensemble=localhost:59952 2015-06-15 03:03:41,846 DEBUG [Thread-55-EventThread] zookeeper.ZooKeeperWatcher(508): master:556820x0, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-15 03:03:41,848 DEBUG [Thread-55-EventThread] zookeeper.ZooKeeperWatcher(592): master:55682-0x14df52ac8120012 connected 2015-06-15 03:03:41,848 DEBUG [Thread-55] zookeeper.RecoverableZooKeeper(595): Node /hbase already exists 2015-06-15 03:03:41,856 DEBUG [Thread-55] zookeeper.ZKUtil(490): master:55682-0x14df52ac8120012, quorum=localhost:59952, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2015-06-15 03:03:41,857 DEBUG [Thread-55] zookeeper.ZKUtil(490): master:55682-0x14df52ac8120012, quorum=localhost:59952, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2015-06-15 03:03:41,870 INFO [RpcServer.responder] ipc.RpcServer$Responder(914): RpcServer.responder: starting 2015-06-15 03:03:41,884 INFO [RpcServer.listener,port=55682] ipc.RpcServer$Listener(726): RpcServer.listener,port=55682: starting 2015-06-15 03:03:41,884 DEBUG [Thread-55] ipc.RpcExecutor(114): B.default Start Handler index=0 queue=0 2015-06-15 03:03:41,888 DEBUG [Thread-55] ipc.RpcExecutor(114): B.default Start Handler index=1 queue=0 2015-06-15 03:03:41,888 DEBUG [Thread-55] ipc.RpcExecutor(114): B.default Start Handler index=2 queue=0 2015-06-15 03:03:41,891 DEBUG [Thread-55] ipc.RpcExecutor(114): B.default Start Handler index=3 queue=0 2015-06-15 03:03:41,892 DEBUG [Thread-55] ipc.RpcExecutor(114): B.default Start Handler index=4 queue=0 2015-06-15 03:03:41,892 DEBUG [Thread-55] ipc.RpcExecutor(114): Priority Start Handler index=0 queue=0 2015-06-15 03:03:41,893 DEBUG [Thread-55] ipc.RpcExecutor(114): Priority Start Handler index=1 queue=1 2015-06-15 03:03:41,895 DEBUG [Thread-55] ipc.RpcExecutor(114): Priority Start Handler index=2 queue=0 2015-06-15 03:03:41,896 DEBUG [Thread-55] ipc.RpcExecutor(114): Priority Start Handler index=3 queue=1 2015-06-15 03:03:41,902 DEBUG [Thread-55] ipc.RpcExecutor(114): Priority Start Handler index=4 queue=0 2015-06-15 03:03:41,911 DEBUG [Thread-55] ipc.RpcExecutor(114): Priority Start Handler index=5 queue=1 2015-06-15 03:03:41,921 DEBUG [Thread-55] ipc.RpcExecutor(114): Priority Start Handler index=6 queue=0 2015-06-15 03:03:41,924 DEBUG [Thread-55] ipc.RpcExecutor(114): Priority Start Handler index=7 queue=1 2015-06-15 03:03:41,946 DEBUG [Thread-55] ipc.RpcExecutor(114): Priority Start Handler index=8 queue=0 2015-06-15 03:03:41,957 DEBUG [Thread-55] ipc.RpcExecutor(114): Priority Start Handler index=9 queue=1 2015-06-15 03:03:41,965 DEBUG [Thread-55] ipc.RpcExecutor(114): Priority Start Handler index=10 queue=0 2015-06-15 03:03:41,981 DEBUG [Thread-55] ipc.RpcExecutor(114): Priority Start Handler index=11 queue=1 2015-06-15 03:03:42,007 DEBUG [Thread-55] ipc.RpcExecutor(114): Priority Start Handler index=12 queue=0 2015-06-15 03:03:42,019 DEBUG [Thread-55] ipc.RpcExecutor(114): Priority Start Handler index=13 queue=1 2015-06-15 03:03:42,025 DEBUG [Thread-55] ipc.RpcExecutor(114): Priority Start Handler index=14 queue=0 2015-06-15 03:03:42,033 DEBUG [Thread-55] ipc.RpcExecutor(114): Priority Start Handler index=15 queue=1 2015-06-15 03:03:42,042 DEBUG [Thread-55] ipc.RpcExecutor(114): Priority Start Handler index=16 queue=0 2015-06-15 03:03:42,062 DEBUG [Thread-55] ipc.RpcExecutor(114): Priority Start Handler index=17 queue=1 2015-06-15 03:03:42,078 DEBUG [Thread-55] ipc.RpcExecutor(114): Priority Start Handler index=18 queue=0 2015-06-15 03:03:42,082 DEBUG [Thread-55] ipc.RpcExecutor(114): Priority Start Handler index=19 queue=1 2015-06-15 03:03:42,096 DEBUG [Thread-55] ipc.RpcExecutor(114): Replication Start Handler index=0 queue=0 2015-06-15 03:03:42,104 DEBUG [Thread-55] ipc.RpcExecutor(114): Replication Start Handler index=1 queue=0 2015-06-15 03:03:42,125 DEBUG [Thread-55] ipc.RpcExecutor(114): Replication Start Handler index=2 queue=0 2015-06-15 03:03:42,126 INFO [Thread-55] master.HMaster(368): hbase.rootdir=file:/tmp/hbase-root/hbase, hbase.cluster.distributed=false 2015-06-15 03:03:42,132 INFO [Thread-55] master.HMaster(1616): Adding backup master ZNode /hbase/backup-masters/ndimiduk-hbase-1-1-1RC0-1.openstacklocal,55682,1434337421822 2015-06-15 03:03:42,135 DEBUG [Thread-55] zookeeper.ZKUtil(490): master:55682-0x14df52ac8120012, quorum=localhost:59952, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/ndimiduk-hbase-1-1-1RC0-1.openstacklocal,55682,1434337421822 2015-06-15 03:03:42,147 DEBUG [Thread-55] executor.ExecutorService(99): Starting executor service name=MASTER_OPEN_REGION-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:03:42,148 DEBUG [Thread-55] executor.ExecutorService(99): Starting executor service name=MASTER_CLOSE_REGION-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:03:42,149 DEBUG [Thread-55] executor.ExecutorService(99): Starting executor service name=MASTER_SERVER_OPERATIONS-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:03:42,149 DEBUG [Thread-55] executor.ExecutorService(99): Starting executor service name=MASTER_META_SERVER_OPERATIONS-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:03:42,151 DEBUG [ndimiduk-hbase-1-1-1RC0-1:55682.activeMasterManager] zookeeper.RecoverableZooKeeper(595): Node /hbase/master already exists 2015-06-15 03:03:42,152 DEBUG [ndimiduk-hbase-1-1-1RC0-1:55682.activeMasterManager] zookeeper.ZKUtil(490): master:55682-0x14df52ac8120012, quorum=localhost:59952, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2015-06-15 03:03:42,154 INFO [ndimiduk-hbase-1-1-1RC0-1:55682.activeMasterManager] master.ActiveMasterManager(216): Another master is the active master, ndimiduk-hbase-1-1-1RC0-1.openstacklocal,60005,1434337400084; waiting to become the next active master 2015-06-15 03:03:42,156 INFO [Thread-55] zookeeper.ZKTableStateManager(101): Moving table t state from ENABLED to DISABLING 2015-06-15 03:03:42,209 INFO [Thread-55] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=OPEN, ts=1434337422209, server=example.org,1234,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=OFFLINE, ts=1434337422209, server=example.org,1234,5678} 2015-06-15 03:03:42,210 INFO [Thread-55] master.RegionStates(600): Offlined cae5ea3b1868354349837ee2dd0d4d89 from example.org,1234,5678 2015-06-15 03:03:42,213 DEBUG [Thread-55] zookeeper.ZKAssign(497): master:55682-0x14df52ac8120012, quorum=localhost:59952, baseZNode=/hbase Deleting any existing unassigned nodes 2015-06-15 03:03:42,214 INFO [Thread-55] master.AssignmentManager(643): Clean cluster startup. Assigning user regions 2015-06-15 03:03:42,215 INFO [Thread-55] master.AssignmentManager(3021): The table t is in DISABLING state. Hence recovering by moving the table to DISABLED state. 2015-06-15 03:03:42,261 INFO [Thread-55] handler.DisableTableHandler(129): Attempting to disable table t 2015-06-15 03:03:42,261 INFO [Thread-55] zookeeper.ZKTableStateManager(101): Moving table t state from DISABLING to DISABLING 2015-06-15 03:03:42,264 INFO [Thread-55] zookeeper.ZKTableStateManager(101): Moving table t state from DISABLING to DISABLED 2015-06-15 03:03:42,273 INFO [Thread-55] handler.DisableTableHandler(191): Disabled table, t, is done=true 2015-06-15 03:03:42,274 INFO [Thread-55] master.AssignmentManager(499): Joined the cluster in 107ms, failover=false 2015-06-15 03:03:42,274 INFO [Thread-55] zookeeper.ZKTableStateManager(101): Moving table t state from DISABLED to ENABLED 2015-06-15 03:03:42,278 DEBUG [main] zookeeper.ZKAssign(497): mockedServer-0x14df52ac8120011, quorum=localhost:59952, baseZNode=/hbase Deleting any existing unassigned nodes 2015-06-15 03:03:42,315 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 21140 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:03:42,360 INFO [main] hbase.ResourceChecker(171): after: master.TestAssignmentManager#testDisablingTableRegionsAssignmentDuringCleanClusterStartup Thread=115 (was 72) Potentially hanging thread: ndimiduk-hbase-1-1-1RC0-1:55682.activeMasterManager java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.master.ActiveMasterManager.blockUntilBecomingActiveMaster(ActiveMasterManager.java:225) org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1645) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RpcServer.reader=4,bindAddress=ndimiduk-hbase-1-1-1RC0-1.openstacklocal,port=55682 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:627) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:614) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RpcServer.reader=0,bindAddress=ndimiduk-hbase-1-1-1RC0-1.openstacklocal,port=55682 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:627) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:614) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PriorityRpcServer.handler=17,queue=1,port=55682 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PriorityRpcServer.handler=4,queue=0,port=55682 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RpcServer.reader=8,bindAddress=ndimiduk-hbase-1-1-1RC0-1.openstacklocal,port=55682 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:627) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:614) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PriorityRpcServer.handler=15,queue=1,port=55682 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: B.defaultRpcServer.handler=3,queue=0,port=55682 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) org.apache.hadoop.hbase.util.BoundedPriorityBlockingQueue.take(BoundedPriorityBlockingQueue.java:214) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PriorityRpcServer.handler=5,queue=1,port=55682 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PriorityRpcServer.handler=8,queue=0,port=55682 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: ReplicationRpcServer.handler=2,queue=0,port=55682 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RpcServer.reader=5,bindAddress=ndimiduk-hbase-1-1-1RC0-1.openstacklocal,port=55682 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:627) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:614) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: Thread-55-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) Potentially hanging thread: PriorityRpcServer.handler=9,queue=1,port=55682 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RpcServer.reader=7,bindAddress=ndimiduk-hbase-1-1-1RC0-1.openstacklocal,port=55682 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:627) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:614) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: B.defaultRpcServer.handler=2,queue=0,port=55682 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) org.apache.hadoop.hbase.util.BoundedPriorityBlockingQueue.take(BoundedPriorityBlockingQueue.java:214) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RpcServer.reader=2,bindAddress=ndimiduk-hbase-1-1-1RC0-1.openstacklocal,port=55682 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:627) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:614) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PriorityRpcServer.handler=11,queue=1,port=55682 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PriorityRpcServer.handler=19,queue=1,port=55682 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PriorityRpcServer.handler=6,queue=0,port=55682 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PriorityRpcServer.handler=16,queue=0,port=55682 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: B.defaultRpcServer.handler=4,queue=0,port=55682 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) org.apache.hadoop.hbase.util.BoundedPriorityBlockingQueue.take(BoundedPriorityBlockingQueue.java:214) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: B.defaultRpcServer.handler=0,queue=0,port=55682 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) org.apache.hadoop.hbase.util.BoundedPriorityBlockingQueue.take(BoundedPriorityBlockingQueue.java:214) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RpcServer.reader=3,bindAddress=ndimiduk-hbase-1-1-1RC0-1.openstacklocal,port=55682 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:627) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:614) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PriorityRpcServer.handler=12,queue=0,port=55682 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PriorityRpcServer.handler=0,queue=0,port=55682 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RpcServer.reader=6,bindAddress=ndimiduk-hbase-1-1-1RC0-1.openstacklocal,port=55682 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:627) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:614) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PriorityRpcServer.handler=7,queue=1,port=55682 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: B.defaultRpcServer.handler=1,queue=0,port=55682 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) org.apache.hadoop.hbase.util.BoundedPriorityBlockingQueue.take(BoundedPriorityBlockingQueue.java:214) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PriorityRpcServer.handler=14,queue=0,port=55682 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RpcServer.listener,port=55682 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) org.apache.hadoop.hbase.ipc.RpcServer$Listener.run(RpcServer.java:730) Potentially hanging thread: ReplicationRpcServer.handler=0,queue=0,port=55682 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PriorityRpcServer.handler=3,queue=1,port=55682 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PriorityRpcServer.handler=10,queue=0,port=55682 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PriorityRpcServer.handler=18,queue=0,port=55682 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RpcServer.reader=9,bindAddress=ndimiduk-hbase-1-1-1RC0-1.openstacklocal,port=55682 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:627) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:614) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PriorityRpcServer.handler=2,queue=0,port=55682 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PriorityRpcServer.handler=1,queue=1,port=55682 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: ReplicationRpcServer.handler=1,queue=0,port=55682 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RpcServer.reader=1,bindAddress=ndimiduk-hbase-1-1-1RC0-1.openstacklocal,port=55682 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:627) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:614) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PriorityRpcServer.handler=13,queue=1,port=55682 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: Thread-55-SendThread(localhost:59952) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) - Thread LEAK? -, OpenFileDescriptor=302 (was 262) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=32768 (was 32768), SystemLoadAverage=620 (was 620), ProcessCount=100 (was 100), AvailableMemoryMB=5017 (was 5039), ConnectionCount=3 (was 2) - ConnectionCount LEAK? - 2015-06-15 03:03:42,418 INFO [main] hbase.ResourceChecker(147): before: master.TestAssignmentManager#testBalanceOnMasterFailoverScenarioWithOpenedNode Thread=115, OpenFileDescriptor=304, MaxFileDescriptor=32768, SystemLoadAverage=620, ProcessCount=100, AvailableMemoryMB=5017, ConnectionCount=3 2015-06-15 03:03:42,419 INFO [main] zookeeper.RecoverableZooKeeper(121): Process identifier=mockedServer connecting to ZooKeeper ensemble=localhost:59952 2015-06-15 03:03:42,459 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer0x0, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-15 03:03:42,463 DEBUG [main] zookeeper.RecoverableZooKeeper(595): Node /hbase already exists 2015-06-15 03:03:42,463 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(592): mockedServer-0x14df52ac8120013 connected 2015-06-15 03:03:42,524 DEBUG [Thread-87] executor.ExecutorService(99): Starting executor service name=MASTER_OPEN_REGION-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:03:42,525 DEBUG [Thread-87] executor.ExecutorService(99): Starting executor service name=MASTER_CLOSE_REGION-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:03:42,525 DEBUG [Thread-87] executor.ExecutorService(99): Starting executor service name=MASTER_SERVER_OPERATIONS-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:03:42,525 DEBUG [Thread-87] executor.ExecutorService(99): Starting executor service name=MASTER_META_SERVER_OPERATIONS-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:03:42,533 INFO [Thread-87] master.RegionStates(1112): Transition null to {cae5ea3b1868354349837ee2dd0d4d89 state=OPEN, ts=1434337422533, server=example.org,1234,5678} 2015-06-15 03:03:42,534 DEBUG [Thread-87] master.RegionStates(460): Onlined cae5ea3b1868354349837ee2dd0d4d89 on example.org,1234,5678 {ENCODED => cae5ea3b1868354349837ee2dd0d4d89, NAME => 't,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89.', STARTKEY => '', ENDKEY => ''} 2015-06-15 03:03:42,534 DEBUG [Thread-87] master.AssignmentManager(2470): Starting unassign of t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. (offlining), current state: {cae5ea3b1868354349837ee2dd0d4d89 state=OPEN, ts=1434337422533, server=example.org,1234,5678} 2015-06-15 03:03:42,534 DEBUG [Thread-87] zookeeper.ZKAssign(527): mockedServer-0x14df52ac8120013, quorum=localhost:59952, baseZNode=/hbase Creating unassigned node cae5ea3b1868354349837ee2dd0d4d89 in a CLOSING state 2015-06-15 03:03:42,541 DEBUG [Thread-55-EventThread] zookeeper.ZooKeeperWatcher(508): master:55682-0x14df52ac8120012, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-06-15 03:03:42,545 INFO [Thread-87] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=OPEN, ts=1434337422533, server=example.org,1234,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=PENDING_CLOSE, ts=1434337422545, server=example.org,1234,5678} 2015-06-15 03:03:42,545 WARN [Thread-87] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:42,546 WARN [Thread-87] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:42,546 WARN [Thread-87] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:42,546 WARN [Thread-87] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:42,547 WARN [Thread-87] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:42,547 WARN [Thread-87] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:42,547 WARN [Thread-87] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:42,548 WARN [Thread-87] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:42,548 WARN [Thread-87] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:42,548 WARN [Thread-87] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:42,548 WARN [Thread-87] master.RegionStates(1103): Failed to open/close cae5ea3b1868354349837ee2dd0d4d89 on example.org,1234,5678, set to FAILED_CLOSE 2015-06-15 03:03:42,549 INFO [Thread-87] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=PENDING_CLOSE, ts=1434337422545, server=example.org,1234,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=FAILED_CLOSE, ts=1434337422549, server=example.org,1234,5678} 2015-06-15 03:03:42,594 WARN [RunAmJoinCluster] master.RegionStates(339): Tried to create a state for a region already in RegionStates, used existing: {cae5ea3b1868354349837ee2dd0d4d89 state=FAILED_CLOSE, ts=1434337422549, server=example.org,1234,5678}, ignored new: OPEN 2015-06-15 03:03:42,595 DEBUG [RunAmJoinCluster] master.AssignmentManager(540): Found {ENCODED => cae5ea3b1868354349837ee2dd0d4d89, NAME => 't,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89.', STARTKEY => '', ENDKEY => ''} out on cluster 2015-06-15 03:03:42,595 INFO [RunAmJoinCluster] master.AssignmentManager(623): Found regions out on cluster or in RIT; presuming failover 2015-06-15 03:03:42,596 INFO [Thread-87] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=FAILED_CLOSE, ts=1434337422549, server=example.org,1234,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=PENDING_CLOSE, ts=1434337422596, server=example.org,1234,5678} 2015-06-15 03:03:42,597 DEBUG [Thread-87] zookeeper.ZKAssign(805): mockedServer-0x14df52ac8120013, quorum=localhost:59952, baseZNode=/hbase Transitioning cae5ea3b1868354349837ee2dd0d4d89 from M_ZK_REGION_CLOSING to RS_ZK_REGION_CLOSED 2015-06-15 03:03:42,599 INFO [RunAmJoinCluster] master.AssignmentManager(767): Processing cae5ea3b1868354349837ee2dd0d4d89 in state: M_ZK_REGION_CLOSING 2015-06-15 03:03:42,599 INFO [RunAmJoinCluster] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=PENDING_CLOSE, ts=1434337422596, server=example.org,1234,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=CLOSING, ts=1434337422599, server=example.org,1234,5678} 2015-06-15 03:03:42,611 INFO [RunAmJoinCluster] master.AssignmentManager(898): Processed region cae5ea3b1868354349837ee2dd0d4d89 in state M_ZK_REGION_CLOSING, on server: example.org,1234,5678 2015-06-15 03:03:42,611 DEBUG [Thread-87] zookeeper.ZKAssign(880): mockedServer-0x14df52ac8120013, quorum=localhost:59952, baseZNode=/hbase Transitioned node cae5ea3b1868354349837ee2dd0d4d89 from M_ZK_REGION_CLOSING to RS_ZK_REGION_CLOSED 2015-06-15 03:03:42,611 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer-0x14df52ac8120013, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/cae5ea3b1868354349837ee2dd0d4d89 2015-06-15 03:03:42,615 INFO [RunAmJoinCluster] master.AssignmentManager(499): Joined the cluster in 57ms, failover=true 2015-06-15 03:03:42,630 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:42,630 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:42,631 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:42,631 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:42,632 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:42,633 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:42,633 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:42,633 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:42,633 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:42,634 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:42,634 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.RegionStates(1103): Failed to open/close cae5ea3b1868354349837ee2dd0d4d89 on example.org,1234,5678, set to FAILED_CLOSE 2015-06-15 03:03:42,634 INFO [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=CLOSING, ts=1434337422599, server=example.org,1234,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=FAILED_CLOSE, ts=1434337422634, server=example.org,1234,5678} 2015-06-15 03:03:42,637 DEBUG [AM.ZK.Worker-pool48-t1] master.AssignmentManager(954): Handling RS_ZK_REGION_CLOSED, server=example.org,1234,5678, region=cae5ea3b1868354349837ee2dd0d4d89, current_state={cae5ea3b1868354349837ee2dd0d4d89 state=FAILED_CLOSE, ts=1434337422634, server=example.org,1234,5678} 2015-06-15 03:03:42,637 DEBUG [AM.ZK.Worker-pool48-t1] handler.ClosedRegionHandler(93): Handling CLOSED event for cae5ea3b1868354349837ee2dd0d4d89 2015-06-15 03:03:42,638 INFO [AM.ZK.Worker-pool48-t1] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=FAILED_CLOSE, ts=1434337422634, server=example.org,1234,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=CLOSED, ts=1434337422637, server=example.org,1234,5678} 2015-06-15 03:03:42,644 DEBUG [AM.ZK.Worker-pool48-t1] master.AssignmentManager(2381): Found an existing plan for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. destination server is example.org,0,5678 accepted as a dest server = true 2015-06-15 03:03:42,644 DEBUG [AM.ZK.Worker-pool48-t1] master.AssignmentManager(2419): Using pre-existing plan for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89.; plan=hri=t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89., src=, dest=example.org,0,5678 2015-06-15 03:03:42,644 INFO [AM.ZK.Worker-pool48-t1] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=CLOSED, ts=1434337422637, server=example.org,1234,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=OFFLINE, ts=1434337422644, server=example.org,1234,5678} 2015-06-15 03:03:42,644 DEBUG [AM.ZK.Worker-pool48-t1] zookeeper.ZKAssign(205): mockedServer-0x14df52ac8120013, quorum=localhost:59952, baseZNode=/hbase Creating (or updating) unassigned node cae5ea3b1868354349837ee2dd0d4d89 with OFFLINE state 2015-06-15 03:03:42,650 INFO [AM.ZK.Worker-pool48-t1] master.AssignmentManager(2122): Assigning t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. to example.org,0,5678 2015-06-15 03:03:42,650 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer-0x14df52ac8120013, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/cae5ea3b1868354349837ee2dd0d4d89 2015-06-15 03:03:42,650 INFO [AM.ZK.Worker-pool48-t1] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=OFFLINE, ts=1434337422644, server=example.org,1234,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=PENDING_OPEN, ts=1434337422650, server=example.org,0,5678} 2015-06-15 03:03:42,655 DEBUG [Thread-87] zookeeper.ZKAssign(805): mockedServer-0x14df52ac8120013, quorum=localhost:59952, baseZNode=/hbase Transitioning cae5ea3b1868354349837ee2dd0d4d89 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-06-15 03:03:42,658 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer-0x14df52ac8120013, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/cae5ea3b1868354349837ee2dd0d4d89 2015-06-15 03:03:42,658 DEBUG [Thread-87] zookeeper.ZKAssign(880): mockedServer-0x14df52ac8120013, quorum=localhost:59952, baseZNode=/hbase Transitioned node cae5ea3b1868354349837ee2dd0d4d89 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-06-15 03:03:42,659 DEBUG [Thread-87] zookeeper.ZKAssign(805): mockedServer-0x14df52ac8120013, quorum=localhost:59952, baseZNode=/hbase Transitioning cae5ea3b1868354349837ee2dd0d4d89 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2015-06-15 03:03:42,662 DEBUG [Thread-87] zookeeper.ZKAssign(880): mockedServer-0x14df52ac8120013, quorum=localhost:59952, baseZNode=/hbase Transitioned node cae5ea3b1868354349837ee2dd0d4d89 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2015-06-15 03:03:42,665 DEBUG [Thread-87] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:42,670 DEBUG [AM.ZK.Worker-pool48-t2] master.AssignmentManager(954): Handling RS_ZK_REGION_OPENED, server=example.org,0,5678, region=cae5ea3b1868354349837ee2dd0d4d89, current_state={cae5ea3b1868354349837ee2dd0d4d89 state=PENDING_OPEN, ts=1434337422650, server=example.org,0,5678} 2015-06-15 03:03:42,671 INFO [AM.ZK.Worker-pool48-t2] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=PENDING_OPEN, ts=1434337422650, server=example.org,0,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=OPEN, ts=1434337422671, server=example.org,0,5678} 2015-06-15 03:03:42,673 DEBUG [AM.ZK.Worker-pool48-t2] coordination.ZkOpenRegionCoordination(350): Handling OPENED of cae5ea3b1868354349837ee2dd0d4d89 from master,1,1; deleting unassigned node 2015-06-15 03:03:42,679 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer-0x14df52ac8120013, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/cae5ea3b1868354349837ee2dd0d4d89 2015-06-15 03:03:42,679 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer-0x14df52ac8120013, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-06-15 03:03:42,682 DEBUG [AM.ZK.Worker-pool48-t2] zookeeper.ZKAssign(479): mockedServer-0x14df52ac8120013, quorum=localhost:59952, baseZNode=/hbase Deleted unassigned node cae5ea3b1868354349837ee2dd0d4d89 in expected state RS_ZK_REGION_OPENED 2015-06-15 03:03:42,702 DEBUG [AM.ZK.Worker-pool48-t5] master.AssignmentManager$4(1343): Znode t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. deleted, state: {cae5ea3b1868354349837ee2dd0d4d89 state=OPEN, ts=1434337422671, server=example.org,0,5678} 2015-06-15 03:03:42,703 DEBUG [AM.ZK.Worker-pool48-t5] master.RegionStates(460): Onlined cae5ea3b1868354349837ee2dd0d4d89 on example.org,0,5678 {ENCODED => cae5ea3b1868354349837ee2dd0d4d89, NAME => 't,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89.', STARTKEY => '', ENDKEY => ''} 2015-06-15 03:03:42,704 INFO [AM.ZK.Worker-pool48-t5] master.RegionStates(472): Offlined cae5ea3b1868354349837ee2dd0d4d89 from example.org,1234,5678 2015-06-15 03:03:42,767 DEBUG [main] zookeeper.ZKAssign(497): mockedServer-0x14df52ac8120013, quorum=localhost:59952, baseZNode=/hbase Deleting any existing unassigned nodes 2015-06-15 03:03:42,784 INFO [main] hbase.ResourceChecker(171): after: master.TestAssignmentManager#testBalanceOnMasterFailoverScenarioWithOpenedNode Thread=115 (was 115), OpenFileDescriptor=302 (was 304), MaxFileDescriptor=32768 (was 32768), SystemLoadAverage=620 (was 620), ProcessCount=100 (was 100), AvailableMemoryMB=5011 (was 5017), ConnectionCount=3 (was 3) 2015-06-15 03:03:42,798 INFO [main] hbase.ResourceChecker(147): before: master.TestAssignmentManager#testSSHWhenDisableTableInProgress Thread=115, OpenFileDescriptor=304, MaxFileDescriptor=32768, SystemLoadAverage=620, ProcessCount=100, AvailableMemoryMB=5017, ConnectionCount=3 2015-06-15 03:03:42,798 INFO [main] zookeeper.RecoverableZooKeeper(121): Process identifier=mockedServer connecting to ZooKeeper ensemble=localhost:59952 2015-06-15 03:03:42,806 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer0x0, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-15 03:03:42,807 DEBUG [main] zookeeper.RecoverableZooKeeper(595): Node /hbase already exists 2015-06-15 03:03:42,808 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(592): mockedServer-0x14df52ac8120014 connected 2015-06-15 03:03:42,817 DEBUG [Thread-88] executor.ExecutorService(99): Starting executor service name=MASTER_OPEN_REGION-testSSHWhenDisableTableInProgress, corePoolSize=3, maxPoolSize=3 2015-06-15 03:03:42,818 DEBUG [Thread-88] executor.ExecutorService(99): Starting executor service name=MASTER_CLOSE_REGION-testSSHWhenDisableTableInProgress, corePoolSize=3, maxPoolSize=3 2015-06-15 03:03:42,818 DEBUG [Thread-88] executor.ExecutorService(99): Starting executor service name=MASTER_SERVER_OPERATIONS-testSSHWhenDisableTableInProgress, corePoolSize=3, maxPoolSize=3 2015-06-15 03:03:42,818 DEBUG [Thread-88] executor.ExecutorService(99): Starting executor service name=MASTER_META_SERVER_OPERATIONS-testSSHWhenDisableTableInProgress, corePoolSize=3, maxPoolSize=3 2015-06-15 03:03:42,820 DEBUG [Thread-88] zookeeper.ZKAssign(497): mockedServer-0x14df52ac8120014, quorum=localhost:59952, baseZNode=/hbase Deleting any existing unassigned nodes 2015-06-15 03:03:42,824 INFO [Thread-88] master.RegionStates(1112): Transition null to {cae5ea3b1868354349837ee2dd0d4d89 state=OPEN, ts=1434337422824, server=example.org,1234,5678} 2015-06-15 03:03:42,824 DEBUG [Thread-88] master.RegionStates(460): Onlined cae5ea3b1868354349837ee2dd0d4d89 on example.org,1234,5678 {ENCODED => cae5ea3b1868354349837ee2dd0d4d89, NAME => 't,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89.', STARTKEY => '', ENDKEY => ''} 2015-06-15 03:03:42,825 INFO [Thread-88] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=OPEN, ts=1434337422824, server=example.org,1234,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=PENDING_CLOSE, ts=1434337422825, server=example.org,1234,5678} 2015-06-15 03:03:42,825 INFO [Thread-88] zookeeper.ZKTableStateManager(101): Moving table t state from ENABLED to DISABLING 2015-06-15 03:03:42,855 INFO [Thread-88] handler.ServerShutdownHandler(217): Skipping log splitting for example.org,1234,5678 2015-06-15 03:03:42,855 INFO [Thread-88] master.RegionStates(651): Transitioning {cae5ea3b1868354349837ee2dd0d4d89 state=PENDING_CLOSE, ts=1434337422825, server=example.org,1234,5678} will be handled by SSH for example.org,1234,5678 2015-06-15 03:03:42,855 INFO [Thread-88] handler.ServerShutdownHandler(238): Reassigning 1 region(s) that example.org,1234,5678 was carrying (and 0 regions(s) that were opening on this server) 2015-06-15 03:03:42,855 INFO [Thread-88] handler.ServerShutdownHandler(393): The table t is disabled. Hence not assigning regioncae5ea3b1868354349837ee2dd0d4d89 2015-06-15 03:03:42,856 INFO [Thread-88] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=PENDING_CLOSE, ts=1434337422825, server=example.org,1234,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=OFFLINE, ts=1434337422856, server=example.org,1234,5678} 2015-06-15 03:03:42,861 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer-0x14df52ac8120014, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/cae5ea3b1868354349837ee2dd0d4d89 2015-06-15 03:03:42,861 DEBUG [Thread-88] zookeeper.ZKAssign(479): mockedServer-0x14df52ac8120014, quorum=localhost:59952, baseZNode=/hbase Deleted unassigned node cae5ea3b1868354349837ee2dd0d4d89 in expected state M_ZK_REGION_CLOSING 2015-06-15 03:03:42,863 DEBUG [Thread-88] master.AssignmentManager(1536): Table being disabled so deleting ZK node and removing from regions in transition, skipping assignment of region t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:42,864 DEBUG [Thread-88] zookeeper.ZKUtil(839): mockedServer-0x14df52ac8120014, quorum=localhost:59952, baseZNode=/hbase Unable to get data of znode /hbase/region-in-transition/cae5ea3b1868354349837ee2dd0d4d89 because node does not exist (not necessarily an error) 2015-06-15 03:03:42,864 DEBUG [Thread-88] master.AssignmentManager(3569): The closed node for cae5ea3b1868354349837ee2dd0d4d89 already deleted 2015-06-15 03:03:42,864 INFO [Thread-88] master.RegionStates(600): Offlined cae5ea3b1868354349837ee2dd0d4d89 from example.org,1234,5678 2015-06-15 03:03:42,864 DEBUG [Thread-88] master.DeadServer(118): Finished processing example.org,1234,5678 2015-06-15 03:03:42,864 INFO [Thread-88] handler.ServerShutdownHandler(347): Finished processing of shutdown of example.org,1234,5678 2015-06-15 03:03:42,868 INFO [Thread-88] zookeeper.ZKTableStateManager(101): Moving table t state from DISABLING to ENABLED 2015-06-15 03:03:42,873 DEBUG [Thread-88] zookeeper.ZKAssign(497): mockedServer-0x14df52ac8120014, quorum=localhost:59952, baseZNode=/hbase Deleting any existing unassigned nodes 2015-06-15 03:03:42,873 DEBUG [Thread-88] executor.ExecutorService(99): Starting executor service name=MASTER_OPEN_REGION-testSSHWhenDisableTableInProgress, corePoolSize=3, maxPoolSize=3 2015-06-15 03:03:42,874 DEBUG [Thread-88] executor.ExecutorService(99): Starting executor service name=MASTER_CLOSE_REGION-testSSHWhenDisableTableInProgress, corePoolSize=3, maxPoolSize=3 2015-06-15 03:03:42,874 DEBUG [Thread-88] executor.ExecutorService(99): Starting executor service name=MASTER_SERVER_OPERATIONS-testSSHWhenDisableTableInProgress, corePoolSize=3, maxPoolSize=3 2015-06-15 03:03:42,874 DEBUG [Thread-88] executor.ExecutorService(99): Starting executor service name=MASTER_META_SERVER_OPERATIONS-testSSHWhenDisableTableInProgress, corePoolSize=3, maxPoolSize=3 2015-06-15 03:03:42,874 DEBUG [Thread-88] zookeeper.ZKAssign(497): mockedServer-0x14df52ac8120014, quorum=localhost:59952, baseZNode=/hbase Deleting any existing unassigned nodes 2015-06-15 03:03:42,881 INFO [Thread-88] master.RegionStates(1112): Transition null to {cae5ea3b1868354349837ee2dd0d4d89 state=OPEN, ts=1434337422880, server=example.org,1234,5678} 2015-06-15 03:03:42,881 DEBUG [Thread-88] master.RegionStates(460): Onlined cae5ea3b1868354349837ee2dd0d4d89 on example.org,1234,5678 {ENCODED => cae5ea3b1868354349837ee2dd0d4d89, NAME => 't,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89.', STARTKEY => '', ENDKEY => ''} 2015-06-15 03:03:42,882 INFO [Thread-88] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=OPEN, ts=1434337422880, server=example.org,1234,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=PENDING_CLOSE, ts=1434337422882, server=example.org,1234,5678} 2015-06-15 03:03:42,883 INFO [Thread-88] zookeeper.ZKTableStateManager(101): Moving table t state from ENABLED to DISABLED 2015-06-15 03:03:42,933 INFO [Thread-88] handler.ServerShutdownHandler(217): Skipping log splitting for example.org,1234,5678 2015-06-15 03:03:42,933 INFO [Thread-88] master.RegionStates(651): Transitioning {cae5ea3b1868354349837ee2dd0d4d89 state=PENDING_CLOSE, ts=1434337422882, server=example.org,1234,5678} will be handled by SSH for example.org,1234,5678 2015-06-15 03:03:42,933 INFO [Thread-88] handler.ServerShutdownHandler(238): Reassigning 1 region(s) that example.org,1234,5678 was carrying (and 0 regions(s) that were opening on this server) 2015-06-15 03:03:42,933 INFO [Thread-88] handler.ServerShutdownHandler(380): The table t was disabled. Hence not proceeding. 2015-06-15 03:03:42,933 INFO [Thread-88] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=PENDING_CLOSE, ts=1434337422882, server=example.org,1234,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=OFFLINE, ts=1434337422933, server=example.org,1234,5678} 2015-06-15 03:03:42,937 DEBUG [Thread-88] zookeeper.ZKAssign(479): mockedServer-0x14df52ac8120014, quorum=localhost:59952, baseZNode=/hbase Deleted unassigned node cae5ea3b1868354349837ee2dd0d4d89 in expected state M_ZK_REGION_CLOSING 2015-06-15 03:03:42,937 DEBUG [Thread-88] master.AssignmentManager(1536): Table being disabled so deleting ZK node and removing from regions in transition, skipping assignment of region t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:42,937 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer-0x14df52ac8120014, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/cae5ea3b1868354349837ee2dd0d4d89 2015-06-15 03:03:42,938 DEBUG [Thread-88] zookeeper.ZKUtil(839): mockedServer-0x14df52ac8120014, quorum=localhost:59952, baseZNode=/hbase Unable to get data of znode /hbase/region-in-transition/cae5ea3b1868354349837ee2dd0d4d89 because node does not exist (not necessarily an error) 2015-06-15 03:03:42,938 DEBUG [Thread-88] master.AssignmentManager(3569): The closed node for cae5ea3b1868354349837ee2dd0d4d89 already deleted 2015-06-15 03:03:42,938 INFO [Thread-88] master.RegionStates(600): Offlined cae5ea3b1868354349837ee2dd0d4d89 from example.org,1234,5678 2015-06-15 03:03:42,938 DEBUG [Thread-88] master.DeadServer(118): Finished processing example.org,1234,5678 2015-06-15 03:03:42,938 INFO [Thread-88] handler.ServerShutdownHandler(347): Finished processing of shutdown of example.org,1234,5678 2015-06-15 03:03:42,939 INFO [Thread-88] zookeeper.ZKTableStateManager(101): Moving table t state from DISABLED to ENABLED 2015-06-15 03:03:42,940 ERROR [main-EventThread] zookeeper.ClientCnxn$EventThread(524): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@b8b4416 rejected from java.util.concurrent.ThreadPoolExecutor@64230645[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110) at org.apache.hadoop.hbase.master.AssignmentManager.zkEventWorkersSubmit(AssignmentManager.java:1291) at org.apache.hadoop.hbase.master.AssignmentManager.nodeDeleted(AssignmentManager.java:1322) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:532) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) 2015-06-15 03:03:42,941 DEBUG [Thread-88] zookeeper.ZKAssign(497): mockedServer-0x14df52ac8120014, quorum=localhost:59952, baseZNode=/hbase Deleting any existing unassigned nodes 2015-06-15 03:03:42,942 DEBUG [main] zookeeper.ZKAssign(497): mockedServer-0x14df52ac8120014, quorum=localhost:59952, baseZNode=/hbase Deleting any existing unassigned nodes 2015-06-15 03:03:42,996 INFO [main] hbase.ResourceChecker(171): after: master.TestAssignmentManager#testSSHWhenDisableTableInProgress Thread=115 (was 115), OpenFileDescriptor=302 (was 304), MaxFileDescriptor=32768 (was 32768), SystemLoadAverage=620 (was 620), ProcessCount=100 (was 100), AvailableMemoryMB=5016 (was 5017), ConnectionCount=3 (was 3) 2015-06-15 03:03:43,012 INFO [main] hbase.ResourceChecker(147): before: master.TestAssignmentManager#testOpenCloseRegionRPCIntendedForPreviousServer Thread=115, OpenFileDescriptor=304, MaxFileDescriptor=32768, SystemLoadAverage=620, ProcessCount=100, AvailableMemoryMB=5016, ConnectionCount=3 2015-06-15 03:03:43,013 INFO [main] zookeeper.RecoverableZooKeeper(121): Process identifier=mockedServer connecting to ZooKeeper ensemble=localhost:59952 2015-06-15 03:03:43,025 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer0x0, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-15 03:03:43,026 DEBUG [main] zookeeper.RecoverableZooKeeper(595): Node /hbase already exists 2015-06-15 03:03:43,027 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(592): mockedServer-0x14df52ac8120015 connected 2015-06-15 03:03:43,038 WARN [Thread-89] master.AssignmentManager(1947): Assigning a region not in region states: {ENCODED => cae5ea3b1868354349837ee2dd0d4d89, NAME => 't,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89.', STARTKEY => '', ENDKEY => ''} 2015-06-15 03:03:43,039 DEBUG [Thread-89] master.AssignmentManager(2381): Found an existing plan for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. destination server is example.org,0,5678 accepted as a dest server = true 2015-06-15 03:03:43,039 DEBUG [Thread-89] master.AssignmentManager(2419): Using pre-existing plan for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89.; plan=hri=t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89., src=, dest=example.org,0,5678 2015-06-15 03:03:43,039 DEBUG [Thread-89] zookeeper.ZKAssign(205): mockedServer-0x14df52ac8120015, quorum=localhost:59952, baseZNode=/hbase Creating (or updating) unassigned node cae5ea3b1868354349837ee2dd0d4d89 with OFFLINE state 2015-06-15 03:03:43,044 INFO [Thread-89] master.AssignmentManager(2122): Assigning t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. to example.org,0,5678 2015-06-15 03:03:43,044 INFO [Thread-89] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=OFFLINE, ts=1434337423039, server=null} to {cae5ea3b1868354349837ee2dd0d4d89 state=PENDING_OPEN, ts=1434337423044, server=example.org,0,5678} 2015-06-15 03:03:43,045 WARN [Thread-89] master.AssignmentManager(2214): Failed assignment of t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. to example.org,0,5678, trying to assign elsewhere instead; try=1 of 100 org.apache.hadoop.hbase.DoNotRetryIOException at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2136) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1590) at org.apache.hadoop.hbase.master.TestAssignmentManager.testOpenCloseRegionRPCIntendedForPreviousServer(TestAssignmentManager.java:1511) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) 2015-06-15 03:03:43,046 DEBUG [Thread-89] master.AssignmentManager(2381): Found an existing plan for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. destination server is example.org,0,5678 accepted as a dest server = true 2015-06-15 03:03:43,196 DEBUG [Thread-89] master.AssignmentManager(2413): No previous transition plan found (or ignoring an existing plan) for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89.; generated random plan=hri=t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89., src=, dest=example.org,0,5678; 2 (online=2) available servers, forceNewPlan=true 2015-06-15 03:03:43,196 INFO [Thread-89] master.AssignmentManager(2122): Assigning t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. to example.org,0,5678 2015-06-15 03:03:43,197 WARN [Thread-89] master.AssignmentManager(2214): Failed assignment of t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. to example.org,0,5678, trying to assign elsewhere instead; try=2 of 100 org.apache.hadoop.hbase.DoNotRetryIOException at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2136) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1590) at org.apache.hadoop.hbase.master.TestAssignmentManager.testOpenCloseRegionRPCIntendedForPreviousServer(TestAssignmentManager.java:1511) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) 2015-06-15 03:03:43,197 DEBUG [Thread-89] master.AssignmentManager(2381): Found an existing plan for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. destination server is example.org,0,5678 accepted as a dest server = true 2015-06-15 03:03:43,204 DEBUG [Thread-89] master.AssignmentManager(2413): No previous transition plan found (or ignoring an existing plan) for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89.; generated random plan=hri=t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89., src=, dest=example.org,1234,5678; 2 (online=2) available servers, forceNewPlan=true 2015-06-15 03:03:43,204 INFO [Thread-89] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=PENDING_OPEN, ts=1434337423197, server=example.org,0,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=OFFLINE, ts=1434337423204, server=example.org,0,5678} 2015-06-15 03:03:43,212 DEBUG [Thread-89] zookeeper.ZKAssign(205): mockedServer-0x14df52ac8120015, quorum=localhost:59952, baseZNode=/hbase Creating (or updating) unassigned node cae5ea3b1868354349837ee2dd0d4d89 with OFFLINE state 2015-06-15 03:03:43,219 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer-0x14df52ac8120015, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/cae5ea3b1868354349837ee2dd0d4d89 2015-06-15 03:03:43,223 INFO [Thread-89] master.AssignmentManager(2122): Assigning t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. to example.org,1234,5678 2015-06-15 03:03:43,224 INFO [Thread-89] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=OFFLINE, ts=1434337423212, server=example.org,0,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=PENDING_OPEN, ts=1434337423224, server=example.org,1234,5678} 2015-06-15 03:03:43,224 DEBUG [main] zookeeper.ZKAssign(497): mockedServer-0x14df52ac8120015, quorum=localhost:59952, baseZNode=/hbase Deleting any existing unassigned nodes 2015-06-15 03:03:43,229 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer-0x14df52ac8120015, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/cae5ea3b1868354349837ee2dd0d4d89 2015-06-15 03:03:43,249 INFO [main] hbase.ResourceChecker(171): after: master.TestAssignmentManager#testOpenCloseRegionRPCIntendedForPreviousServer Thread=116 (was 115) Potentially hanging thread: main-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) - Thread LEAK? -, OpenFileDescriptor=302 (was 304), MaxFileDescriptor=32768 (was 32768), SystemLoadAverage=620 (was 620), ProcessCount=100 (was 100), AvailableMemoryMB=5016 (was 5016), ConnectionCount=3 (was 3) 2015-06-15 03:03:43,278 INFO [main] hbase.ResourceChecker(147): before: master.TestAssignmentManager#testMasterRestartShouldRemoveStaleZnodesOfUnknownTableAsForMeta Thread=115, OpenFileDescriptor=304, MaxFileDescriptor=32768, SystemLoadAverage=620, ProcessCount=100, AvailableMemoryMB=5015, ConnectionCount=3 2015-06-15 03:03:43,278 INFO [main] zookeeper.RecoverableZooKeeper(121): Process identifier=mockedServer connecting to ZooKeeper ensemble=localhost:59952 2015-06-15 03:03:43,289 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer0x0, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-15 03:03:43,294 DEBUG [main] zookeeper.RecoverableZooKeeper(595): Node /hbase already exists 2015-06-15 03:03:43,299 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(592): mockedServer-0x14df52ac8120016 connected 2015-06-15 03:03:43,310 INFO [Thread-90] client.ConnectionUtils(103): master/ndimiduk-hbase-1-1-1RC0-1.openstacklocal/172.22.113.182:0 server-side HConnection retries=35000 2015-06-15 03:03:43,310 INFO [Thread-90] ipc.SimpleRpcScheduler(128): Using deadline as user call queue, count=1 2015-06-15 03:03:43,313 INFO [Thread-90] ipc.RpcServer$Listener(595): master/ndimiduk-hbase-1-1-1RC0-1.openstacklocal/172.22.113.182:0: started 10 reader(s). 2015-06-15 03:03:43,327 DEBUG [Thread-90] fs.HFileSystem(221): The file system is not a DistributedFileSystem. Skipping on block location reordering 2015-06-15 03:03:43,328 INFO [Thread-90] zookeeper.RecoverableZooKeeper(121): Process identifier=master:44787 connecting to ZooKeeper ensemble=localhost:59952 2015-06-15 03:03:43,337 DEBUG [Thread-90-EventThread] zookeeper.ZooKeeperWatcher(508): master:447870x0, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-15 03:03:43,339 DEBUG [Thread-90-EventThread] zookeeper.ZooKeeperWatcher(592): master:44787-0x14df52ac8120017 connected 2015-06-15 03:03:43,339 DEBUG [Thread-90] zookeeper.RecoverableZooKeeper(595): Node /hbase already exists 2015-06-15 03:03:43,345 DEBUG [Thread-90] zookeeper.ZKUtil(490): master:44787-0x14df52ac8120017, quorum=localhost:59952, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2015-06-15 03:03:43,347 DEBUG [Thread-90] zookeeper.ZKUtil(490): master:44787-0x14df52ac8120017, quorum=localhost:59952, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2015-06-15 03:03:43,352 INFO [RpcServer.responder] ipc.RpcServer$Responder(914): RpcServer.responder: starting 2015-06-15 03:03:43,356 INFO [RpcServer.listener,port=44787] ipc.RpcServer$Listener(726): RpcServer.listener,port=44787: starting 2015-06-15 03:03:43,366 DEBUG [Thread-90] ipc.RpcExecutor(114): B.default Start Handler index=0 queue=0 2015-06-15 03:03:43,373 DEBUG [Thread-90] ipc.RpcExecutor(114): B.default Start Handler index=1 queue=0 2015-06-15 03:03:43,373 DEBUG [Thread-90] ipc.RpcExecutor(114): B.default Start Handler index=2 queue=0 2015-06-15 03:03:43,374 DEBUG [Thread-90] ipc.RpcExecutor(114): B.default Start Handler index=3 queue=0 2015-06-15 03:03:43,374 DEBUG [Thread-90] ipc.RpcExecutor(114): B.default Start Handler index=4 queue=0 2015-06-15 03:03:43,376 DEBUG [Thread-90] ipc.RpcExecutor(114): Priority Start Handler index=0 queue=0 2015-06-15 03:03:43,376 DEBUG [Thread-90] ipc.RpcExecutor(114): Priority Start Handler index=1 queue=1 2015-06-15 03:03:43,376 DEBUG [Thread-90] ipc.RpcExecutor(114): Priority Start Handler index=2 queue=0 2015-06-15 03:03:43,377 DEBUG [Thread-90] ipc.RpcExecutor(114): Priority Start Handler index=3 queue=1 2015-06-15 03:03:43,377 DEBUG [Thread-90] ipc.RpcExecutor(114): Priority Start Handler index=4 queue=0 2015-06-15 03:03:43,378 DEBUG [Thread-90] ipc.RpcExecutor(114): Priority Start Handler index=5 queue=1 2015-06-15 03:03:43,378 DEBUG [Thread-90] ipc.RpcExecutor(114): Priority Start Handler index=6 queue=0 2015-06-15 03:03:43,378 DEBUG [Thread-90] ipc.RpcExecutor(114): Priority Start Handler index=7 queue=1 2015-06-15 03:03:43,379 DEBUG [Thread-90] ipc.RpcExecutor(114): Priority Start Handler index=8 queue=0 2015-06-15 03:03:43,379 DEBUG [Thread-90] ipc.RpcExecutor(114): Priority Start Handler index=9 queue=1 2015-06-15 03:03:43,379 DEBUG [Thread-90] ipc.RpcExecutor(114): Priority Start Handler index=10 queue=0 2015-06-15 03:03:43,380 DEBUG [Thread-90] ipc.RpcExecutor(114): Priority Start Handler index=11 queue=1 2015-06-15 03:03:43,380 DEBUG [Thread-90] ipc.RpcExecutor(114): Priority Start Handler index=12 queue=0 2015-06-15 03:03:43,381 DEBUG [Thread-90] ipc.RpcExecutor(114): Priority Start Handler index=13 queue=1 2015-06-15 03:03:43,381 DEBUG [Thread-90] ipc.RpcExecutor(114): Priority Start Handler index=14 queue=0 2015-06-15 03:03:43,385 DEBUG [Thread-90] ipc.RpcExecutor(114): Priority Start Handler index=15 queue=1 2015-06-15 03:03:43,386 DEBUG [Thread-90] ipc.RpcExecutor(114): Priority Start Handler index=16 queue=0 2015-06-15 03:03:43,386 DEBUG [Thread-90] ipc.RpcExecutor(114): Priority Start Handler index=17 queue=1 2015-06-15 03:03:43,387 DEBUG [Thread-90] ipc.RpcExecutor(114): Priority Start Handler index=18 queue=0 2015-06-15 03:03:43,387 DEBUG [Thread-90] ipc.RpcExecutor(114): Priority Start Handler index=19 queue=1 2015-06-15 03:03:43,388 DEBUG [Thread-90] ipc.RpcExecutor(114): Replication Start Handler index=0 queue=0 2015-06-15 03:03:43,388 DEBUG [Thread-90] ipc.RpcExecutor(114): Replication Start Handler index=1 queue=0 2015-06-15 03:03:43,403 DEBUG [Thread-90] ipc.RpcExecutor(114): Replication Start Handler index=2 queue=0 2015-06-15 03:03:43,403 INFO [Thread-90] master.HMaster(368): hbase.rootdir=file:/tmp/hbase-root/hbase, hbase.cluster.distributed=false 2015-06-15 03:03:43,403 INFO [Thread-90] master.HMaster(1616): Adding backup master ZNode /hbase/backup-masters/ndimiduk-hbase-1-1-1RC0-1.openstacklocal,44787,1434337423327 2015-06-15 03:03:43,406 DEBUG [Thread-90] zookeeper.ZKUtil(490): master:44787-0x14df52ac8120017, quorum=localhost:59952, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/ndimiduk-hbase-1-1-1RC0-1.openstacklocal,44787,1434337423327 2015-06-15 03:03:43,411 DEBUG [Thread-90] executor.ExecutorService(99): Starting executor service name=MASTER_OPEN_REGION-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:03:43,411 DEBUG [Thread-90] executor.ExecutorService(99): Starting executor service name=MASTER_CLOSE_REGION-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:03:43,412 DEBUG [Thread-90] executor.ExecutorService(99): Starting executor service name=MASTER_SERVER_OPERATIONS-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:03:43,412 DEBUG [Thread-90] executor.ExecutorService(99): Starting executor service name=MASTER_META_SERVER_OPERATIONS-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:03:43,417 DEBUG [ndimiduk-hbase-1-1-1RC0-1:44787.activeMasterManager] zookeeper.RecoverableZooKeeper(595): Node /hbase/master already exists 2015-06-15 03:03:43,418 DEBUG [ndimiduk-hbase-1-1-1RC0-1:44787.activeMasterManager] zookeeper.ZKUtil(490): master:44787-0x14df52ac8120017, quorum=localhost:59952, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2015-06-15 03:03:43,420 INFO [ndimiduk-hbase-1-1-1RC0-1:44787.activeMasterManager] master.ActiveMasterManager(216): Another master is the active master, ndimiduk-hbase-1-1-1RC0-1.openstacklocal,60005,1434337400084; waiting to become the next active master 2015-06-15 03:03:43,420 INFO [Thread-90] zookeeper.ZKTableStateManager(101): Moving table dummyTable state from null to ENABLING 2015-06-15 03:03:43,441 DEBUG [Thread-90] master.AssignmentManager(540): Found {ENCODED => cae5ea3b1868354349837ee2dd0d4d89, NAME => 't,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89.', STARTKEY => '', ENDKEY => ''} out on cluster 2015-06-15 03:03:43,441 INFO [Thread-90] master.AssignmentManager(623): Found regions out on cluster or in RIT; presuming failover 2015-06-15 03:03:43,446 INFO [Thread-90] master.AssignmentManager(3045): The table dummyTable is in ENABLING state. Hence recovering by moving the table to ENABLED state. 2015-06-15 03:03:43,452 WARN [Thread-90] master.AssignmentManager(3055): Table dummyTable not found in hbase:meta to recover. 2015-06-15 03:03:43,452 INFO [Thread-90] master.AssignmentManager(499): Joined the cluster in 22ms, failover=true 2015-06-15 03:03:43,452 DEBUG [main] zookeeper.ZKAssign(497): mockedServer-0x14df52ac8120016, quorum=localhost:59952, baseZNode=/hbase Deleting any existing unassigned nodes 2015-06-15 03:03:43,482 INFO [main] hbase.ResourceChecker(171): after: master.TestAssignmentManager#testMasterRestartShouldRemoveStaleZnodesOfUnknownTableAsForMeta Thread=158 (was 115) Potentially hanging thread: B.defaultRpcServer.handler=2,queue=0,port=44787 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) org.apache.hadoop.hbase.util.BoundedPriorityBlockingQueue.take(BoundedPriorityBlockingQueue.java:214) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: B.defaultRpcServer.handler=3,queue=0,port=44787 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) org.apache.hadoop.hbase.util.BoundedPriorityBlockingQueue.take(BoundedPriorityBlockingQueue.java:214) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PriorityRpcServer.handler=8,queue=0,port=44787 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RpcServer.reader=8,bindAddress=ndimiduk-hbase-1-1-1RC0-1.openstacklocal,port=44787 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:627) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:614) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PriorityRpcServer.handler=1,queue=1,port=44787 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: B.defaultRpcServer.handler=4,queue=0,port=44787 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) org.apache.hadoop.hbase.util.BoundedPriorityBlockingQueue.take(BoundedPriorityBlockingQueue.java:214) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RpcServer.reader=3,bindAddress=ndimiduk-hbase-1-1-1RC0-1.openstacklocal,port=44787 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:627) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:614) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: ReplicationRpcServer.handler=1,queue=0,port=44787 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RpcServer.reader=7,bindAddress=ndimiduk-hbase-1-1-1RC0-1.openstacklocal,port=44787 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:627) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:614) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RpcServer.reader=2,bindAddress=ndimiduk-hbase-1-1-1RC0-1.openstacklocal,port=44787 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:627) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:614) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PriorityRpcServer.handler=10,queue=0,port=44787 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RpcServer.reader=9,bindAddress=ndimiduk-hbase-1-1-1RC0-1.openstacklocal,port=44787 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:627) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:614) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: B.defaultRpcServer.handler=0,queue=0,port=44787 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) org.apache.hadoop.hbase.util.BoundedPriorityBlockingQueue.take(BoundedPriorityBlockingQueue.java:214) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: ndimiduk-hbase-1-1-1RC0-1:44787.activeMasterManager java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.master.ActiveMasterManager.blockUntilBecomingActiveMaster(ActiveMasterManager.java:225) org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1645) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PriorityRpcServer.handler=12,queue=0,port=44787 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PriorityRpcServer.handler=0,queue=0,port=44787 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: ReplicationRpcServer.handler=0,queue=0,port=44787 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PriorityRpcServer.handler=19,queue=1,port=44787 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PriorityRpcServer.handler=2,queue=0,port=44787 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: B.defaultRpcServer.handler=1,queue=0,port=44787 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) org.apache.hadoop.hbase.util.BoundedPriorityBlockingQueue.take(BoundedPriorityBlockingQueue.java:214) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PriorityRpcServer.handler=13,queue=1,port=44787 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: ReplicationRpcServer.handler=2,queue=0,port=44787 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RpcServer.reader=1,bindAddress=ndimiduk-hbase-1-1-1RC0-1.openstacklocal,port=44787 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:627) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:614) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RpcServer.listener,port=44787 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) org.apache.hadoop.hbase.ipc.RpcServer$Listener.run(RpcServer.java:730) Potentially hanging thread: PriorityRpcServer.handler=16,queue=0,port=44787 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RpcServer.reader=4,bindAddress=ndimiduk-hbase-1-1-1RC0-1.openstacklocal,port=44787 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:627) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:614) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RpcServer.reader=5,bindAddress=ndimiduk-hbase-1-1-1RC0-1.openstacklocal,port=44787 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:627) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:614) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PriorityRpcServer.handler=7,queue=1,port=44787 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PriorityRpcServer.handler=9,queue=1,port=44787 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PriorityRpcServer.handler=17,queue=1,port=44787 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PriorityRpcServer.handler=11,queue=1,port=44787 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PriorityRpcServer.handler=14,queue=0,port=44787 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RpcServer.reader=0,bindAddress=ndimiduk-hbase-1-1-1RC0-1.openstacklocal,port=44787 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:627) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:614) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PriorityRpcServer.handler=5,queue=1,port=44787 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PriorityRpcServer.handler=15,queue=1,port=44787 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RpcServer.reader=6,bindAddress=ndimiduk-hbase-1-1-1RC0-1.openstacklocal,port=44787 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:627) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:614) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PriorityRpcServer.handler=3,queue=1,port=44787 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PriorityRpcServer.handler=18,queue=0,port=44787 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: Thread-90-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) Potentially hanging thread: Thread-90-SendThread(localhost:59952) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) Potentially hanging thread: PriorityRpcServer.handler=4,queue=0,port=44787 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PriorityRpcServer.handler=6,queue=0,port=44787 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) - Thread LEAK? -, OpenFileDescriptor=344 (was 304) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=32768 (was 32768), SystemLoadAverage=620 (was 620), ProcessCount=100 (was 100), AvailableMemoryMB=5008 (was 5015), ConnectionCount=4 (was 3) - ConnectionCount LEAK? - 2015-06-15 03:03:43,497 INFO [main] hbase.ResourceChecker(147): before: master.TestAssignmentManager#testBalanceOnMasterFailoverScenarioWithOfflineNode Thread=158, OpenFileDescriptor=344, MaxFileDescriptor=32768, SystemLoadAverage=620, ProcessCount=100, AvailableMemoryMB=5008, ConnectionCount=4 2015-06-15 03:03:43,497 INFO [main] zookeeper.RecoverableZooKeeper(121): Process identifier=mockedServer connecting to ZooKeeper ensemble=localhost:59952 2015-06-15 03:03:43,506 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer0x0, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-15 03:03:43,507 DEBUG [main] zookeeper.RecoverableZooKeeper(595): Node /hbase already exists 2015-06-15 03:03:43,508 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(592): mockedServer-0x14df52ac8120018 connected 2015-06-15 03:03:43,531 DEBUG [Thread-122] executor.ExecutorService(99): Starting executor service name=MASTER_OPEN_REGION-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:03:43,531 DEBUG [Thread-122] executor.ExecutorService(99): Starting executor service name=MASTER_CLOSE_REGION-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:03:43,531 DEBUG [Thread-122] executor.ExecutorService(99): Starting executor service name=MASTER_SERVER_OPERATIONS-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:03:43,532 DEBUG [Thread-122] executor.ExecutorService(99): Starting executor service name=MASTER_META_SERVER_OPERATIONS-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:03:43,534 INFO [Thread-122] master.RegionStates(1112): Transition null to {cae5ea3b1868354349837ee2dd0d4d89 state=OPEN, ts=1434337423534, server=example.org,1234,5678} 2015-06-15 03:03:43,535 DEBUG [Thread-122] master.RegionStates(460): Onlined cae5ea3b1868354349837ee2dd0d4d89 on example.org,1234,5678 {ENCODED => cae5ea3b1868354349837ee2dd0d4d89, NAME => 't,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89.', STARTKEY => '', ENDKEY => ''} 2015-06-15 03:03:43,535 DEBUG [Thread-122] master.AssignmentManager(2470): Starting unassign of t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. (offlining), current state: {cae5ea3b1868354349837ee2dd0d4d89 state=OPEN, ts=1434337423534, server=example.org,1234,5678} 2015-06-15 03:03:43,536 DEBUG [Thread-122] zookeeper.ZKAssign(527): mockedServer-0x14df52ac8120018, quorum=localhost:59952, baseZNode=/hbase Creating unassigned node cae5ea3b1868354349837ee2dd0d4d89 in a CLOSING state 2015-06-15 03:03:43,542 DEBUG [Thread-90-EventThread] zookeeper.ZooKeeperWatcher(508): master:44787-0x14df52ac8120017, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-06-15 03:03:43,543 INFO [Thread-122] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=OPEN, ts=1434337423534, server=example.org,1234,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=PENDING_CLOSE, ts=1434337423543, server=example.org,1234,5678} 2015-06-15 03:03:43,544 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,544 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,544 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,545 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,547 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,548 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,548 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,548 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,548 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,550 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,550 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,550 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,551 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,551 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,551 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,552 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,552 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,552 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,552 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,553 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,553 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,553 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,554 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,554 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,554 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,554 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,555 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,555 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,555 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,556 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,556 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,556 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,557 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,557 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,557 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,557 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,558 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,558 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,558 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,559 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,559 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,559 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,559 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,560 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,560 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,560 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,561 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,561 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,561 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,562 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,562 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,563 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,564 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,564 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,564 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,565 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,565 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,565 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,566 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,566 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,566 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,567 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,567 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,567 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,568 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,568 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,568 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,568 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,569 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,569 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,573 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,574 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,574 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,586 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,588 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,596 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,597 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,597 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,598 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,606 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,606 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,607 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,608 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,609 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,609 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,609 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,610 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,610 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,610 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,613 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,615 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,620 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,624 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,625 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,625 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,630 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,630 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,630 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,631 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,633 WARN [Thread-122] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,633 WARN [Thread-122] master.RegionStates(1103): Failed to open/close cae5ea3b1868354349837ee2dd0d4d89 on example.org,1234,5678, set to FAILED_CLOSE 2015-06-15 03:03:43,633 INFO [Thread-122] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=PENDING_CLOSE, ts=1434337423543, server=example.org,1234,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=FAILED_CLOSE, ts=1434337423633, server=example.org,1234,5678} 2015-06-15 03:03:43,663 WARN [RunAmJoinCluster] master.RegionStates(339): Tried to create a state for a region already in RegionStates, used existing: {cae5ea3b1868354349837ee2dd0d4d89 state=FAILED_CLOSE, ts=1434337423633, server=example.org,1234,5678}, ignored new: OPEN 2015-06-15 03:03:43,665 DEBUG [RunAmJoinCluster] master.AssignmentManager(540): Found {ENCODED => cae5ea3b1868354349837ee2dd0d4d89, NAME => 't,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89.', STARTKEY => '', ENDKEY => ''} out on cluster 2015-06-15 03:03:43,665 INFO [RunAmJoinCluster] master.AssignmentManager(623): Found regions out on cluster or in RIT; presuming failover 2015-06-15 03:03:43,666 INFO [Thread-122] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=FAILED_CLOSE, ts=1434337423633, server=example.org,1234,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=PENDING_CLOSE, ts=1434337423666, server=example.org,1234,5678} 2015-06-15 03:03:43,666 DEBUG [Thread-122] zookeeper.ZKAssign(805): mockedServer-0x14df52ac8120018, quorum=localhost:59952, baseZNode=/hbase Transitioning cae5ea3b1868354349837ee2dd0d4d89 from M_ZK_REGION_CLOSING to RS_ZK_REGION_CLOSED 2015-06-15 03:03:43,671 INFO [RunAmJoinCluster] master.AssignmentManager(767): Processing cae5ea3b1868354349837ee2dd0d4d89 in state: M_ZK_REGION_CLOSING 2015-06-15 03:03:43,672 INFO [RunAmJoinCluster] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=PENDING_CLOSE, ts=1434337423666, server=example.org,1234,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=CLOSING, ts=1434337423672, server=example.org,1234,5678} 2015-06-15 03:03:43,673 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer-0x14df52ac8120018, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/cae5ea3b1868354349837ee2dd0d4d89 2015-06-15 03:03:43,673 DEBUG [Thread-122] zookeeper.ZKAssign(880): mockedServer-0x14df52ac8120018, quorum=localhost:59952, baseZNode=/hbase Transitioned node cae5ea3b1868354349837ee2dd0d4d89 from M_ZK_REGION_CLOSING to RS_ZK_REGION_CLOSED 2015-06-15 03:03:43,684 DEBUG [AM.ZK.Worker-pool63-t1] master.AssignmentManager(954): Handling RS_ZK_REGION_CLOSED, server=example.org,1234,5678, region=cae5ea3b1868354349837ee2dd0d4d89, current_state={cae5ea3b1868354349837ee2dd0d4d89 state=CLOSING, ts=1434337423672, server=example.org,1234,5678} 2015-06-15 03:03:43,685 INFO [RunAmJoinCluster] master.AssignmentManager(898): Processed region cae5ea3b1868354349837ee2dd0d4d89 in state M_ZK_REGION_CLOSING, on server: example.org,1234,5678 2015-06-15 03:03:43,691 DEBUG [AM.ZK.Worker-pool63-t1] handler.ClosedRegionHandler(93): Handling CLOSED event for cae5ea3b1868354349837ee2dd0d4d89 2015-06-15 03:03:43,691 INFO [AM.ZK.Worker-pool63-t1] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=CLOSING, ts=1434337423672, server=example.org,1234,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=CLOSED, ts=1434337423691, server=example.org,1234,5678} 2015-06-15 03:03:43,694 INFO [RunAmJoinCluster] master.AssignmentManager(499): Joined the cluster in 60ms, failover=true 2015-06-15 03:03:43,695 DEBUG [AM.ZK.Worker-pool63-t1] master.AssignmentManager(2381): Found an existing plan for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. destination server is example.org,0,5678 accepted as a dest server = true 2015-06-15 03:03:43,695 DEBUG [AM.ZK.Worker-pool63-t1] master.AssignmentManager(2419): Using pre-existing plan for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89.; plan=hri=t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89., src=, dest=example.org,0,5678 2015-06-15 03:03:43,695 INFO [AM.ZK.Worker-pool63-t1] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=CLOSED, ts=1434337423691, server=example.org,1234,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=OFFLINE, ts=1434337423695, server=example.org,1234,5678} 2015-06-15 03:03:43,695 DEBUG [AM.ZK.Worker-pool63-t1] zookeeper.ZKAssign(205): mockedServer-0x14df52ac8120018, quorum=localhost:59952, baseZNode=/hbase Creating (or updating) unassigned node cae5ea3b1868354349837ee2dd0d4d89 with OFFLINE state 2015-06-15 03:03:43,699 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer-0x14df52ac8120018, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/cae5ea3b1868354349837ee2dd0d4d89 2015-06-15 03:03:43,700 INFO [AM.ZK.Worker-pool63-t1] master.AssignmentManager(2122): Assigning t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. to example.org,0,5678 2015-06-15 03:03:43,700 INFO [AM.ZK.Worker-pool63-t1] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=OFFLINE, ts=1434337423695, server=example.org,1234,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=PENDING_OPEN, ts=1434337423700, server=example.org,0,5678} 2015-06-15 03:03:43,702 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,702 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,703 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,703 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,703 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,703 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,704 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,704 DEBUG [Thread-122] zookeeper.ZKAssign(805): mockedServer-0x14df52ac8120018, quorum=localhost:59952, baseZNode=/hbase Transitioning cae5ea3b1868354349837ee2dd0d4d89 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-06-15 03:03:43,708 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,709 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,709 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,709 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,709 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,709 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,710 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,710 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,710 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,710 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,712 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,712 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,712 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,713 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,713 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,714 DEBUG [Thread-122] zookeeper.ZKAssign(880): mockedServer-0x14df52ac8120018, quorum=localhost:59952, baseZNode=/hbase Transitioned node cae5ea3b1868354349837ee2dd0d4d89 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-06-15 03:03:43,714 DEBUG [Thread-122] zookeeper.ZKAssign(805): mockedServer-0x14df52ac8120018, quorum=localhost:59952, baseZNode=/hbase Transitioning cae5ea3b1868354349837ee2dd0d4d89 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2015-06-15 03:03:43,714 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer-0x14df52ac8120018, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/cae5ea3b1868354349837ee2dd0d4d89 2015-06-15 03:03:43,716 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,716 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,717 DEBUG [Thread-122] zookeeper.ZKAssign(880): mockedServer-0x14df52ac8120018, quorum=localhost:59952, baseZNode=/hbase Transitioned node cae5ea3b1868354349837ee2dd0d4d89 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2015-06-15 03:03:43,720 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:43,720 DEBUG [AM.ZK.Worker-pool63-t2] master.AssignmentManager(954): Handling RS_ZK_REGION_OPENED, server=example.org,0,5678, region=cae5ea3b1868354349837ee2dd0d4d89, current_state={cae5ea3b1868354349837ee2dd0d4d89 state=PENDING_OPEN, ts=1434337423700, server=example.org,0,5678} 2015-06-15 03:03:43,721 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,721 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,722 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,722 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,722 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,723 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,723 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,723 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,724 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,724 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,724 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,725 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,731 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,731 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,733 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,734 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,734 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,735 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,736 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,736 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,737 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,737 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,737 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,738 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,738 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,738 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,739 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,739 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,740 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,740 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,741 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,741 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,742 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,743 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,743 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,744 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,744 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,747 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,748 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,748 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,748 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,749 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,749 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,749 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,750 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,750 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,750 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,751 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,752 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,752 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,753 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,753 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,753 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,754 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,754 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,754 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,754 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,755 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,755 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,755 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,756 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,756 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,756 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,757 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,757 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,758 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,758 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,759 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,759 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,759 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,760 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,760 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,760 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,760 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,761 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,761 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,761 WARN [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.RegionStates(1103): Failed to open/close cae5ea3b1868354349837ee2dd0d4d89 on example.org,0,5678, set to FAILED_CLOSE 2015-06-15 03:03:43,761 INFO [MASTER_SERVER_OPERATIONS-mockedAMExecutor-0] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=PENDING_OPEN, ts=1434337423700, server=example.org,0,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=FAILED_CLOSE, ts=1434337423761, server=example.org,0,5678} 2015-06-15 03:03:43,762 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(975): Region state changed from {cae5ea3b1868354349837ee2dd0d4d89 state=PENDING_OPEN, ts=1434337423700, server=example.org,0,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=FAILED_CLOSE, ts=1434337423761, server=example.org,0,5678}, while acquiring lock 2015-06-15 03:03:43,762 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1089): Received OPENED for cae5ea3b1868354349837ee2dd0d4d89 from example.org,0,5678 but the region isn't PENDING_OPEN/OPENING here: {cae5ea3b1868354349837ee2dd0d4d89 state=FAILED_CLOSE, ts=1434337423761, server=example.org,0,5678} 2015-06-15 03:03:43,762 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,762 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,762 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,763 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,763 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,763 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,763 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,764 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,764 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,764 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,765 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,765 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,766 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,767 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,767 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,768 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,768 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,768 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,768 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,769 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,769 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,769 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,770 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,770 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,770 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,771 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,771 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,771 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,772 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,772 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,772 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,773 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,773 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,773 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,774 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,774 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,774 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,774 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,775 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,775 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,776 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,776 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,776 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,776 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,777 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,777 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,778 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,778 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,778 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,779 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,779 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,779 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,780 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,780 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,780 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,780 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,781 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,782 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,783 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,783 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,784 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,784 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,784 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,785 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,785 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,785 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,786 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,786 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,786 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,786 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,787 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,787 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,787 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,788 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,788 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,788 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,788 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,789 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,789 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,789 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,790 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,790 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,791 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,791 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,791 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,792 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,792 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,793 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,793 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,793 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,794 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,794 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,794 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,795 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,795 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,795 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,796 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,796 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,796 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,796 WARN [AM.ZK.Worker-pool63-t2] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,799 DEBUG [AM.ZK.Worker-pool63-t3] master.AssignmentManager(954): Handling RS_ZK_REGION_OPENED, server=example.org,0,5678, region=cae5ea3b1868354349837ee2dd0d4d89, current_state={cae5ea3b1868354349837ee2dd0d4d89 state=FAILED_CLOSE, ts=1434337423761, server=example.org,0,5678} 2015-06-15 03:03:43,799 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1089): Received OPENED for cae5ea3b1868354349837ee2dd0d4d89 from example.org,0,5678 but the region isn't PENDING_OPEN/OPENING here: {cae5ea3b1868354349837ee2dd0d4d89 state=FAILED_CLOSE, ts=1434337423761, server=example.org,0,5678} 2015-06-15 03:03:43,799 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,800 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,800 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,800 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,801 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,801 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,801 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,801 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,802 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,802 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,802 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,803 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,803 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,803 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,803 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,804 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,804 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,804 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,805 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,805 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,805 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,806 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,806 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,806 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,807 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,807 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,807 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,808 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,808 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,808 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,808 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,809 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,809 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,809 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,810 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,810 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,810 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,811 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,811 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,811 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,811 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,812 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,812 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,812 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,813 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,813 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,813 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,813 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,814 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,814 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,814 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,815 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,815 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,815 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,815 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,816 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,816 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,817 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,817 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,817 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,817 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,818 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,819 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,819 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,819 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,819 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,820 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,822 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:43,822 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,823 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,823 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,823 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,824 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,824 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,824 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,825 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,825 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,825 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,826 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,826 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,826 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,828 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,828 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,828 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,828 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,829 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,829 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,829 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,832 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,833 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,833 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,833 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,834 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,834 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,835 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,835 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,835 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,835 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,836 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,836 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,836 WARN [AM.ZK.Worker-pool63-t3] master.AssignmentManager(1866): Server example.org,0,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:03:43,842 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 22668 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:03:43,924 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:44,026 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:44,129 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:44,232 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:44,334 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:44,437 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:44,539 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:44,641 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:44,743 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:44,846 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:44,948 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:45,049 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:45,152 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:45,255 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:45,348 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 24174 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:03:45,357 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:45,459 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:45,562 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:45,664 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:45,767 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:45,870 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:45,973 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:46,077 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:46,183 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:46,290 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:46,394 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:46,497 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:46,599 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:46,701 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:46,803 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:46,863 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 25689 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:03:46,907 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:47,009 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:47,111 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:47,213 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:47,316 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:47,418 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:47,521 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:47,624 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:47,726 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:47,830 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:47,934 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:48,036 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:48,141 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:48,243 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:48,347 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:48,373 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 27199 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:03:48,449 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:48,553 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:48,658 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:48,760 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:48,864 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:48,967 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:49,071 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:49,173 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:49,276 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:49,379 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:49,482 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:49,586 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:49,689 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:49,791 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:49,878 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 28704 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:03:49,894 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:49,997 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:50,100 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:50,202 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:50,305 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:50,408 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:50,511 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:50,614 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:50,717 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:50,820 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:50,922 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:51,024 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:51,112 DEBUG [ProcedureExecutorTimeoutThread] procedure2.ProcedureExecutor$CompletedProcedureCleaner(152): No completed procedures to cleanup. 2015-06-15 03:03:51,126 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:51,229 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:51,331 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:51,385 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 30211 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:03:51,433 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:51,535 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:51,638 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:51,740 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:51,842 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:51,945 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:52,047 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:52,150 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:52,253 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:52,356 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:52,459 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:52,563 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:52,667 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:52,773 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:52,876 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:52,893 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 31719 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:03:52,978 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:53,081 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:53,184 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:53,287 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:53,391 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:53,495 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:53,598 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:53,700 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:53,803 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:53,906 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:54,008 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:54,114 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:54,217 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:54,319 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:54,401 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 33227 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:03:54,420 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:54,523 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:54,625 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:54,727 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:54,831 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:54,933 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:55,036 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:55,139 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:55,243 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:55,346 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:55,448 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:55,551 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:55,654 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:55,756 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:55,859 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:55,912 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 34738 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:03:55,962 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:56,065 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:56,167 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:56,269 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:56,372 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:56,476 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:56,580 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:56,683 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:56,784 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:56,886 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:56,988 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:57,092 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:57,195 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:57,298 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:57,401 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:57,418 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 36244 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:03:57,502 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:57,607 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:57,709 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:57,811 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:57,913 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:58,016 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:58,120 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:58,223 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:58,326 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:58,430 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:58,532 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:58,635 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:58,738 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:58,842 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:58,926 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 37752 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:03:58,945 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:59,048 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:59,150 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:59,254 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:59,357 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:59,460 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:59,563 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:59,666 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:59,769 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:59,872 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:03:59,975 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:00,078 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:00,182 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:00,285 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:00,388 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:00,432 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 39258 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:04:00,496 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:00,598 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:00,701 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:00,806 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:00,909 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:01,012 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:01,115 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:01,218 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:01,321 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:01,424 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:01,527 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:01,629 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:01,731 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:01,835 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:01,938 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:01,938 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 40764 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:04:02,040 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:02,144 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:02,247 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:02,350 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:02,452 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:02,554 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:02,656 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:02,758 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:02,860 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:02,962 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:03,063 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:03,165 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:03,267 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:03,369 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:03,442 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 42268 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:04:03,471 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:03,573 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:03,674 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:03,777 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:03,878 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:03,980 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:04,082 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:04,184 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:04,285 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:04,387 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:04,488 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:04,590 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:04,691 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:04,793 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:04,895 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:04,946 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 43771 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:04:04,997 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:05,099 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:05,200 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:05,302 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:05,404 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:05,507 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:05,609 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:05,711 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:05,813 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:05,914 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:06,016 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:06,117 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:06,219 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:06,320 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:06,422 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:06,452 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 45278 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:04:06,523 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:06,625 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:06,727 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:06,828 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:06,930 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:07,032 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:07,134 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:07,235 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:07,338 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:07,439 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:07,541 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:07,644 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:07,745 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:07,847 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:07,950 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:07,959 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 46785 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:04:08,052 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:08,153 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:08,257 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:08,361 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:08,462 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:08,564 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:08,669 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:08,771 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:08,872 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:08,974 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:09,075 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:09,177 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:09,279 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:09,382 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:09,462 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 48288 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:04:09,484 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:09,586 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:09,688 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:09,790 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:09,892 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:09,996 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:10,098 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:10,202 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:10,304 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:10,406 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:10,508 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:10,610 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:10,712 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:10,814 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:10,919 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:10,978 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 49804 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:04:11,021 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:11,124 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:11,226 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:11,330 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:11,433 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:11,536 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:11,638 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:11,741 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:11,845 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:11,946 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:12,052 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:12,154 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:12,256 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:12,358 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:12,460 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:12,483 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 51309 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:04:12,564 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:12,665 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:12,767 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:12,877 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:12,978 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:13,080 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:13,182 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:13,283 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:13,387 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:13,488 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:13,592 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:13,695 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:13,796 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:13,898 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:13,990 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 52816 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:04:13,999 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:14,101 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:14,203 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:14,305 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:14,406 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:14,507 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:14,609 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:14,710 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:14,812 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:14,914 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:15,016 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:15,117 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:15,219 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:15,321 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:15,422 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:15,502 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 54328 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:04:15,524 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:15,625 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:15,727 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:15,828 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:15,931 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:16,033 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:16,206 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:16,310 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:16,413 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:16,515 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:16,618 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:16,720 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:16,821 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:16,924 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:17,011 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 55837 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:04:17,026 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:17,127 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:17,232 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:17,334 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:17,437 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:17,540 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:17,641 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:17,743 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:17,844 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:17,946 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:18,048 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:18,152 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:18,253 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:18,356 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:18,458 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:18,517 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 57343 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:04:18,559 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:18,661 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:18,762 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:18,864 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:18,965 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:19,071 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:19,173 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:19,274 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:19,378 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:19,480 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:19,581 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:19,683 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:19,785 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:19,886 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:19,990 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:20,022 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 58848 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:04:20,092 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:20,194 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:20,295 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:20,397 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:20,498 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:20,600 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:20,702 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:20,803 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:20,905 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:21,007 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:21,108 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:21,113 DEBUG [ProcedureExecutorTimeoutThread] procedure2.ProcedureExecutor$CompletedProcedureCleaner(152): No completed procedures to cleanup. 2015-06-15 03:04:21,186 DEBUG [ndimiduk-hbase-1-1-1RC0-1.openstacklocal,60005,1434337400084_ChoreService_1] master.ReplicationLogCleaner$1(74): Didn't find this log in ZK, deleting: hregion-91709255.default.1434313922177 2015-06-15 03:04:21,187 DEBUG [ndimiduk-hbase-1-1-1RC0-1.openstacklocal,60005,1434337400084_ChoreService_1] cleaner.CleanerChore(253): Removing: file:/tmp/hbase-root/hbase/oldWALs/hregion-91709255.default.1434313922177 from archive 2015-06-15 03:04:21,210 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:21,222 DEBUG [ndimiduk-hbase-1-1-1RC0-1.openstacklocal,60005,1434337400084_ChoreService_1] cleaner.CleanerChore(253): Removing: file:/tmp/hbase-root/hbase/archive/data/default/table/7a5555a1f99fa3d2a506762f4772bfc1/family/252a7cd06f424165b2a01319f352092e from archive 2015-06-15 03:04:21,223 DEBUG [ndimiduk-hbase-1-1-1RC0-1.openstacklocal,60005,1434337400084_ChoreService_1] cleaner.CleanerChore(253): Removing: file:/tmp/hbase-root/hbase/archive/data/default/table/7a5555a1f99fa3d2a506762f4772bfc1/family/08685bdcd9c149a4b462b93c2d748bb5 from archive 2015-06-15 03:04:21,223 DEBUG [ndimiduk-hbase-1-1-1RC0-1.openstacklocal,60005,1434337400084_ChoreService_1] cleaner.CleanerChore(253): Removing: file:/tmp/hbase-root/hbase/archive/data/default/table/7a5555a1f99fa3d2a506762f4772bfc1/family/219f5716772840c885abe37e2c6faa78 from archive 2015-06-15 03:04:21,224 DEBUG [ndimiduk-hbase-1-1-1RC0-1.openstacklocal,60005,1434337400084_ChoreService_1] cleaner.CleanerChore(253): Removing: file:/tmp/hbase-root/hbase/archive/data/default/table/7a5555a1f99fa3d2a506762f4772bfc1/family/ebb9a3a2a8624adb8f7fc5d54dce2245 from archive 2015-06-15 03:04:21,237 DEBUG [ndimiduk-hbase-1-1-1RC0-1.openstacklocal,60005,1434337400084_ChoreService_1] cleaner.CleanerChore(253): Removing: file:/tmp/hbase-root/hbase/archive/data/default/table/5226d17e7147ff347803403cb7d78f58/family/c0ebc84e70c1450d913b36af13c64e76 from archive 2015-06-15 03:04:21,251 DEBUG [ndimiduk-hbase-1-1-1RC0-1.openstacklocal,60005,1434337400084_ChoreService_1] cleaner.CleanerChore(253): Removing: file:/tmp/hbase-root/hbase/archive/data/default/table/5226d17e7147ff347803403cb7d78f58/family/4b51d77a87684ced8ed75161cf44e1da from archive 2015-06-15 03:04:21,251 DEBUG [ndimiduk-hbase-1-1-1RC0-1.openstacklocal,60005,1434337400084_ChoreService_1] cleaner.CleanerChore(253): Removing: file:/tmp/hbase-root/hbase/archive/data/default/table/5226d17e7147ff347803403cb7d78f58/family/a5b1457cc60a4986b550c6ffd1283c09 from archive 2015-06-15 03:04:21,252 DEBUG [ndimiduk-hbase-1-1-1RC0-1.openstacklocal,60005,1434337400084_ChoreService_1] cleaner.CleanerChore(253): Removing: file:/tmp/hbase-root/hbase/archive/data/default/table/5226d17e7147ff347803403cb7d78f58/family/4eb1dc59f52b4a36a7eac3b1ba3f1822 from archive 2015-06-15 03:04:21,253 DEBUG [ndimiduk-hbase-1-1-1RC0-1.openstacklocal,60005,1434337400084_ChoreService_1] cleaner.CleanerChore(253): Removing: file:/tmp/hbase-root/hbase/archive/data/default/table/5226d17e7147ff347803403cb7d78f58/family/89188eb869574182948a16049839569e from archive 2015-06-15 03:04:21,257 DEBUG [ndimiduk-hbase-1-1-1RC0-1.openstacklocal,60005,1434337400084_ChoreService_1] cleaner.CleanerChore(253): Removing: file:/tmp/hbase-root/hbase/archive/data/default/table/3d9b4260e8571ed1bbe1c6e4c7b685b5/family/2cf5428a85e04e208ff32e232f122b1a from archive 2015-06-15 03:04:21,272 DEBUG [ndimiduk-hbase-1-1-1RC0-1.openstacklocal,60005,1434337400084_ChoreService_1] cleaner.CleanerChore(253): Removing: file:/tmp/hbase-root/hbase/archive/data/default/table/3d9b4260e8571ed1bbe1c6e4c7b685b5/family/2b7915cd7ef74003ab54fe4b3e57da63 from archive 2015-06-15 03:04:21,272 DEBUG [ndimiduk-hbase-1-1-1RC0-1.openstacklocal,60005,1434337400084_ChoreService_1] cleaner.CleanerChore(253): Removing: file:/tmp/hbase-root/hbase/archive/data/default/table/3d9b4260e8571ed1bbe1c6e4c7b685b5/family/f04ef9f330454affa37af88b8e15bfec from archive 2015-06-15 03:04:21,288 DEBUG [ndimiduk-hbase-1-1-1RC0-1.openstacklocal,60005,1434337400084_ChoreService_1] cleaner.CleanerChore(253): Removing: file:/tmp/hbase-root/hbase/archive/data/default/table/fbcbc43564262921618704db6094446c/family/79ff6fadf07848f1b64e6d0c9f8707c9 from archive 2015-06-15 03:04:21,291 DEBUG [ndimiduk-hbase-1-1-1RC0-1.openstacklocal,60005,1434337400084_ChoreService_1] cleaner.CleanerChore(253): Removing: file:/tmp/hbase-root/hbase/archive/data/default/table/fbcbc43564262921618704db6094446c/family/20993bb0b2a2401885d37692d95373fa from archive 2015-06-15 03:04:21,291 DEBUG [ndimiduk-hbase-1-1-1RC0-1.openstacklocal,60005,1434337400084_ChoreService_1] cleaner.CleanerChore(253): Removing: file:/tmp/hbase-root/hbase/archive/data/default/table/fbcbc43564262921618704db6094446c/family/c9c6410b79b340b4aa8ba842cf5ac6a2 from archive 2015-06-15 03:04:21,292 DEBUG [ndimiduk-hbase-1-1-1RC0-1.openstacklocal,60005,1434337400084_ChoreService_1] cleaner.CleanerChore(253): Removing: file:/tmp/hbase-root/hbase/archive/data/default/table/fbcbc43564262921618704db6094446c/family/98659b56fe524f50a95631b2614b91c4 from archive 2015-06-15 03:04:21,293 DEBUG [ndimiduk-hbase-1-1-1RC0-1.openstacklocal,60005,1434337400084_ChoreService_1] cleaner.CleanerChore(253): Removing: file:/tmp/hbase-root/hbase/archive/data/default/table/fbcbc43564262921618704db6094446c/family/3bc4cdc20bbe419796f4732959440dc1 from archive 2015-06-15 03:04:21,299 DEBUG [ndimiduk-hbase-1-1-1RC0-1.openstacklocal,60005,1434337400084_ChoreService_1] cleaner.CleanerChore(253): Removing: file:/tmp/hbase-root/hbase/archive/data/default/table/4a9d39b39870d6da4c7587060ad4308b/family/c3903dec4b6444d3a2c0951761938264 from archive 2015-06-15 03:04:21,300 DEBUG [ndimiduk-hbase-1-1-1RC0-1.openstacklocal,60005,1434337400084_ChoreService_1] cleaner.CleanerChore(253): Removing: file:/tmp/hbase-root/hbase/archive/data/default/table/4a9d39b39870d6da4c7587060ad4308b/family/53204a35fe2449b1ae39cf52106aa95d from archive 2015-06-15 03:04:21,301 DEBUG [ndimiduk-hbase-1-1-1RC0-1.openstacklocal,60005,1434337400084_ChoreService_1] cleaner.CleanerChore(253): Removing: file:/tmp/hbase-root/hbase/archive/data/default/table/4a9d39b39870d6da4c7587060ad4308b/family/f2916bbcfa764f2385a95f5ce994a895 from archive 2015-06-15 03:04:21,302 DEBUG [ndimiduk-hbase-1-1-1RC0-1.openstacklocal,60005,1434337400084_ChoreService_1] cleaner.CleanerChore(253): Removing: file:/tmp/hbase-root/hbase/archive/data/default/table/4a9d39b39870d6da4c7587060ad4308b/family/390bb36a04f64344b4dac22441169197 from archive 2015-06-15 03:04:21,312 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:21,314 DEBUG [ndimiduk-hbase-1-1-1RC0-1.openstacklocal,60005,1434337400084_ChoreService_1] cleaner.CleanerChore(253): Removing: file:/tmp/hbase-root/hbase/archive/data/default/table/ec514b03f44987467497f340e8f6817b/family/337f77fcbe1945398aaaabacc58e9f16 from archive 2015-06-15 03:04:21,314 DEBUG [ndimiduk-hbase-1-1-1RC0-1.openstacklocal,60005,1434337400084_ChoreService_1] cleaner.CleanerChore(253): Removing: file:/tmp/hbase-root/hbase/archive/data/default/table/ec514b03f44987467497f340e8f6817b/family/1d7d802bdf1e4ac18d45709657be720f from archive 2015-06-15 03:04:21,315 DEBUG [ndimiduk-hbase-1-1-1RC0-1.openstacklocal,60005,1434337400084_ChoreService_1] cleaner.CleanerChore(253): Removing: file:/tmp/hbase-root/hbase/archive/data/default/table/ec514b03f44987467497f340e8f6817b/family/d3388f6395564c1fa08e0603f8e8f8ce from archive 2015-06-15 03:04:21,413 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:21,515 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:21,526 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 60351 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:04:21,616 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:21,718 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:21,819 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:21,921 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:22,022 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:22,124 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:22,225 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:22,327 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:22,429 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:22,530 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:22,631 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:22,733 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:22,835 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:22,936 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:23,030 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 61856 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:04:23,039 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:23,141 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:23,244 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:23,346 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:23,447 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:23,549 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:23,650 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:23,754 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:23,855 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:23,957 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:24,058 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:24,160 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:24,262 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:24,363 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:24,465 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:24,542 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 63368 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:04:24,567 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:24,668 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:24,770 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:24,871 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:24,973 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:25,074 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:25,176 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:25,278 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:25,379 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:25,481 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:25,583 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:25,684 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:25,786 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:25,887 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:25,989 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:26,047 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 64873 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:04:26,090 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:26,192 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:26,293 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:26,398 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:26,499 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:26,601 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:26,703 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:26,805 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:26,906 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:27,011 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:27,113 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:27,215 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:27,316 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:27,418 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:27,519 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:27,558 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 66384 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:04:27,621 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:27,722 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:27,824 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:27,925 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:28,027 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:28,128 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:28,230 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:28,331 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:28,435 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:28,536 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:28,638 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:28,740 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:28,843 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:28,946 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:29,047 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:29,065 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 67891 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:04:29,153 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:29,254 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:29,359 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:29,461 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:29,562 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:29,666 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:29,768 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:29,870 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:29,971 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:30,073 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:30,174 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:30,276 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:30,377 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:30,479 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:30,570 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 69396 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:04:30,580 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:30,682 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:30,783 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:30,885 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:30,987 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:31,088 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:31,190 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:31,292 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:31,394 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:31,495 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:31,598 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:31,700 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:31,803 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:31,904 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:32,006 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:32,076 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 70902 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:04:32,109 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:32,211 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:32,313 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:32,415 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:32,517 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:32,620 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:32,722 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:32,824 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:32,925 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:33,027 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:33,128 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:33,231 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:33,332 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:33,435 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:33,537 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:33,585 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 72411 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:04:33,639 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:33,740 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:33,843 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:33,948 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:34,050 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:34,155 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:34,257 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:34,359 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:34,464 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:34,565 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:34,669 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:34,771 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:34,872 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:34,974 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:35,077 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:35,098 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 73924 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:04:35,178 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:35,280 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:35,382 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:35,483 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:35,585 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:35,686 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:35,787 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:35,889 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:35,990 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:36,092 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:36,193 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:36,299 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:36,401 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:36,502 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:36,604 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:36,608 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 75434 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:04:36,705 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:36,808 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:36,909 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:37,011 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:37,112 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:37,214 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:37,316 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:37,417 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:37,520 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:37,621 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:37,722 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:37,824 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:37,928 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:38,033 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:38,117 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 76943 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:04:38,135 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:38,236 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:38,338 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:38,441 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:38,544 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:38,646 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:38,750 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:38,851 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:38,952 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:39,054 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:39,155 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:39,257 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:39,358 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:39,463 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:39,565 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:39,629 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 78455 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:04:39,667 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:39,768 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:39,870 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:39,971 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:40,072 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:40,182 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:40,284 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:40,386 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:40,487 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:40,588 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:40,690 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:40,792 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:40,893 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:40,995 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:41,096 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:41,135 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 79961 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:04:41,197 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:41,299 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:41,401 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:41,503 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:41,604 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:41,705 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:41,809 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:41,914 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:42,016 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:42,117 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:42,219 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:42,321 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:42,423 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:42,524 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:42,627 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:42,647 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 81473 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:04:42,729 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:42,831 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:42,933 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:43,035 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:43,137 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:43,239 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:43,341 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:43,443 DEBUG [Thread-122] zookeeper.ZKAssign(1016): Waiting on RIT: [cae5ea3b1868354349837ee2dd0d4d89] 2015-06-15 03:04:43,518 DEBUG [main] zookeeper.ZKAssign(497): mockedServer-0x14df52ac8120018, quorum=localhost:59952, baseZNode=/hbase Deleting any existing unassigned nodes 2015-06-15 03:04:43,524 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer-0x14df52ac8120018, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/cae5ea3b1868354349837ee2dd0d4d89 2015-06-15 03:04:43,524 ERROR [main-EventThread] zookeeper.ClientCnxn$EventThread(524): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@ff8d5a7 rejected from java.util.concurrent.ThreadPoolExecutor@1992cf22[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110) at org.apache.hadoop.hbase.master.AssignmentManager.zkEventWorkersSubmit(AssignmentManager.java:1291) at org.apache.hadoop.hbase.master.AssignmentManager.nodeDeleted(AssignmentManager.java:1322) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:532) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) 2015-06-15 03:04:43,525 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer-0x14df52ac8120018, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-06-15 03:04:43,525 ERROR [main-EventThread] zookeeper.ClientCnxn$EventThread(524): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1febda1f rejected from java.util.concurrent.ThreadPoolExecutor@1992cf22[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110) at org.apache.hadoop.hbase.master.AssignmentManager.nodeChildrenChanged(AssignmentManager.java:1421) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:546) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) 2015-06-15 03:04:43,605 INFO [main] hbase.ResourceChecker(171): after: master.TestAssignmentManager#testBalanceOnMasterFailoverScenarioWithOfflineNode Thread=157 (was 158), OpenFileDescriptor=344 (was 344), MaxFileDescriptor=32768 (was 32768), SystemLoadAverage=421 (was 620), ProcessCount=98 (was 100), AvailableMemoryMB=5983 (was 5008) - AvailableMemoryMB LEAK? -, ConnectionCount=4 (was 4) 2015-06-15 03:04:43,609 INFO [main] zookeeper.MiniZooKeeperCluster(316): Shutdown MiniZK cluster with all ZK servers 2015-06-15 03:04:43,706 DEBUG [Thread-14-EventThread] zookeeper.ZooKeeperWatcher(508): master:60005-0x14df52ac812000a, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2015-06-15 03:04:43,706 DEBUG [Thread-55-EventThread] zookeeper.ZooKeeperWatcher(508): master:55682-0x14df52ac8120012, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2015-06-15 03:04:43,707 DEBUG [Thread-55-EventThread] zookeeper.ZooKeeperWatcher(597): master:55682-0x14df52ac8120012, quorum=localhost:59952, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring 2015-06-15 03:04:43,706 DEBUG [Thread-14-EventThread] zookeeper.ZooKeeperWatcher(597): master:60005-0x14df52ac812000a, quorum=localhost:59952, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring 2015-06-15 03:04:43,707 DEBUG [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(508): hconnection-0x6cec66ea-0x14df52ac812000c, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2015-06-15 03:04:43,707 DEBUG [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(508): replicationLogCleaner-0x14df52ac812000e, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2015-06-15 03:04:43,708 DEBUG [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(597): replicationLogCleaner-0x14df52ac812000e, quorum=localhost:59952, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring 2015-06-15 03:04:43,707 DEBUG [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(597): hconnection-0x6cec66ea-0x14df52ac812000c, quorum=localhost:59952, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring 2015-06-15 03:04:43,708 DEBUG [Thread-90-EventThread] zookeeper.ZooKeeperWatcher(508): master:44787-0x14df52ac8120017, quorum=localhost:59952, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2015-06-15 03:04:43,708 DEBUG [Thread-90-EventThread] zookeeper.ZooKeeperWatcher(597): master:44787-0x14df52ac8120017, quorum=localhost:59952, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring 2015-06-15 03:04:43,708 INFO [main] zookeeper.MiniZooKeeperCluster(273): Started MiniZooKeeperCluster and ran successful 'stat' on client port=52025 2015-06-15 03:04:43,726 INFO [main] hbase.ResourceChecker(147): before: master.TestAssignmentManager#testBalanceOnMasterFailoverScenarioWithOfflineNode Thread=159, OpenFileDescriptor=335, MaxFileDescriptor=32768, SystemLoadAverage=421, ProcessCount=98, AvailableMemoryMB=5980, ConnectionCount=4 2015-06-15 03:04:43,727 INFO [main] zookeeper.RecoverableZooKeeper(121): Process identifier=mockedServer connecting to ZooKeeper ensemble=localhost:52025 2015-06-15 03:04:43,739 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer0x0, quorum=localhost:52025, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-15 03:04:43,740 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(592): mockedServer-0x14df52d1ba50000 connected 2015-06-15 03:04:43,741 DEBUG [main] zookeeper.RecoverableZooKeeper(595): Node /hbase already exists 2015-06-15 03:04:43,802 DEBUG [Thread-124] executor.ExecutorService(99): Starting executor service name=MASTER_OPEN_REGION-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:04:43,807 DEBUG [Thread-124] executor.ExecutorService(99): Starting executor service name=MASTER_CLOSE_REGION-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:04:43,807 DEBUG [Thread-124] executor.ExecutorService(99): Starting executor service name=MASTER_SERVER_OPERATIONS-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:04:43,808 DEBUG [Thread-124] executor.ExecutorService(99): Starting executor service name=MASTER_META_SERVER_OPERATIONS-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:04:43,818 INFO [Thread-124] master.RegionStates(1112): Transition null to {cae5ea3b1868354349837ee2dd0d4d89 state=OPEN, ts=1434337483818, server=example.org,1234,5678} 2015-06-15 03:04:43,818 DEBUG [Thread-124] master.RegionStates(460): Onlined cae5ea3b1868354349837ee2dd0d4d89 on example.org,1234,5678 {ENCODED => cae5ea3b1868354349837ee2dd0d4d89, NAME => 't,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89.', STARTKEY => '', ENDKEY => ''} 2015-06-15 03:04:43,818 DEBUG [Thread-124] master.AssignmentManager(2470): Starting unassign of t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. (offlining), current state: {cae5ea3b1868354349837ee2dd0d4d89 state=OPEN, ts=1434337483818, server=example.org,1234,5678} 2015-06-15 03:04:43,819 DEBUG [Thread-124] zookeeper.ZKAssign(527): mockedServer-0x14df52d1ba50000, quorum=localhost:52025, baseZNode=/hbase Creating unassigned node cae5ea3b1868354349837ee2dd0d4d89 in a CLOSING state 2015-06-15 03:04:43,825 INFO [Thread-124] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=OPEN, ts=1434337483818, server=example.org,1234,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=PENDING_CLOSE, ts=1434337483825, server=example.org,1234,5678} 2015-06-15 03:04:43,825 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,826 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,826 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,831 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,833 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,835 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,835 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,835 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,836 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,836 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,836 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,840 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,840 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,840 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,841 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,841 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,841 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,841 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,842 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,844 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,844 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,845 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,846 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,863 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,864 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,864 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,869 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,869 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,869 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,870 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,870 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,870 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,870 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,871 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,871 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,871 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,874 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,875 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,876 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,876 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,878 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,880 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,880 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,881 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,882 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,883 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,883 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,885 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,885 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,885 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,886 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,886 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,887 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,889 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,889 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,889 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,890 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,890 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,893 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,893 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,893 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,894 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,894 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,894 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,895 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,895 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,895 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,899 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,899 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,900 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,900 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,900 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,900 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,901 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,901 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,904 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,906 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,909 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,912 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,913 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,913 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,915 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,915 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,915 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,916 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,916 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,918 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,919 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,928 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,928 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,928 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,929 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,943 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,947 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,948 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,948 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,953 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,955 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,957 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,962 WARN [Thread-124] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:04:43,963 WARN [Thread-124] master.RegionStates(1103): Failed to open/close cae5ea3b1868354349837ee2dd0d4d89 on example.org,1234,5678, set to FAILED_CLOSE 2015-06-15 03:04:43,963 INFO [Thread-124] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=PENDING_CLOSE, ts=1434337483825, server=example.org,1234,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=FAILED_CLOSE, ts=1434337483963, server=example.org,1234,5678} 2015-06-15 03:04:43,985 ERROR [RunAmJoinCluster] server.NIOServerCnxnFactory$1(44): Thread Thread[RunAmJoinCluster,5,main] died java.lang.NullPointerException at org.apache.hadoop.hbase.MetaTableAccessor.fullScan(MetaTableAccessor.java:611) at org.apache.hadoop.hbase.MetaTableAccessor.fullScanOfMeta(MetaTableAccessor.java:143) at org.apache.hadoop.hbase.master.AssignmentManager.rebuildUserRegions(AssignmentManager.java:2923) at org.apache.hadoop.hbase.master.AssignmentManager.joinCluster(AssignmentManager.java:486) at org.apache.hadoop.hbase.master.TestAssignmentManager$4.run(TestAssignmentManager.java:1372) 2015-06-15 03:04:44,158 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 82984 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:04:45,667 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 84493 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:04:47,181 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 86007 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:04:48,690 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 87516 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:04:50,200 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 89026 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:04:51,113 DEBUG [ProcedureExecutorTimeoutThread] procedure2.ProcedureExecutor$CompletedProcedureCleaner(152): No completed procedures to cleanup. 2015-06-15 03:04:51,709 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 90535 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:04:53,218 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 92044 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:04:54,725 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 93551 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:04:56,240 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 95065 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:04:57,746 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 96572 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:04:59,254 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 98080 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:05:00,760 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 99586 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:05:02,267 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 101093 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:05:03,772 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 102598 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:05:05,275 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 104101 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:05:06,781 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 105607 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:05:08,286 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 107112 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:05:09,793 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 108619 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:05:11,298 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 110124 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:05:12,808 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 111633 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:05:14,321 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 113147 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:05:15,832 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 114657 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:05:17,346 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 116172 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:05:18,857 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 117683 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:05:20,367 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 119193 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:05:21,114 DEBUG [ProcedureExecutorTimeoutThread] procedure2.ProcedureExecutor$CompletedProcedureCleaner(152): No completed procedures to cleanup. 2015-06-15 03:05:21,877 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 120703 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:05:23,392 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 122218 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:05:24,901 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 123727 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:05:26,417 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 125242 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:05:27,926 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 126752 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:05:29,434 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 128260 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:05:30,942 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 129768 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:05:32,455 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 131281 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:05:33,964 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 132789 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:05:35,468 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 134294 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:05:36,971 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 135797 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:05:38,477 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 137303 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:05:39,983 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 138809 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:05:41,488 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 140314 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:05:42,994 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 141820 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:05:43,791 DEBUG [main] zookeeper.ZKAssign(497): mockedServer-0x14df52d1ba50000, quorum=localhost:52025, baseZNode=/hbase Deleting any existing unassigned nodes 2015-06-15 03:05:43,795 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer-0x14df52d1ba50000, quorum=localhost:52025, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/cae5ea3b1868354349837ee2dd0d4d89 2015-06-15 03:05:43,796 ERROR [main-EventThread] zookeeper.ClientCnxn$EventThread(524): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@64e151fd rejected from java.util.concurrent.ThreadPoolExecutor@4cdecc9[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110) at org.apache.hadoop.hbase.master.AssignmentManager.zkEventWorkersSubmit(AssignmentManager.java:1291) at org.apache.hadoop.hbase.master.AssignmentManager.nodeDeleted(AssignmentManager.java:1322) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:532) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) 2015-06-15 03:05:43,847 INFO [main] hbase.ResourceChecker(171): after: master.TestAssignmentManager#testBalanceOnMasterFailoverScenarioWithOfflineNode Thread=158 (was 159), OpenFileDescriptor=339 (was 335) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=32768 (was 32768), SystemLoadAverage=442 (was 421) - SystemLoadAverage LEAK? -, ProcessCount=98 (was 98), AvailableMemoryMB=6424 (was 5980) - AvailableMemoryMB LEAK? -, ConnectionCount=5 (was 4) - ConnectionCount LEAK? - 2015-06-15 03:05:43,850 INFO [main] zookeeper.MiniZooKeeperCluster(316): Shutdown MiniZK cluster with all ZK servers 2015-06-15 03:05:43,974 INFO [main] zookeeper.MiniZooKeeperCluster(273): Started MiniZooKeeperCluster and ran successful 'stat' on client port=50250 2015-06-15 03:05:44,015 INFO [main] hbase.ResourceChecker(147): before: master.TestAssignmentManager#testBalanceOnMasterFailoverScenarioWithOfflineNode Thread=159, OpenFileDescriptor=335, MaxFileDescriptor=32768, SystemLoadAverage=442, ProcessCount=98, AvailableMemoryMB=6408, ConnectionCount=5 2015-06-15 03:05:44,016 INFO [main] zookeeper.RecoverableZooKeeper(121): Process identifier=mockedServer connecting to ZooKeeper ensemble=localhost:50250 2015-06-15 03:05:44,040 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer0x0, quorum=localhost:50250, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-15 03:05:44,041 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(592): mockedServer-0x14df52e06fa0000 connected 2015-06-15 03:05:44,046 DEBUG [main] zookeeper.RecoverableZooKeeper(595): Node /hbase already exists 2015-06-15 03:05:44,064 DEBUG [Thread-126] executor.ExecutorService(99): Starting executor service name=MASTER_OPEN_REGION-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:05:44,070 DEBUG [Thread-126] executor.ExecutorService(99): Starting executor service name=MASTER_CLOSE_REGION-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:05:44,071 DEBUG [Thread-126] executor.ExecutorService(99): Starting executor service name=MASTER_SERVER_OPERATIONS-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:05:44,072 DEBUG [Thread-126] executor.ExecutorService(99): Starting executor service name=MASTER_META_SERVER_OPERATIONS-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:05:44,079 INFO [Thread-126] master.RegionStates(1112): Transition null to {cae5ea3b1868354349837ee2dd0d4d89 state=OPEN, ts=1434337544079, server=example.org,1234,5678} 2015-06-15 03:05:44,092 DEBUG [Thread-126] master.RegionStates(460): Onlined cae5ea3b1868354349837ee2dd0d4d89 on example.org,1234,5678 {ENCODED => cae5ea3b1868354349837ee2dd0d4d89, NAME => 't,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89.', STARTKEY => '', ENDKEY => ''} 2015-06-15 03:05:44,092 DEBUG [Thread-126] master.AssignmentManager(2470): Starting unassign of t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. (offlining), current state: {cae5ea3b1868354349837ee2dd0d4d89 state=OPEN, ts=1434337544079, server=example.org,1234,5678} 2015-06-15 03:05:44,092 DEBUG [Thread-126] zookeeper.ZKAssign(527): mockedServer-0x14df52e06fa0000, quorum=localhost:50250, baseZNode=/hbase Creating unassigned node cae5ea3b1868354349837ee2dd0d4d89 in a CLOSING state 2015-06-15 03:05:44,095 INFO [Thread-126] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=OPEN, ts=1434337544079, server=example.org,1234,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=PENDING_CLOSE, ts=1434337544095, server=example.org,1234,5678} 2015-06-15 03:05:44,095 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,096 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,096 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,096 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,097 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,098 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,098 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,099 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,099 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,103 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,103 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,103 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,104 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,107 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,107 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,108 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,112 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,112 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,112 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,112 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,113 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,115 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,115 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,115 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,115 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,116 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,121 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,128 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,128 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,128 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,129 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,129 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,129 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,129 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,130 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,133 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,133 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,134 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,134 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,134 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,134 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,135 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,135 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,135 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,135 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,136 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,137 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,137 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,140 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,140 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,141 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,141 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,141 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,141 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,142 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,142 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,142 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,142 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,143 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,143 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,144 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,144 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,144 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,146 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,146 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,147 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,147 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,147 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,147 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,148 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,148 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,148 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,149 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,149 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,149 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,150 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,150 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,150 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,151 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,151 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,151 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,151 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,152 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,152 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,152 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,152 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,152 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,153 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,161 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,164 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,167 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,168 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,168 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,169 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,169 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,170 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,170 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,170 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,171 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,171 WARN [Thread-126] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:05:44,172 WARN [Thread-126] master.RegionStates(1103): Failed to open/close cae5ea3b1868354349837ee2dd0d4d89 on example.org,1234,5678, set to FAILED_CLOSE 2015-06-15 03:05:44,172 INFO [Thread-126] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=PENDING_CLOSE, ts=1434337544095, server=example.org,1234,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=FAILED_CLOSE, ts=1434337544172, server=example.org,1234,5678} 2015-06-15 03:05:44,181 ERROR [RunAmJoinCluster] server.NIOServerCnxnFactory$1(44): Thread Thread[RunAmJoinCluster,5,main] died java.lang.NullPointerException at org.apache.hadoop.hbase.MetaTableAccessor.fullScan(MetaTableAccessor.java:611) at org.apache.hadoop.hbase.MetaTableAccessor.fullScanOfMeta(MetaTableAccessor.java:143) at org.apache.hadoop.hbase.master.AssignmentManager.rebuildUserRegions(AssignmentManager.java:2923) at org.apache.hadoop.hbase.master.AssignmentManager.joinCluster(AssignmentManager.java:486) at org.apache.hadoop.hbase.master.TestAssignmentManager$4.run(TestAssignmentManager.java:1372) 2015-06-15 03:05:44,497 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 143323 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:05:46,003 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 144829 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:05:47,508 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 146334 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:05:49,012 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 147838 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:05:50,529 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 149355 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:05:51,114 DEBUG [ProcedureExecutorTimeoutThread] procedure2.ProcedureExecutor$CompletedProcedureCleaner(152): No completed procedures to cleanup. 2015-06-15 03:05:52,035 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 150861 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:05:53,540 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 152366 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:05:55,046 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 153872 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:05:56,555 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 155381 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:05:58,064 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 156890 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:05:59,574 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 158400 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:06:01,081 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 159907 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:06:02,589 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 161415 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:06:04,096 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 162922 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:06:05,603 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 164429 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:06:07,113 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 165938 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:06:08,621 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 167447 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:06:10,129 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 168955 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:06:11,640 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 170466 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:06:13,149 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 171975 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:06:14,659 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 173485 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:06:16,171 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 174997 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:06:17,678 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 176504 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:06:19,184 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 178010 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:06:20,689 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 179515 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:06:21,115 DEBUG [ProcedureExecutorTimeoutThread] procedure2.ProcedureExecutor$CompletedProcedureCleaner(152): No completed procedures to cleanup. 2015-06-15 03:06:22,194 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 181020 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:06:23,698 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 182524 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:06:25,203 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 184029 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:06:26,714 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 185540 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:06:28,223 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 187049 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:06:29,731 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 188557 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:06:31,239 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 190065 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:06:32,743 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 191569 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:06:34,251 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 193077 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:06:35,756 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 194582 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:06:37,264 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 196090 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:06:38,771 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 197597 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:06:40,285 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 199111 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:06:41,790 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 200616 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:06:43,296 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 202122 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:06:44,055 DEBUG [main] zookeeper.ZKAssign(497): mockedServer-0x14df52e06fa0000, quorum=localhost:50250, baseZNode=/hbase Deleting any existing unassigned nodes 2015-06-15 03:06:44,064 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer-0x14df52e06fa0000, quorum=localhost:50250, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/cae5ea3b1868354349837ee2dd0d4d89 2015-06-15 03:06:44,064 ERROR [main-EventThread] zookeeper.ClientCnxn$EventThread(524): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6d180fdb rejected from java.util.concurrent.ThreadPoolExecutor@46b92ad1[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110) at org.apache.hadoop.hbase.master.AssignmentManager.zkEventWorkersSubmit(AssignmentManager.java:1291) at org.apache.hadoop.hbase.master.AssignmentManager.nodeDeleted(AssignmentManager.java:1322) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:532) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) 2015-06-15 03:06:44,092 INFO [main] hbase.ResourceChecker(171): after: master.TestAssignmentManager#testBalanceOnMasterFailoverScenarioWithOfflineNode Thread=157 (was 159), OpenFileDescriptor=339 (was 335) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=32768 (was 32768), SystemLoadAverage=493 (was 442) - SystemLoadAverage LEAK? -, ProcessCount=98 (was 98), AvailableMemoryMB=5360 (was 6408), ConnectionCount=6 (was 5) - ConnectionCount LEAK? - 2015-06-15 03:06:44,094 INFO [main] zookeeper.MiniZooKeeperCluster(316): Shutdown MiniZK cluster with all ZK servers 2015-06-15 03:06:44,233 INFO [main] zookeeper.MiniZooKeeperCluster(273): Started MiniZooKeeperCluster and ran successful 'stat' on client port=59682 2015-06-15 03:06:44,276 INFO [main] hbase.ResourceChecker(147): before: master.TestAssignmentManager#testBalanceOnMasterFailoverScenarioWithOfflineNode Thread=159, OpenFileDescriptor=335, MaxFileDescriptor=32768, SystemLoadAverage=493, ProcessCount=98, AvailableMemoryMB=5358, ConnectionCount=6 2015-06-15 03:06:44,277 INFO [main] zookeeper.RecoverableZooKeeper(121): Process identifier=mockedServer connecting to ZooKeeper ensemble=localhost:59682 2015-06-15 03:06:44,312 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer0x0, quorum=localhost:59682, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-15 03:06:44,314 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(592): mockedServer-0x14df52ef25e0000 connected 2015-06-15 03:06:44,316 DEBUG [main] zookeeper.RecoverableZooKeeper(595): Node /hbase already exists 2015-06-15 03:06:44,378 DEBUG [Thread-128] executor.ExecutorService(99): Starting executor service name=MASTER_OPEN_REGION-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:06:44,379 DEBUG [Thread-128] executor.ExecutorService(99): Starting executor service name=MASTER_CLOSE_REGION-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:06:44,379 DEBUG [Thread-128] executor.ExecutorService(99): Starting executor service name=MASTER_SERVER_OPERATIONS-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:06:44,380 DEBUG [Thread-128] executor.ExecutorService(99): Starting executor service name=MASTER_META_SERVER_OPERATIONS-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:06:44,399 INFO [Thread-128] master.RegionStates(1112): Transition null to {cae5ea3b1868354349837ee2dd0d4d89 state=OPEN, ts=1434337604399, server=example.org,1234,5678} 2015-06-15 03:06:44,400 DEBUG [Thread-128] master.RegionStates(460): Onlined cae5ea3b1868354349837ee2dd0d4d89 on example.org,1234,5678 {ENCODED => cae5ea3b1868354349837ee2dd0d4d89, NAME => 't,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89.', STARTKEY => '', ENDKEY => ''} 2015-06-15 03:06:44,400 DEBUG [Thread-128] master.AssignmentManager(2470): Starting unassign of t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. (offlining), current state: {cae5ea3b1868354349837ee2dd0d4d89 state=OPEN, ts=1434337604399, server=example.org,1234,5678} 2015-06-15 03:06:44,400 DEBUG [Thread-128] zookeeper.ZKAssign(527): mockedServer-0x14df52ef25e0000, quorum=localhost:59682, baseZNode=/hbase Creating unassigned node cae5ea3b1868354349837ee2dd0d4d89 in a CLOSING state 2015-06-15 03:06:44,412 INFO [Thread-128] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=OPEN, ts=1434337604399, server=example.org,1234,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=PENDING_CLOSE, ts=1434337604412, server=example.org,1234,5678} 2015-06-15 03:06:44,412 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,412 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,412 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,413 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,413 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,413 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,414 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,414 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,414 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,414 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,415 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,415 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,415 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,416 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,416 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,428 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,428 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,429 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,429 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,429 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,429 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,430 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,430 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,430 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,431 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,433 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,434 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,434 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,434 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,435 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,435 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,435 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,435 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,436 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,438 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,438 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,449 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,451 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,451 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,451 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,451 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,454 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,455 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,455 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,455 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,456 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,456 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,457 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,457 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,457 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,458 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,458 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,458 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,459 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,459 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,459 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,459 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,460 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,460 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,460 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,461 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,461 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,461 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,461 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,462 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,462 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,462 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,463 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,463 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,463 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,463 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,463 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,464 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,464 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,464 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,465 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,465 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,465 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,465 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,466 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,466 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,467 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,467 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,467 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,468 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,468 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,468 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,469 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,469 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,469 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,470 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,470 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,470 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,470 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,471 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,471 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,475 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,475 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,476 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,476 WARN [Thread-128] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:06:44,476 WARN [Thread-128] master.RegionStates(1103): Failed to open/close cae5ea3b1868354349837ee2dd0d4d89 on example.org,1234,5678, set to FAILED_CLOSE 2015-06-15 03:06:44,477 INFO [Thread-128] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=PENDING_CLOSE, ts=1434337604412, server=example.org,1234,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=FAILED_CLOSE, ts=1434337604477, server=example.org,1234,5678} 2015-06-15 03:06:44,484 ERROR [RunAmJoinCluster] server.NIOServerCnxnFactory$1(44): Thread Thread[RunAmJoinCluster,5,main] died java.lang.NullPointerException at org.apache.hadoop.hbase.MetaTableAccessor.fullScan(MetaTableAccessor.java:611) at org.apache.hadoop.hbase.MetaTableAccessor.fullScanOfMeta(MetaTableAccessor.java:143) at org.apache.hadoop.hbase.master.AssignmentManager.rebuildUserRegions(AssignmentManager.java:2923) at org.apache.hadoop.hbase.master.AssignmentManager.joinCluster(AssignmentManager.java:486) at org.apache.hadoop.hbase.master.TestAssignmentManager$4.run(TestAssignmentManager.java:1372) 2015-06-15 03:06:44,810 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 203636 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:06:46,323 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 205149 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:06:47,833 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 206659 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:06:49,340 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 208166 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:06:50,846 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 209672 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:06:51,115 DEBUG [ProcedureExecutorTimeoutThread] procedure2.ProcedureExecutor$CompletedProcedureCleaner(152): No completed procedures to cleanup. 2015-06-15 03:06:52,356 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 211181 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:06:53,867 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 212693 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:06:55,376 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 214202 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:06:56,882 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 215708 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:06:58,390 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 217216 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:06:59,900 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 218726 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:07:01,408 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 220234 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:07:02,915 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 221741 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:07:04,421 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 223247 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:07:05,929 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 224755 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:07:07,438 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 226264 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:07:08,941 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 227767 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:07:10,448 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 229274 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:07:11,959 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 230785 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:07:13,468 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 232294 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:07:14,979 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 233805 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:07:16,489 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 235315 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:07:17,997 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 236823 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:07:19,504 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 238330 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:07:21,012 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 239838 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:07:21,115 DEBUG [ProcedureExecutorTimeoutThread] procedure2.ProcedureExecutor$CompletedProcedureCleaner(152): No completed procedures to cleanup. 2015-06-15 03:07:22,518 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 241344 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:07:24,023 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 242849 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:07:25,536 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 244362 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:07:27,045 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 245871 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:07:28,551 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 247377 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:07:30,062 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 248888 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:07:31,571 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 250397 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:07:33,084 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 251910 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:07:34,594 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 253420 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:07:36,104 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 254930 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:07:37,614 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 256440 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:07:39,125 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 257951 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:07:40,631 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 259456 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:07:42,139 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 260965 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:07:43,644 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 262470 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:07:44,341 DEBUG [main] zookeeper.ZKAssign(497): mockedServer-0x14df52ef25e0000, quorum=localhost:59682, baseZNode=/hbase Deleting any existing unassigned nodes 2015-06-15 03:07:44,344 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer-0x14df52ef25e0000, quorum=localhost:59682, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/cae5ea3b1868354349837ee2dd0d4d89 2015-06-15 03:07:44,345 ERROR [main-EventThread] zookeeper.ClientCnxn$EventThread(524): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@713b6f83 rejected from java.util.concurrent.ThreadPoolExecutor@75163b88[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110) at org.apache.hadoop.hbase.master.AssignmentManager.zkEventWorkersSubmit(AssignmentManager.java:1291) at org.apache.hadoop.hbase.master.AssignmentManager.nodeDeleted(AssignmentManager.java:1322) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:532) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) 2015-06-15 03:07:44,411 INFO [main] hbase.ResourceChecker(171): after: master.TestAssignmentManager#testBalanceOnMasterFailoverScenarioWithOfflineNode Thread=158 (was 159), OpenFileDescriptor=339 (was 335) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=32768 (was 32768), SystemLoadAverage=827 (was 493) - SystemLoadAverage LEAK? -, ProcessCount=98 (was 98), AvailableMemoryMB=6757 (was 5358) - AvailableMemoryMB LEAK? -, ConnectionCount=7 (was 6) - ConnectionCount LEAK? - 2015-06-15 03:07:44,415 INFO [main] zookeeper.MiniZooKeeperCluster(316): Shutdown MiniZK cluster with all ZK servers 2015-06-15 03:07:44,606 INFO [main] zookeeper.MiniZooKeeperCluster(273): Started MiniZooKeeperCluster and ran successful 'stat' on client port=49821 2015-06-15 03:07:44,673 INFO [main] hbase.ResourceChecker(147): before: master.TestAssignmentManager#testBalanceOnMasterFailoverScenarioWithOfflineNode Thread=159, OpenFileDescriptor=335, MaxFileDescriptor=32768, SystemLoadAverage=827, ProcessCount=98, AvailableMemoryMB=6750, ConnectionCount=7 2015-06-15 03:07:44,674 INFO [main] zookeeper.RecoverableZooKeeper(121): Process identifier=mockedServer connecting to ZooKeeper ensemble=localhost:49821 2015-06-15 03:07:44,721 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer0x0, quorum=localhost:49821, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-15 03:07:44,722 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(592): mockedServer-0x14df52fde290000 connected 2015-06-15 03:07:44,723 DEBUG [main] zookeeper.RecoverableZooKeeper(595): Node /hbase already exists 2015-06-15 03:07:44,758 DEBUG [Thread-130] executor.ExecutorService(99): Starting executor service name=MASTER_OPEN_REGION-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:07:44,759 DEBUG [Thread-130] executor.ExecutorService(99): Starting executor service name=MASTER_CLOSE_REGION-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:07:44,759 DEBUG [Thread-130] executor.ExecutorService(99): Starting executor service name=MASTER_SERVER_OPERATIONS-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:07:44,759 DEBUG [Thread-130] executor.ExecutorService(99): Starting executor service name=MASTER_META_SERVER_OPERATIONS-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2015-06-15 03:07:44,773 INFO [Thread-130] master.RegionStates(1112): Transition null to {cae5ea3b1868354349837ee2dd0d4d89 state=OPEN, ts=1434337664773, server=example.org,1234,5678} 2015-06-15 03:07:44,773 DEBUG [Thread-130] master.RegionStates(460): Onlined cae5ea3b1868354349837ee2dd0d4d89 on example.org,1234,5678 {ENCODED => cae5ea3b1868354349837ee2dd0d4d89, NAME => 't,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89.', STARTKEY => '', ENDKEY => ''} 2015-06-15 03:07:44,774 DEBUG [Thread-130] master.AssignmentManager(2470): Starting unassign of t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. (offlining), current state: {cae5ea3b1868354349837ee2dd0d4d89 state=OPEN, ts=1434337664773, server=example.org,1234,5678} 2015-06-15 03:07:44,775 DEBUG [Thread-130] zookeeper.ZKAssign(527): mockedServer-0x14df52fde290000, quorum=localhost:49821, baseZNode=/hbase Creating unassigned node cae5ea3b1868354349837ee2dd0d4d89 in a CLOSING state 2015-06-15 03:07:44,784 INFO [Thread-130] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=OPEN, ts=1434337664773, server=example.org,1234,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=PENDING_CLOSE, ts=1434337664784, server=example.org,1234,5678} 2015-06-15 03:07:44,785 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,785 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,785 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,786 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,786 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,786 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,786 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,786 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,787 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,787 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,787 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,787 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,798 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,799 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,799 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,799 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,799 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,799 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,801 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,802 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,802 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,802 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,802 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,803 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,803 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,803 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,804 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,804 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,807 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,807 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,808 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,808 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,808 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,808 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,808 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,809 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,809 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,809 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,809 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,810 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,828 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,828 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,829 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,829 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,829 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,829 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,830 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,830 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,830 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,830 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,830 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,831 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,845 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,846 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,846 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,846 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,846 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,847 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,847 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,847 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,847 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,848 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,848 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,848 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,849 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,849 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,849 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,849 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,850 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,850 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,868 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,868 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,868 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,869 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,869 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,869 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,869 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,870 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,870 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,870 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,870 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,871 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,871 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,871 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,871 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,872 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,889 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,889 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,889 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,890 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,890 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,890 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,890 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,891 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,891 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,891 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,891 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,892 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,892 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,895 WARN [Thread-130] master.AssignmentManager(1866): Server example.org,1234,5678 region CLOSE RPC returned false for t,,1434337330882.cae5ea3b1868354349837ee2dd0d4d89. 2015-06-15 03:07:44,902 WARN [Thread-130] master.RegionStates(1103): Failed to open/close cae5ea3b1868354349837ee2dd0d4d89 on example.org,1234,5678, set to FAILED_CLOSE 2015-06-15 03:07:44,902 INFO [Thread-130] master.RegionStates(1112): Transition {cae5ea3b1868354349837ee2dd0d4d89 state=PENDING_CLOSE, ts=1434337664784, server=example.org,1234,5678} to {cae5ea3b1868354349837ee2dd0d4d89 state=FAILED_CLOSE, ts=1434337664902, server=example.org,1234,5678} 2015-06-15 03:07:44,918 ERROR [RunAmJoinCluster] server.NIOServerCnxnFactory$1(44): Thread Thread[RunAmJoinCluster,5,main] died java.lang.NullPointerException at org.apache.hadoop.hbase.MetaTableAccessor.fullScan(MetaTableAccessor.java:611) at org.apache.hadoop.hbase.MetaTableAccessor.fullScanOfMeta(MetaTableAccessor.java:143) at org.apache.hadoop.hbase.master.AssignmentManager.rebuildUserRegions(AssignmentManager.java:2923) at org.apache.hadoop.hbase.master.AssignmentManager.joinCluster(AssignmentManager.java:486) at org.apache.hadoop.hbase.master.TestAssignmentManager$4.run(TestAssignmentManager.java:1372) 2015-06-15 03:07:45,167 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 263993 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:07:46,674 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 265500 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:07:48,178 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 267004 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:07:49,686 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 268512 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:07:51,116 DEBUG [ProcedureExecutorTimeoutThread] procedure2.ProcedureExecutor$CompletedProcedureCleaner(152): No completed procedures to cleanup. 2015-06-15 03:07:51,196 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 270022 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:07:52,702 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 271528 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:07:54,205 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 273031 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:07:55,711 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 274537 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:07:57,225 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 276051 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:07:58,730 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 277556 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:08:00,239 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 279065 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:08:01,747 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 280573 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:08:03,253 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 282079 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:08:04,762 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 283588 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:08:06,271 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 285097 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:08:07,781 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 286607 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:08:09,290 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 288116 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:08:10,797 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 289623 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:08:12,301 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 291127 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:08:13,813 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 292639 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:08:15,322 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 294148 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:08:16,828 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 295654 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:08:18,335 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 297161 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:08:19,840 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 298666 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:08:21,117 DEBUG [ProcedureExecutorTimeoutThread] procedure2.ProcedureExecutor$CompletedProcedureCleaner(152): No completed procedures to cleanup. 2015-06-15 03:08:21,347 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 300173 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:08:22,856 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 301682 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:08:24,366 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 303192 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:08:25,874 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 304700 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:08:27,379 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 306205 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:08:28,885 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 307711 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:08:30,389 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 309215 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:08:31,898 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 310724 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:08:33,402 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 312228 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:08:34,911 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 313737 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:08:36,415 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 315241 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:08:37,919 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 316745 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:08:39,422 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 318248 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:08:40,926 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 319752 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:08:42,430 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 321256 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:08:43,934 INFO [ndimiduk-hbase-1-1-1RC0-1:60005.activeMasterManager] master.ServerManager(1025): Waiting for region servers count to settle; currently checked in 0, slept for 322760 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2015-06-15 03:08:44,745 DEBUG [main] zookeeper.ZKAssign(497): mockedServer-0x14df52fde290000, quorum=localhost:49821, baseZNode=/hbase Deleting any existing unassigned nodes 2015-06-15 03:08:44,748 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(508): mockedServer-0x14df52fde290000, quorum=localhost:49821, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/cae5ea3b1868354349837ee2dd0d4d89 2015-06-15 03:08:44,749 ERROR [main-EventThread] zookeeper.ClientCnxn$EventThread(524): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6fb1a043 rejected from java.util.concurrent.ThreadPoolExecutor@7690c7fe[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110) at org.apache.hadoop.hbase.master.AssignmentManager.zkEventWorkersSubmit(AssignmentManager.java:1291) at org.apache.hadoop.hbase.master.AssignmentManager.nodeDeleted(AssignmentManager.java:1322) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:532) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) 2015-06-15 03:08:44,800 INFO [main] hbase.ResourceChecker(171): after: master.TestAssignmentManager#testBalanceOnMasterFailoverScenarioWithOfflineNode Thread=158 (was 159), OpenFileDescriptor=339 (was 335) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=32768 (was 32768), SystemLoadAverage=606 (was 827), ProcessCount=98 (was 98), AvailableMemoryMB=6310 (was 6750), ConnectionCount=8 (was 7) - ConnectionCount LEAK? - 2015-06-15 03:08:44,802 INFO [main] zookeeper.MiniZooKeeperCluster(316): Shutdown MiniZK cluster with all ZK servers