2014-12-18 00:45:17,698 INFO [main] hbase.ResourceChecker(147): before: master.TestAssignmentManager#testMasterRestartWhenTableInEnabling Thread=12, OpenFileDescriptor=137, MaxFileDescriptor=32768, SystemLoadAverage=507, ProcessCount=460, AvailableMemoryMB=6701, ConnectionCount=1 2014-12-18 00:45:17,701 INFO [main] zookeeper.RecoverableZooKeeper(121): Process identifier=mockedServer connecting to ZooKeeper ensemble=localhost:57178 2014-12-18 00:45:17,715 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(311): mockedServer, quorum=localhost:57178, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2014-12-18 00:45:17,717 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(395): mockedServer-0x14a5adb0aba0009 connected 2014-12-18 00:45:17,723 DEBUG [main] zookeeper.RecoverableZooKeeper(531): Node /hbase already exists 2014-12-18 00:45:17,790 INFO [Thread-15] client.ConnectionUtils(102): master/kiyo.gq1.ygridcore.net/68.142.244.61:0 server-side HConnection retries=350 2014-12-18 00:45:17,939 INFO [Thread-15] ipc.SimpleRpcScheduler(123): Using deadline as user call queue, count=1 2014-12-18 00:45:17,946 INFO [Thread-15] ipc.RpcServer$Listener(538): master/kiyo.gq1.ygridcore.net/68.142.244.61:0: started 10 reader(s). 2014-12-18 00:45:17,954 DEBUG [Thread-15] fs.HFileSystem(221): The file system is not a DistributedFileSystem. Skipping on block location reordering 2014-12-18 00:45:17,998 INFO [Thread-15] zookeeper.RecoverableZooKeeper(121): Process identifier=master:38816 connecting to ZooKeeper ensemble=localhost:57178 2014-12-18 00:45:18,012 DEBUG [Thread-15-EventThread] zookeeper.ZooKeeperWatcher(311): master:38816, quorum=localhost:57178, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2014-12-18 00:45:18,014 DEBUG [Thread-15-EventThread] zookeeper.ZooKeeperWatcher(395): master:38816-0x14a5adb0aba000a connected 2014-12-18 00:45:18,022 DEBUG [Thread-15] zookeeper.RecoverableZooKeeper(531): Node /hbase already exists 2014-12-18 00:45:18,032 DEBUG [Thread-15] zookeeper.ZKUtil(430): master:38816-0x14a5adb0aba000a, quorum=localhost:57178, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2014-12-18 00:45:18,033 DEBUG [Thread-15] zookeeper.ZKUtil(430): master:38816-0x14a5adb0aba000a, quorum=localhost:57178, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2014-12-18 00:45:18,037 INFO [RpcServer.responder] ipc.RpcServer$Responder(857): RpcServer.responder: starting 2014-12-18 00:45:18,037 INFO [RpcServer.listener,port=38816] ipc.RpcServer$Listener(669): RpcServer.listener,port=38816: starting 2014-12-18 00:45:18,037 DEBUG [Thread-15] ipc.RpcExecutor(99): B.default Start Handler index=0 queue=0 2014-12-18 00:45:18,038 DEBUG [Thread-15] ipc.RpcExecutor(99): B.default Start Handler index=1 queue=0 2014-12-18 00:45:18,038 DEBUG [Thread-15] ipc.RpcExecutor(99): B.default Start Handler index=2 queue=0 2014-12-18 00:45:18,039 DEBUG [Thread-15] ipc.RpcExecutor(99): B.default Start Handler index=3 queue=0 2014-12-18 00:45:18,039 DEBUG [Thread-15] ipc.RpcExecutor(99): B.default Start Handler index=4 queue=0 2014-12-18 00:45:18,039 DEBUG [Thread-15] ipc.RpcExecutor(99): Priority Start Handler index=0 queue=0 2014-12-18 00:45:18,040 DEBUG [Thread-15] ipc.RpcExecutor(99): Priority Start Handler index=1 queue=0 2014-12-18 00:45:18,040 DEBUG [Thread-15] ipc.RpcExecutor(99): Priority Start Handler index=2 queue=0 2014-12-18 00:45:18,040 DEBUG [Thread-15] ipc.RpcExecutor(99): Priority Start Handler index=3 queue=0 2014-12-18 00:45:18,041 DEBUG [Thread-15] ipc.RpcExecutor(99): Priority Start Handler index=4 queue=0 2014-12-18 00:45:18,041 DEBUG [Thread-15] ipc.RpcExecutor(99): Priority Start Handler index=5 queue=0 2014-12-18 00:45:18,041 DEBUG [Thread-15] ipc.RpcExecutor(99): Priority Start Handler index=6 queue=0 2014-12-18 00:45:18,041 DEBUG [Thread-15] ipc.RpcExecutor(99): Priority Start Handler index=7 queue=0 2014-12-18 00:45:18,042 DEBUG [Thread-15] ipc.RpcExecutor(99): Priority Start Handler index=8 queue=0 2014-12-18 00:45:18,042 DEBUG [Thread-15] ipc.RpcExecutor(99): Priority Start Handler index=9 queue=0 2014-12-18 00:45:18,042 DEBUG [Thread-15] ipc.RpcExecutor(99): Replication Start Handler index=0 queue=0 2014-12-18 00:45:18,042 DEBUG [Thread-15] ipc.RpcExecutor(99): Replication Start Handler index=1 queue=0 2014-12-18 00:45:18,043 DEBUG [Thread-15] ipc.RpcExecutor(99): Replication Start Handler index=2 queue=0 2014-12-18 00:45:18,045 INFO [Thread-15] master.HMaster(340): hbase.rootdir=file:/tmp/hbase-hortonzy/hbase, hbase.cluster.distributed=false 2014-12-18 00:45:18,056 INFO [Thread-15] master.HMaster(1396): Adding backup master ZNode /hbase/backup-masters/kiyo.gq1.ygridcore.net,38816,1418863517953 2014-12-18 00:45:18,071 DEBUG [Thread-15] zookeeper.ZKUtil(428): master:38816-0x14a5adb0aba000a, quorum=localhost:57178, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/kiyo.gq1.ygridcore.net,38816,1418863517953 2014-12-18 00:45:18,085 DEBUG [Thread-15] executor.ExecutorService(99): Starting executor service name=MASTER_OPEN_REGION-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2014-12-18 00:45:18,085 DEBUG [Thread-15] executor.ExecutorService(99): Starting executor service name=MASTER_CLOSE_REGION-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2014-12-18 00:45:18,085 DEBUG [Thread-15] executor.ExecutorService(99): Starting executor service name=MASTER_SERVER_OPERATIONS-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2014-12-18 00:45:18,086 DEBUG [Thread-15] executor.ExecutorService(99): Starting executor service name=MASTER_META_SERVER_OPERATIONS-mockedAMExecutor, corePoolSize=3, maxPoolSize=3 2014-12-18 00:45:18,095 DEBUG [Thread-15-EventThread] zookeeper.ZooKeeperWatcher(311): master:38816-0x14a5adb0aba000a, quorum=localhost:57178, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2014-12-18 00:45:18,097 DEBUG [kiyo:38816.activeMasterManager] zookeeper.ZKUtil(428): master:38816-0x14a5adb0aba000a, quorum=localhost:57178, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2014-12-18 00:45:18,098 INFO [kiyo:38816.activeMasterManager] master.ActiveMasterManager(166): Deleting ZNode for /hbase/backup-masters/kiyo.gq1.ygridcore.net,38816,1418863517953 from backup master directory 2014-12-18 00:45:18,098 DEBUG [Thread-15-EventThread] zookeeper.ZKUtil(428): master:38816-0x14a5adb0aba000a, quorum=localhost:57178, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2014-12-18 00:45:18,099 DEBUG [Thread-15-EventThread] master.ActiveMasterManager(122): A master is now available 2014-12-18 00:45:18,103 WARN [Thread-15] zookeeper.ZKTableStateManager(100): Moving table t state from ENABLED to ENABLING 2014-12-18 00:45:18,186 DEBUG [Thread-15-EventThread] zookeeper.ZooKeeperWatcher(311): master:38816-0x14a5adb0aba000a, quorum=localhost:57178, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/kiyo.gq1.ygridcore.net,38816,1418863517953 2014-12-18 00:45:18,187 WARN [kiyo:38816.activeMasterManager] hbase.ZNodeClearer(58): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2014-12-18 00:45:18,188 INFO [kiyo:38816.activeMasterManager] master.ActiveMasterManager(175): Registered Active Master=kiyo.gq1.ygridcore.net,38816,1418863517953 2014-12-18 00:45:18,221 DEBUG [kiyo:38816.activeMasterManager] util.FSTableDescriptors(727): Current tableInfoPath = file:/tmp/hbase-hortonzy/hbase/data/hbase/meta/.tabledesc/.tableinfo.0000000003 2014-12-18 00:45:18,229 FATAL [kiyo:38816.activeMasterManager] master.HMaster$1(1431): Failed to become active master java.lang.IllegalArgumentException: Illegal character <10> at 0. Namespaces can only contain 'alphanumeric characters': i.e. [a-zA-Z_0-9]: hbasemeta at org.apache.hadoop.hbase.TableName.isLegalNamespaceName(TableName.java:215) at org.apache.hadoop.hbase.TableName.isLegalNamespaceName(TableName.java:204) at org.apache.hadoop.hbase.TableName.(TableName.java:307) at org.apache.hadoop.hbase.TableName.createTableNameIfNecessary(TableName.java:344) at org.apache.hadoop.hbase.TableName.valueOf(TableName.java:465) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.toTableName(ProtobufUtil.java:2686) at org.apache.hadoop.hbase.HTableDescriptor.convert(HTableDescriptor.java:1525) at org.apache.hadoop.hbase.HTableDescriptor.parseFrom(HTableDescriptor.java:1487) at org.apache.hadoop.hbase.util.FSTableDescriptors.readTableDescriptor(FSTableDescriptors.java:526) at org.apache.hadoop.hbase.util.FSTableDescriptors.createTableDescriptorForTableDirectory(FSTableDescriptors.java:730) at org.apache.hadoop.hbase.util.FSTableDescriptors.createTableDescriptor(FSTableDescriptors.java:706) at org.apache.hadoop.hbase.util.FSTableDescriptors.createTableDescriptor(FSTableDescriptors.java:693) at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:466) at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:145) at org.apache.hadoop.hbase.master.MasterFileSystem.(MasterFileSystem.java:125) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:592) at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:166) at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1427) at java.lang.Thread.run(Thread.java:724) 2014-12-18 00:45:18,230 FATAL [kiyo:38816.activeMasterManager] master.HMaster(1802): Unhandled exception. Starting shutdown. java.lang.IllegalArgumentException: Illegal character <10> at 0. Namespaces can only contain 'alphanumeric characters': i.e. [a-zA-Z_0-9]: hbasemeta at org.apache.hadoop.hbase.TableName.isLegalNamespaceName(TableName.java:215) at org.apache.hadoop.hbase.TableName.isLegalNamespaceName(TableName.java:204) at org.apache.hadoop.hbase.TableName.(TableName.java:307) at org.apache.hadoop.hbase.TableName.createTableNameIfNecessary(TableName.java:344) at org.apache.hadoop.hbase.TableName.valueOf(TableName.java:465) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.toTableName(ProtobufUtil.java:2686) at org.apache.hadoop.hbase.HTableDescriptor.convert(HTableDescriptor.java:1525) at org.apache.hadoop.hbase.HTableDescriptor.parseFrom(HTableDescriptor.java:1487) at org.apache.hadoop.hbase.util.FSTableDescriptors.readTableDescriptor(FSTableDescriptors.java:526) at org.apache.hadoop.hbase.util.FSTableDescriptors.createTableDescriptorForTableDirectory(FSTableDescriptors.java:730) at org.apache.hadoop.hbase.util.FSTableDescriptors.createTableDescriptor(FSTableDescriptors.java:706) at org.apache.hadoop.hbase.util.FSTableDescriptors.createTableDescriptor(FSTableDescriptors.java:693) at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:466) at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:145) at org.apache.hadoop.hbase.master.MasterFileSystem.(MasterFileSystem.java:125) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:592) at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:166) at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1427) at java.lang.Thread.run(Thread.java:724) 2014-12-18 00:45:18,230 INFO [kiyo:38816.activeMasterManager] regionserver.HRegionServer(1759): STOPPED: Unhandled exception. Starting shutdown. 2014-12-18 00:45:18,255 INFO [Thread-15] handler.EnableTableHandler(150): Attempting to enable the table t 2014-12-18 00:45:18,256 WARN [Thread-15] zookeeper.ZKTableStateManager(100): Moving table t state from ENABLING to ENABLING 2014-12-18 00:45:18,285 INFO [Thread-15] handler.EnableTableHandler(224): Table 't' has 1 regions, of which 1 are offline. 2014-12-18 00:45:18,285 INFO [Thread-15] handler.EnableTableHandler(229): Bulk assigning 1 region(s) across 1 server(s), retainAssignment=true 2014-12-18 00:45:18,288 DEBUG [Thread-15] master.GeneralBulkAssigner(177): Timeout-on-RIT=91000 2014-12-18 00:45:18,288 WARN [Thread-15] master.GeneralBulkAssigner(106): bulk assigner is still running after 0ms, shut it down now 2014-12-18 00:45:18,289 DEBUG [Thread-15] master.GeneralBulkAssigner(153): bulk assigning total 1 regions to 1 servers, took 1ms, with 1 regions still in transition 2014-12-18 00:45:18,289 INFO [kiyo.gq1.ygridcore.net,38816,1418863517953-GeneralBulkAssigner-0] master.RegionStates(1012): Transition null to {ddb1e4883d495fc7dfb0630dc044bff9 state=OPEN, ts=1418863518289, server=example.org,1234,5678} 2014-12-18 00:45:18,289 WARN [Thread-15] handler.EnableTableHandler(250): Table 't' wasn't successfully enabled. Status: done=false 2014-12-18 00:45:18,289 INFO [kiyo.gq1.ygridcore.net,38816,1418863517953-GeneralBulkAssigner-0] master.RegionStates(436): Onlined ddb1e4883d495fc7dfb0630dc044bff9 on example.org,1234,5678 2014-12-18 00:45:18,289 WARN [Thread-15] zookeeper.ZKTableStateManager(100): Moving table t state from ENABLING to ENABLED 2014-12-18 00:45:18,314 DEBUG [Thread-15] zookeeper.ZKAssign(497): mockedServer-0x14a5adb0aba0009, quorum=localhost:57178, baseZNode=/hbase Deleting any existing unassigned nodes 2014-12-18 00:45:18,316 DEBUG [main] zookeeper.ZKAssign(497): mockedServer-0x14a5adb0aba0009, quorum=localhost:57178, baseZNode=/hbase Deleting any existing unassigned nodes 2014-12-18 00:45:18,425 INFO [main] hbase.ResourceChecker(171): after: master.TestAssignmentManager#testMasterRestartWhenTableInEnabling Thread=45 (was 12) Potentially hanging thread: RpcServer.reader=2,port=38816 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:570) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:557) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:724) Potentially hanging thread: PriorityRpcServer.handler=0,queue=0,port=38816 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:109) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:92) java.lang.Thread.run(Thread.java:724) Potentially hanging thread: PriorityRpcServer.handler=2,queue=0,port=38816 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:109) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:92) java.lang.Thread.run(Thread.java:724) Potentially hanging thread: PriorityRpcServer.handler=3,queue=0,port=38816 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:109) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:92) java.lang.Thread.run(Thread.java:724) Potentially hanging thread: PriorityRpcServer.handler=4,queue=0,port=38816 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:109) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:92) java.lang.Thread.run(Thread.java:724) Potentially hanging thread: PriorityRpcServer.handler=8,queue=0,port=38816 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:109) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:92) java.lang.Thread.run(Thread.java:724) Potentially hanging thread: PriorityRpcServer.handler=7,queue=0,port=38816 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:109) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:92) java.lang.Thread.run(Thread.java:724) Potentially hanging thread: PriorityRpcServer.handler=1,queue=0,port=38816 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:109) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:92) java.lang.Thread.run(Thread.java:724) Potentially hanging thread: RpcServer.reader=3,port=38816 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:570) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:557) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:724) Potentially hanging thread: B.defaultRpcServer.handler=1,queue=0,port=38816 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) org.apache.hadoop.hbase.util.BoundedPriorityBlockingQueue.take(BoundedPriorityBlockingQueue.java:213) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:109) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:92) java.lang.Thread.run(Thread.java:724) Potentially hanging thread: RpcServer.reader=9,port=38816 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:570) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:557) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:724) Potentially hanging thread: Thread-37 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.master.HMaster$InitializationMonitor.run(HMaster.java:202) java.lang.Thread.run(Thread.java:724) Potentially hanging thread: PriorityRpcServer.handler=5,queue=0,port=38816 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:109) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:92) java.lang.Thread.run(Thread.java:724) Potentially hanging thread: ReplicationRpcServer.handler=2,queue=0,port=38816 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:109) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:92) java.lang.Thread.run(Thread.java:724) Potentially hanging thread: RpcServer.reader=0,port=38816 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:570) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:557) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:724) Potentially hanging thread: Thread-15-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) Potentially hanging thread: PriorityRpcServer.handler=9,queue=0,port=38816 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:109) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:92) java.lang.Thread.run(Thread.java:724) Potentially hanging thread: ReplicationRpcServer.handler=0,queue=0,port=38816 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:109) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:92) java.lang.Thread.run(Thread.java:724) Potentially hanging thread: RpcServer.reader=4,port=38816 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:570) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:557) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:724) Potentially hanging thread: RpcServer.reader=6,port=38816 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:570) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:557) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:724) Potentially hanging thread: RpcServer.reader=8,port=38816 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:570) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:557) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:724) Potentially hanging thread: B.defaultRpcServer.handler=3,queue=0,port=38816 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) org.apache.hadoop.hbase.util.BoundedPriorityBlockingQueue.take(BoundedPriorityBlockingQueue.java:213) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:109) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:92) java.lang.Thread.run(Thread.java:724) Potentially hanging thread: RpcServer.listener,port=38816 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) org.apache.hadoop.hbase.ipc.RpcServer$Listener.run(RpcServer.java:673) Potentially hanging thread: B.defaultRpcServer.handler=2,queue=0,port=38816 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) org.apache.hadoop.hbase.util.BoundedPriorityBlockingQueue.take(BoundedPriorityBlockingQueue.java:213) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:109) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:92) java.lang.Thread.run(Thread.java:724) Potentially hanging thread: RpcServer.reader=5,port=38816 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:570) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:557) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:724) Potentially hanging thread: RpcServer.reader=1,port=38816 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:570) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:557) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:724) Potentially hanging thread: PriorityRpcServer.handler=6,queue=0,port=38816 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:109) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:92) java.lang.Thread.run(Thread.java:724) Potentially hanging thread: RpcServer.responder sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)