2010-04-22 11:20:16,575 DEBUG org.apache.hadoop.hbase.master.HMaster: Processing todo: ProcessServerShutdown of monster01.sf.cloudera.com,60020,1271952610453 2010-04-22 11:20:16,575 INFO org.apache.hadoop.hbase.master.RegionServerOperation: process shutdown of server monster01.sf.cloudera.com,60020,1271952610453: logSplit: false, rootRescanned: false, numberOfMetaRegions: 1, onlineMetaRegions.size(): 0 2010-04-22 11:20:16,576 INFO org.apache.hadoop.hbase.regionserver.HLog: Splitting 1 hlog(s) in hdfs://monster01.sf.cloudera.com:11020/hbase/.logs/monster01.sf.cloudera.com,60020,1271952610453 2010-04-22 11:20:16,577 WARN org.apache.hadoop.hbase.master.HMaster: Processing pending operations: ProcessServerShutdown of monster01.sf.cloudera.com,60020,1271952610453 java.io.IOException: Failed to open hdfs://monster01.sf.cloudera.com:11020/hbase/.logs/monster01.sf.cloudera.com,60020,1271952610453/hlog.dat.1271952610653 for append at org.apache.hadoop.hbase.regionserver.HLog.recoverLog(HLog.java:1408) at org.apache.hadoop.hbase.regionserver.HLog.splitLog(HLog.java:1077) at org.apache.hadoop.hbase.regionserver.HLog.splitLog(HLog.java:983) at org.apache.hadoop.hbase.master.ProcessServerShutdown.process(ProcessServerShutdown.java:286) at org.apache.hadoop.hbase.master.HMaster.processToDoQueue(HMaster.java:496) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:431) Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot create file/hbase/.logs/monster01.sf.cloudera.com,60020,1271952610453/hlog.dat.1271952610653. Name node is in safe mode. The reported blocks 2732 has reached the threshold 0.9990 of total blocks 2732. Safe mode will be turned off automatically in 0 seconds. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1023) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:1172) at org.apache.hadoop.hdfs.server.namenode.NameNode.append(NameNode.java:422) at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:966) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:962) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:960) at org.apache.hadoop.ipc.Client.call(Client.java:740) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) at $Proxy0.append(Unknown Source) at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59) at $Proxy0.append(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.append(DFSClient.java:520) at org.apache.hadoop.hdfs.DistributedFileSystem.append(DistributedFileSystem.java:186) at org.apache.hadoop.fs.FileSystem.append(FileSystem.java:572) at org.apache.hadoop.hbase.regionserver.HLog.recoverLog(HLog.java:1394) ... 5 more 2010-04-22 11:20:16,577 DEBUG org.apache.hadoop.hbase.master.HMaster: Processing todo: ProcessServerShutdown of monster01.sf.cloudera.com,60020,1271952610453 2010-04-22 11:20:16,577 INFO org.apache.hadoop.hbase.master.RegionServerOperation: process shutdown of server monster01.sf.cloudera.com,60020,1271952610453: logSplit: false, rootRescanned: false, numberOfMetaRegions: 1, onlineMetaRegions.size(): 0 2010-04-22 11:20:16,579 INFO org.apache.hadoop.hbase.regionserver.HLog: Splitting 1 hlog(s) in hdfs://monster01.sf.cloudera.com:11020/hbase/.logs/monster01.sf.cloudera.com,60020,1271952610453 2010-04-22 11:20:17,376 INFO org.apache.hadoop.hbase.master.BaseScanner: All 0 .META. region(s) scanned 2010-04-22 11:20:20,296 INFO org.apache.hadoop.hbase.regionserver.HLog: Past out lease recovery 2010-04-22 11:20:20,297 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Splitting hlog 1 of 1: hdfs://monster01.sf.cloudera.com:11020/hbase/.logs/monster01.sf.cloudera.com,60020,1271952610453/hlog.dat.1271952610653, length=116 2010-04-22 11:20:20,301 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed=0 entries from hdfs://monster01.sf.cloudera.com:11020/hbase/.logs/monster01.sf.cloudera.com,60020,1271952610453/hlog.dat.1271952610653 2010-04-22 11:20:20,301 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Successfully split Hlog file. Deleting hdfs://monster01.sf.cloudera.com:11020/hbase/.logs/monster01.sf.cloudera.com,60020,1271952610453/hlog.dat.1271952610653 2010-04-22 11:20:20,305 INFO org.apache.hadoop.hbase.regionserver.HLog: hlog file splitting completed in 3727 millis for hdfs://monster01.sf.cloudera.com:11020/hbase/.logs/monster01.sf.cloudera.com,60020,1271952610453 2010-04-22 11:20:20,305 INFO org.apache.hadoop.hbase.master.RegionServerOperation: Log split complete, meta reassignment and scanning: 2010-04-22 11:20:20,306 DEBUG org.apache.hadoop.hbase.master.ProcessServerShutdown$ScanRootRegion: process server shutdown scanning root region on 192.168.42.42 2010-04-22 11:20:20,316 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: process server shutdown scanning root region on 192.168.42.42 finished HMaster 2010-04-22 11:20:20,316 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 0 2010-04-22 11:20:20,316 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: Requeuing because not all meta regions are online 2010-04-22 11:20:20,316 DEBUG org.apache.hadoop.hbase.master.HMaster: Processing todo: ProcessRegionClose of TestTable,-926971162<1271834364111>,1271855470685, false, reassign: true 2010-04-22 11:20:20,386 INFO org.apache.hadoop.hbase.master.ProcessRegionClose$1: Process Thread Dump: m null 123 active threads Thread 11361 (IPC Client (47) connection to /192.168.42.41:11071 from todd): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:403) org.apache.hadoop.ipc.Client$Connection.run(Client.java:445) Thread 11360 (IPC Client (47) connection to /192.168.42.41:11071 from todd): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:403) org.apache.hadoop.ipc.Client$Connection.run(Client.java:445) Thread 11358 (IPC Client (47) connection to /192.168.42.42:60020 from an unknown user): State: TIMED_WAITING Blocked count: 5 Waited count: 6 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.ipc.HBaseClient$Connection.waitForWork(HBaseClient.java:406) org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:449) Thread 1653 (IPC Client (47) connection to monster01.sf.cloudera.com/192.168.42.40:11020 from todd): State: RUNNABLE Blocked count: 10085 Waited count: 9984 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:332) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128) java.io.FilterInputStream.read(FilterInputStream.java:116) org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:276) java.io.BufferedInputStream.fill(BufferedInputStream.java:218) java.io.BufferedInputStream.read(BufferedInputStream.java:237) java.io.DataInputStream.readInt(DataInputStream.java:370) org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501) org.apache.hadoop.ipc.Client$Connection.run(Client.java:446) Thread 166 (342218962@qtp0-0-EventThread): State: WAITING Blocked count: 0 Waited count: 3 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3ff19018 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:477) Thread 165 (342218962@qtp0-0-SendThread(monster01.sf.cloudera.com:2222)): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1066) Thread 161 (IPC Server handler 99 on 60000): State: WAITING Blocked count: 125 Waited count: 1788 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 160 (IPC Server handler 98 on 60000): State: WAITING Blocked count: 153 Waited count: 1795 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 159 (IPC Server handler 97 on 60000): State: WAITING Blocked count: 139 Waited count: 1794 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 158 (IPC Server handler 96 on 60000): State: WAITING Blocked count: 153 Waited count: 1804 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 157 (IPC Server handler 95 on 60000): State: WAITING Blocked count: 161 Waited count: 1796 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 156 (IPC Server handler 94 on 60000): State: WAITING Blocked count: 138 Waited count: 1794 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 155 (IPC Server handler 93 on 60000): State: WAITING Blocked count: 154 Waited count: 1794 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 154 (IPC Server handler 92 on 60000): State: WAITING Blocked count: 107 Waited count: 1794 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 153 (IPC Server handler 91 on 60000): State: WAITING Blocked count: 140 Waited count: 1789 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 152 (IPC Server handler 90 on 60000): State: WAITING Blocked count: 159 Waited count: 1797 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 151 (IPC Server handler 89 on 60000): State: WAITING Blocked count: 137 Waited count: 1793 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 150 (IPC Server handler 88 on 60000): State: WAITING Blocked count: 130 Waited count: 1784 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 149 (IPC Server handler 87 on 60000): State: WAITING Blocked count: 141 Waited count: 1795 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 148 (IPC Server handler 86 on 60000): State: WAITING Blocked count: 166 Waited count: 1792 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 147 (IPC Server handler 85 on 60000): State: WAITING Blocked count: 155 Waited count: 1801 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 146 (IPC Server handler 84 on 60000): State: WAITING Blocked count: 115 Waited count: 1788 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 145 (IPC Server handler 83 on 60000): State: WAITING Blocked count: 126 Waited count: 1788 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 144 (IPC Server handler 82 on 60000): State: WAITING Blocked count: 105 Waited count: 1788 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 143 (IPC Server handler 81 on 60000): State: WAITING Blocked count: 148 Waited count: 1794 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 142 (IPC Server handler 80 on 60000): State: WAITING Blocked count: 131 Waited count: 1791 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 141 (IPC Server handler 79 on 60000): State: WAITING Blocked count: 160 Waited count: 1803 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 140 (IPC Server handler 78 on 60000): State: WAITING Blocked count: 146 Waited count: 1797 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 139 (IPC Server handler 77 on 60000): State: WAITING Blocked count: 101 Waited count: 1796 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 138 (IPC Server handler 76 on 60000): State: WAITING Blocked count: 133 Waited count: 1798 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 137 (IPC Server handler 75 on 60000): State: WAITING Blocked count: 153 Waited count: 1795 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 136 (IPC Server handler 74 on 60000): State: WAITING Blocked count: 135 Waited count: 1799 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 135 (IPC Server handler 73 on 60000): State: WAITING Blocked count: 116 Waited count: 1790 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 134 (IPC Server handler 72 on 60000): State: WAITING Blocked count: 135 Waited count: 1790 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 133 (IPC Server handler 71 on 60000): State: WAITING Blocked count: 107 Waited count: 1790 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 132 (IPC Server handler 70 on 60000): State: WAITING Blocked count: 126 Waited count: 1787 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 131 (IPC Server handler 69 on 60000): State: WAITING Blocked count: 148 Waited count: 1798 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 130 (IPC Server handler 68 on 60000): State: WAITING Blocked count: 106 Waited count: 1787 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 129 (IPC Server handler 67 on 60000): State: WAITING Blocked count: 153 Waited count: 1793 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 128 (IPC Server handler 66 on 60000): State: WAITING Blocked count: 119 Waited count: 1791 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 127 (IPC Server handler 65 on 60000): State: WAITING Blocked count: 103 Waited count: 1788 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 126 (IPC Server handler 64 on 60000): State: WAITING Blocked count: 136 Waited count: 1791 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 125 (IPC Server handler 63 on 60000): State: WAITING Blocked count: 111 Waited count: 1791 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 124 (IPC Server handler 62 on 60000): State: WAITING Blocked count: 144 Waited count: 1786 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 123 (IPC Server handler 61 on 60000): State: WAITING Blocked count: 126 Waited count: 1795 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 122 (IPC Server handler 60 on 60000): State: WAITING Blocked count: 127 Waited count: 1800 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 121 (IPC Server handler 59 on 60000): State: WAITING Blocked count: 107 Waited count: 1791 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 120 (IPC Server handler 58 on 60000): State: WAITING Blocked count: 143 Waited count: 1788 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 119 (IPC Server handler 57 on 60000): State: WAITING Blocked count: 135 Waited count: 1793 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 118 (IPC Server handler 56 on 60000): State: WAITING Blocked count: 151 Waited count: 1798 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 117 (IPC Server handler 55 on 60000): State: WAITING Blocked count: 116 Waited count: 1786 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 116 (IPC Server handler 54 on 60000): State: WAITING Blocked count: 109 Waited count: 1798 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 115 (IPC Server handler 53 on 60000): State: WAITING Blocked count: 131 Waited count: 1786 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 114 (IPC Server handler 52 on 60000): State: WAITING Blocked count: 122 Waited count: 1787 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 113 (IPC Server handler 51 on 60000): State: WAITING Blocked count: 126 Waited count: 1794 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 112 (IPC Server handler 50 on 60000): State: WAITING Blocked count: 117 Waited count: 1795 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 111 (IPC Server handler 49 on 60000): State: WAITING Blocked count: 112 Waited count: 1786 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 110 (IPC Server handler 48 on 60000): State: WAITING Blocked count: 149 Waited count: 1788 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 109 (IPC Server handler 47 on 60000): State: WAITING Blocked count: 120 Waited count: 1799 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 108 (IPC Server handler 46 on 60000): State: WAITING Blocked count: 114 Waited count: 1791 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 107 (IPC Server handler 45 on 60000): State: WAITING Blocked count: 130 Waited count: 1797 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 106 (IPC Server handler 44 on 60000): State: WAITING Blocked count: 127 Waited count: 1798 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 105 (IPC Server handler 43 on 60000): State: WAITING Blocked count: 134 Waited count: 1795 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 104 (IPC Server handler 42 on 60000): State: WAITING Blocked count: 102 Waited count: 1791 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 103 (IPC Server handler 41 on 60000): State: WAITING Blocked count: 100 Waited count: 1785 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 102 (IPC Server handler 40 on 60000): State: WAITING Blocked count: 156 Waited count: 1793 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 101 (IPC Server handler 39 on 60000): State: WAITING Blocked count: 131 Waited count: 1792 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 100 (IPC Server handler 38 on 60000): State: WAITING Blocked count: 121 Waited count: 1793 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 99 (IPC Server handler 37 on 60000): State: WAITING Blocked count: 117 Waited count: 1788 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 98 (IPC Server handler 36 on 60000): State: WAITING Blocked count: 129 Waited count: 1792 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 97 (IPC Server handler 35 on 60000): State: WAITING Blocked count: 115 Waited count: 1793 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 96 (IPC Server handler 34 on 60000): State: WAITING Blocked count: 138 Waited count: 1793 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 95 (IPC Server handler 33 on 60000): State: WAITING Blocked count: 143 Waited count: 1788 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 94 (IPC Server handler 32 on 60000): State: WAITING Blocked count: 145 Waited count: 1794 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 93 (IPC Server handler 31 on 60000): State: WAITING Blocked count: 139 Waited count: 1794 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 92 (IPC Server handler 30 on 60000): State: WAITING Blocked count: 139 Waited count: 1793 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 91 (IPC Server handler 29 on 60000): State: WAITING Blocked count: 137 Waited count: 1795 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 90 (IPC Server handler 28 on 60000): State: WAITING Blocked count: 128 Waited count: 1795 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 89 (IPC Server handler 27 on 60000): State: WAITING Blocked count: 144 Waited count: 1790 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 88 (IPC Server handler 26 on 60000): State: WAITING Blocked count: 130 Waited count: 1790 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 87 (IPC Server handler 25 on 60000): State: WAITING Blocked count: 154 Waited count: 1794 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 86 (IPC Server handler 24 on 60000): State: WAITING Blocked count: 135 Waited count: 1793 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 85 (IPC Server handler 23 on 60000): State: WAITING Blocked count: 152 Waited count: 1792 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 84 (IPC Server handler 22 on 60000): State: WAITING Blocked count: 143 Waited count: 1789 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 83 (IPC Server handler 21 on 60000): State: WAITING Blocked count: 150 Waited count: 1795 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 82 (IPC Server handler 20 on 60000): State: WAITING Blocked count: 120 Waited count: 1789 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 81 (IPC Server handler 19 on 60000): State: WAITING Blocked count: 141 Waited count: 1795 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 80 (IPC Server handler 18 on 60000): State: WAITING Blocked count: 167 Waited count: 1801 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 79 (IPC Server handler 17 on 60000): State: WAITING Blocked count: 137 Waited count: 1797 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 78 (IPC Server handler 16 on 60000): State: WAITING Blocked count: 116 Waited count: 1789 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 77 (IPC Server handler 15 on 60000): State: WAITING Blocked count: 109 Waited count: 1795 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 76 (IPC Server handler 14 on 60000): State: WAITING Blocked count: 120 Waited count: 1794 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 75 (IPC Server handler 13 on 60000): State: WAITING Blocked count: 170 Waited count: 1803 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 74 (IPC Server handler 12 on 60000): State: WAITING Blocked count: 160 Waited count: 1798 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 73 (IPC Server handler 11 on 60000): State: WAITING Blocked count: 149 Waited count: 1786 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 72 (IPC Server handler 10 on 60000): State: WAITING Blocked count: 142 Waited count: 1794 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 71 (IPC Server handler 9 on 60000): State: WAITING Blocked count: 128 Waited count: 1797 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 70 (IPC Server handler 8 on 60000): State: WAITING Blocked count: 116 Waited count: 1789 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 69 (IPC Server handler 7 on 60000): State: WAITING Blocked count: 124 Waited count: 1796 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 68 (IPC Server handler 6 on 60000): State: WAITING Blocked count: 158 Waited count: 1799 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 67 (IPC Server handler 5 on 60000): State: WAITING Blocked count: 130 Waited count: 1794 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 66 (IPC Server handler 4 on 60000): State: WAITING Blocked count: 137 Waited count: 1791 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 65 (IPC Server handler 3 on 60000): State: WAITING Blocked count: 152 Waited count: 1801 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 64 (IPC Server handler 2 on 60000): State: WAITING Blocked count: 99 Waited count: 1794 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 63 (IPC Server handler 1 on 60000): State: WAITING Blocked count: 155 Waited count: 1800 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 62 (IPC Server handler 0 on 60000): State: WAITING Blocked count: 146 Waited count: 1802 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@320c321f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Thread 20 (IPC Server listener on 60000): State: RUNNABLE Blocked count: 1 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84) org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:308) Thread 22 (IPC Server Responder): State: RUNNABLE Blocked count: 1 Waited count: 1 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:477) Thread 61 (Timer-1): State: TIMED_WAITING Blocked count: 0 Waited count: 1179 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:509) java.util.TimerThread.run(Timer.java:462) Thread 60 (Timer-0): State: TIMED_WAITING Blocked count: 1 Waited count: 1180 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:509) java.util.TimerThread.run(Timer.java:462) Thread 59 (691769463@qtp0-1 - Acceptor0 SelectChannelConnector@0.0.0.0:60010): State: RUNNABLE Blocked count: 1 Waited count: 1 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:429) org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:185) org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:707) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522) Thread 58 (342218962@qtp0-0): State: TIMED_WAITING Blocked count: 730 Waited count: 1313 Stack: java.lang.Object.wait(Native Method) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:565) Thread 28 (RegionManager.metaScanner): State: TIMED_WAITING Blocked count: 356505 Waited count: 353907 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91) org.apache.hadoop.hbase.Chore.run(Chore.java:76) Thread 27 (RegionManager.rootScanner): State: TIMED_WAITING Blocked count: 2962 Waited count: 3542 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91) org.apache.hadoop.hbase.Chore.run(Chore.java:76) Thread 35 (LeaseChecker): State: TIMED_WAITING Blocked count: 3 Waited count: 35291 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1070) java.lang.Thread.run(Thread.java:619) Thread 29 (DestroyJavaVM): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: Thread 9 (HMaster): State: RUNNABLE Blocked count: 176232 Waited count: 194879 Stack: sun.management.ThreadImpl.getThreadInfo0(Native Method) sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:147) sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:123) org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149) org.apache.hadoop.util.ReflectionUtils.logThreadInfo(ReflectionUtils.java:203) org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:71) org.apache.hadoop.hbase.master.ProcessRegionClose.process(ProcessRegionClose.java:63) org.apache.hadoop.hbase.master.HMaster.processToDoQueue(HMaster.java:496) org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:431) Thread 26 (Thread-11): State: TIMED_WAITING Blocked count: 0 Waited count: 590 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91) org.apache.hadoop.hbase.Chore.run(Chore.java:76) Thread 25 (main-EventThread): State: WAITING Blocked count: 1 Waited count: 122 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2da16f37 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:477) Thread 24 (main-SendThread(monster01.sf.cloudera.com:2222)): State: RUNNABLE Blocked count: 6 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1066) Thread 5 (Signal Dispatcher): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: Thread 3 (Finalizer): State: WAITING Blocked count: 867 Waited count: 859 Waiting on java.lang.ref.ReferenceQueue$Lock@3a838408 Stack: java.lang.Object.wait(Native Method) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134) java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) Thread 2 (Reference Handler): State: WAITING Blocked count: 874 Waited count: 864 Waiting on java.lang.ref.Reference$Lock@481b7adb Stack: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:485) java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) 2010-04-22 11:20:20,386 DEBUG org.apache.hadoop.hbase.master.ProcessRegionClose$1: Exception in RetryableMetaOperation: java.lang.NullPointerException at org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:74) at org.apache.hadoop.hbase.master.ProcessRegionClose.process(ProcessRegionClose.java:63) at org.apache.hadoop.hbase.master.HMaster.processToDoQueue(HMaster.java:496) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:431) 2010-04-22 11:20:20,387 WARN org.apache.hadoop.hbase.master.HMaster: Processing pending operations: ProcessRegionClose of TestTable,-926971162<1271834364111>,1271855470685, false, reassign: true java.lang.RuntimeException: java.lang.NullPointerException at org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:106) at org.apache.hadoop.hbase.master.ProcessRegionClose.process(ProcessRegionClose.java:63) at org.apache.hadoop.hbase.master.HMaster.processToDoQueue(HMaster.java:496) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:431) Caused by: java.lang.NullPointerException at org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:74) ... 3 more 2010-04-22 11:20:20,388 DEBUG org.apache.hadoop.hbase.master.HMaster: Processing todo: ProcessRegionClose of TestTable,-926971162<1271834364111>,1271855470685, false, reassign: true 2010-04-22 11:20:20,388 DEBUG org.apache.hadoop.hbase.master.ProcessRegionClose$1: Exception in RetryableMetaOperation: java.lang.NullPointerException at org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:74) at org.apache.hadoop.hbase.master.ProcessRegionClose.process(ProcessRegionClose.java:63) at org.apache.hadoop.hbase.master.HMaster.processToDoQueue(HMaster.java:496) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:431) 2010-04-22 11:20:20,391 WARN org.apache.hadoop.hbase.master.HMaster: Processing pending operations: ProcessRegionClose of TestTable,-926971162<1271834364111>,1271855470685, false, reassign: true java.lang.RuntimeException: java.lang.NullPointerException at org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:106) at org.apache.hadoop.hbase.master.ProcessRegionClose.process(ProcessRegionClose.java:63) at org.apache.hadoop.hbase.master.HMaster.processToDoQueue(HMaster.java:496) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:431) Caused by: java.lang.NullPointerException at org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:74) ... 3 more 2010-04-22 11:20:20,396 DEBUG org.apache.hadoop.hbase.master.HMaster: Processing todo: ProcessRegionClose of TestTable,-926971162<1271834364111>,1271855470685, false, reassign: true 2010-04-22 11:20:20,396 DEBUG org.apache.hadoop.hbase.master.ProcessRegionClose$1: Exception in RetryableMetaOperation: java.lang.NullPointerException at org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:74) at org.apache.hadoop.hbase.master.ProcessRegionClose.process(ProcessRegionClose.java:63) at org.apache.hadoop.hbase.master.HMaster.processToDoQueue(HMaster.java:496) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:431) 2010-04-22 11:20:20,397 WARN org.apache.hadoop.hbase.master.HMaster: Processing pending operations: ProcessRegionClose of TestTable,-926971162<1271834364111>,1271855470685, false, reassign: true java.lang.RuntimeException: java.lang.NullPointerException at org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:106) at org.apache.hadoop.hbase.master.ProcessRegionClose.process(ProcessRegionClose.java:63) at org.apache.hadoop.hbase.master.HMaster.processToDoQueue(HMaster.java:496) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:431) Caused by: java.lang.NullPointerException at org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:74) ... 3 more 2010-04-22 11:20:20,397 DEBUG org.apache.hadoop.hbase.master.HMaster: Processing todo: ProcessRegionClose of TestTable,-926971162<1271834364111>,1271855470685, false, reassign: true 2010-04-22 11:20:20,397 DEBUG org.apache.hadoop.hbase.master.ProcessRegionClose$1: Exception in RetryableMetaOperation: java.lang.NullPointerException at org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:74) at org.apache.hadoop.hbase.master.ProcessRegionClose.process(ProcessRegionClose.java:63) at org.apache.hadoop.hbase.master.HMaster.processToDoQueue(HMaster.java:496) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:431) 2010-04-22 11:20:20,398 WARN org.apache.hadoop.hbase.master.HMaster: Processing pending operations: ProcessRegionClose of TestTable,-926971162<1271834364111>,1271855470685, false, reassign: true java.lang.RuntimeException: java.lang.NullPointerException at org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:106) at org.apache.hadoop.hbase.master.ProcessRegionClose.process(ProcessRegionClose.java:63) at org.apache.hadoop.hbase.master.HMaster.processToDoQueue(HMaster.java:496) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:431) Caused by: java.lang.NullPointerException at org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:74) ... 3 more 2010-04-22 11:20:20,398 DEBUG org.apache.hadoop.hbase.master.HMaster: Processing todo: ProcessRegionClose of TestTable,-926971162<1271834364111>,1271855470685, false, reassign: true 2010-04-22 11:20:20,398 DEBUG org.apache.hadoop.hbase.master.ProcessRegionClose$1: Exception in RetryableMetaOperation: java.lang.NullPointerException at org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:74) at org.apache.hadoop.hbase.master.ProcessRegionClose.process(ProcessRegionClose.java:63) at org.apache.hadoop.hbase.master.HMaster.processToDoQueue(HMaster.java:496) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:431) 2010-04-22 11:20:20,398 WARN org.apache.hadoop.hbase.master.HMaster: Processing pending operations: ProcessRegionClose of TestTable,-926971162<1271834364111>,1271855470685, false, reassign: true java.lang.RuntimeException: java.lang.NullPointerException at org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:106) at org.apache.hadoop.hbase.master.ProcessRegionClose.process(ProcessRegionClose.java:63) at org.apache.hadoop.hbase.master.HMaster.processToDoQueue(HMaster.java:496) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:431) Caused by: java.lang.NullPointerException at org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:74) ... 3 more 2010-04-22 11:20:20,398 DEBUG org.apache.hadoop.hbase.master.HMaster: Processing todo: ProcessRegionClose of TestTable,-926971162<1271834364111>,1271855470685, false, reassign: true 2010-04-22 11:20:20,398 DEBUG org.apache.hadoop.hbase.master.ProcessRegionClose$1: Exception in RetryableMetaOperation: java.lang.NullPointerException at org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:74) at org.apache.hadoop.hbase.master.ProcessRegionClose.process(ProcessRegionClose.java:63) at org.apache.hadoop.hbase.master.HMaster.processToDoQueue(HMaster.java:496) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:431) 2010-04-22 11:20:20,399 WARN org.apache.hadoop.hbase.master.HMaster: Processing pending operations: ProcessRegionClose of TestTable,-926971162<1271834364111>,1271855470685, false, reassign: true java.lang.RuntimeException: java.lang.NullPointerException at org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:106) at org.apache.hadoop.hbase.master.ProcessRegionClose.process(ProcessRegionClose.java:63) at org.apache.hadoop.hbase.master.HMaster.processToDoQueue(HMaster.java:496) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:431) Caused by: java.lang.NullPointerException at org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:74) ... 3 more 2010-04-22 11:20:20,399 DEBUG org.apache.hadoop.hbase.master.HMaster: Processing todo: ProcessRegionClose of TestTable,-926971162<1271834364111>,1271855470685, false, reassign: true 2010-04-22 11:20:20,399 DEBUG org.apache.hadoop.hbase.master.ProcessRegionClose$1: Exception in RetryableMetaOperation: java.lang.NullPointerException at org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:74) at org.apache.hadoop.hbase.master.ProcessRegionClose.process(ProcessRegionClose.java:63) at org.apache.hadoop.hbase.master.HMaster.processToDoQueue(HMaster.java:496) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:431) 2010-04-22 11:20:20,399 WARN org.apache.hadoop.hbase.master.HMaster: Processing pending operations: ProcessRegionClose of TestTable,-926971162<1271834364111>,1271855470685, false, reassign: true java.lang.RuntimeException: java.lang.NullPointerException at org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:106) at org.apache.hadoop.hbase.master.ProcessRegionClose.process(ProcessRegionClose.java:63) at org.apache.hadoop.hbase.master.HMaster.processToDoQueue(HMaster.java:496) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:431) Caused by: java.lang.NullPointerException at org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:74) ... 3 more 2010-04-22 11:20:20,399 DEBUG org.apache.hadoop.hbase.master.HMaster: Processing todo: ProcessRegionClose of TestTable,-926971162<1271834364111>,1271855470685, false, reassign: true 2010-04-22 11:20:20,400 DEBUG org.apache.hadoop.hbase.master.ProcessRegionClose$1: Exception in RetryableMetaOperation: java.lang.NullPointerException at org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:74) at org.apache.hadoop.hbase.master.ProcessRegionClose.process(ProcessRegionClose.java:63) at org.apache.hadoop.hbase.master.HMaster.processToDoQueue(HMaster.java:496) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:431) 2010-04-22 11:20:20,400 WARN org.apache.hadoop.hbase.master.HMaster: Processing pending operations: ProcessRegionClose of TestTable,-926971162<1271834364111>,1271855470685, false, reassign: true java.lang.RuntimeException: java.lang.NullPointerException at org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:106) at org.apache.hadoop.hbase.master.ProcessRegionClose.process(ProcessRegionClose.java:63) at org.apache.hadoop.hbase.master.HMaster.processToDoQueue(HMaster.java:496) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:431) Caused by: java.lang.NullPointerException at org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:74) ... 3 more 2010-04-22 11:20:20,400 DEBUG org.apache.hadoop.hbase.master.HMaster: Processing todo: ProcessRegionClose of TestTable,-926971162<1271834364111>,1271855470685, false, reassign: true 2010-04-22 11:20:20,400 DEBUG org.apache.hadoop.hbase.master.ProcessRegionClose$1: Exception in RetryableMetaOperation: java.lang.NullPointerException at org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:74) at org.apache.hadoop.hbase.master.ProcessRegionClose.process(ProcessRegionClose.java:63) at org.apache.hadoop.hbase.master.HMaster.processToDoQueue(HMaster.java:496) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:431) 2010-04-22 11:20:20,400 WARN org.apache.hadoop.hbase.master.HMaster: Processing pending operations: ProcessRegionClose of TestTable,-926971162<1271834364111>,1271855470685, false, reassign: true java.lang.RuntimeException: java.lang.NullPointerException at org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:106) at org.apache.hadoop.hbase.master.ProcessRegionClose.process(ProcessRegionClose.java:63) at org.apache.hadoop.hbase.master.HMaster.processToDoQueue(HMaster.java:496) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:431) Caused by: java.lang.NullPointerException at org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:74) ... 3 more 2010-04-22 11:20:20,400 DEBUG org.apache.hadoop.hbase.master.HMaster: Processing todo: ProcessRegionClose of TestTable,-926971162<1271834364111>,1271855470685, false, reassign: true 2010-04-22 11:20:20,400 DEBUG org.apache.hadoop.hbase.master.ProcessRegionClose$1: Exception in RetryableMetaOperation: java.lang.NullPointerException at org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:74) at org.apache.hadoop.hbase.master.ProcessRegionClose.process(ProcessRegionClose.java:63) at org.apache.hadoop.hbase.master.HMaster.processToDoQueue(HMaster.java:496) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:431) 2010-04-22 11:20:20,401 WARN org.apache.hadoop.hbase.master.HMaster: Processing pending operations: ProcessRegionClose of TestTable,-926971162<1271834364111>,1271855470685, false, reassign: true java.lang.RuntimeException: java.lang.NullPointerException at org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:106) at org.apache.hadoop.hbase.master.ProcessRegionClose.process(ProcessRegionClose.java:63) at org.apache.hadoop.hbase.master.HMaster.processToDoQueue(HMaster.java:496) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:431) Caused by: java.lang.NullPointerException at org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:74) ... 3 more 2010-04-22 11:20:20,401 DEBUG org.apache.hadoop.hbase.master.HMaster: Processing todo: ProcessRegionClose of TestTable,-926971162<1271834364111>,1271855470685, false, reassign: true 2010-04-22 11:20:20,401 DEBUG org.apache.hadoop.hbase.master.ProcessRegionClose$1: Exception in RetryableMetaOperation: java.lang.NullPointerException at org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:74) at org.apache.hadoop.hbase.master.ProcessRegionClose.process(ProcessRegionClose.java:63) at org.apache.hadoop.hbase.master.HMaster.processToDoQueue(HMaster.java:496) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:431) 2010-04-22 11:20:20,401 WARN org.apache.hadoop.hbase.master.HMaster: Processing pending operations: ProcessRegionClose of TestTable,-926971162<1271834364111>,1271855470685, false, reassign: true java.lang.RuntimeException: java.lang.NullPointerException at org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:106) at org.apache.hadoop.hbase.master.ProcessRegionClose.process(ProcessRegionClose.java:63) at org.apache.hadoop.hbase.master.HMaster.processToDoQueue(HMaster.java:496) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:431) Caused by: java.lang.NullPointerException at org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:74) ... 3 more 2010-04-22 11:20:20,401 DEBUG org.apache.hadoop.hbase.master.HMaster: Processing todo: ProcessRegionClose of TestTable,-926971162<1271834364111>,1271855470685, false, reassign: true 2010-04-22 11:20:20,401 DEBUG org.apache.hadoop.hbase.master.ProcessRegionClose$1: Exception in RetryableMetaOperation: java.lang.NullPointerException at org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:74) at org.apache.hadoop.hbase.master.ProcessRegionClose.process(ProcessRegionClose.java:63) at org.apache.hadoop.hbase.master.HMaster.processToDoQueue(HMaster.java:496) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:431) 2010-04-22 11:20:20,402 WARN org.apache.hadoop.hbase.master.HMaster: Processing pending operations: ProcessRegionClose of TestTable,-926971162<1271834364111>,1271855470685, false, reassign: true java.lang.RuntimeException: java.lang.NullPointerException at org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:106) at org.apache.hadoop.hbase.master.ProcessRegionClose.process(ProcessRegionClose.java:63) at org.apache.hadoop.hbase.master.HMaster.processToDoQueue(HMaster.java:496) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:431) Caused by: java.lang.NullPointerException at org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:74) ... 3 more 2010-04-22 11:20:20,402 DEBUG org.apache.hadoop.hbase.master.HMaster: Processing todo: ProcessRegionClose of TestTable,-926971162<1271834364111>,1271855470685, false, reassign: true 2010-04-22 11:20:20,402 DEBUG org.apache.hadoop.hbase.master.ProcessRegionClose$1: Exception in RetryableMetaOperation: java.lang.NullPointerException at org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:74) at org.apache.hadoop.hbase.master.ProcessRegionClose.process(ProcessRegionClose.java:63) at org.apache.hadoop.hbase.master.HMaster.processToDoQueue(HMaster.java:496) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:431) 2010-04-22 11:20:20,402 WARN org.apache.hadoop.hbase.master.HMaster: Processing pending operations: ProcessRegionClose of TestTable,-926971162<1271834364111>,1271855470685, false, reassign: true java.lang.RuntimeException: java.lang.NullPointerException at org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:106) at org.apache.hadoop.hbase.master.ProcessRegionClose.process(ProcessRegionClose.java:63) at org.apache.hadoop.hbase.master.HMaster.processToDoQueue(HMaster.java:496) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:431) Caused by: java.lang.NullPointerException at org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:74) ... 3 more 2010-04-22 11:20:20,402 DEBUG org.apache.hadoop.hbase.master.HMaster: Processing todo: ProcessRegionClose of TestTable,-926971162<1271834364111>,1271855470685, false, reassign: true 2010-04-22 11:20:20,402 DEBUG org.apache.hadoop.hbase.master.ProcessRegionClose$1: Exception in RetryableMetaOperation: java.lang.NullPointerException at org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:74) at org.apache.hadoop.hbase.master.ProcessRegionClose.process(ProcessRegionClose.java:63) at org.apache.hadoop.hbase.master.HMaster.processToDoQueue(HMaster.java:496) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:431) 2010-04-22 11:20:20,403 WARN org.apache.hadoop.hbase.master.HMaster: Processing pending operations: ProcessRegionClose of TestTable,-926971162<1271834364111>,1271855470685, false, reassign: true java.lang.RuntimeException: java.lang.NullPointerException at org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:106) at org.apache.hadoop.hbase.master.ProcessRegionClose.process(ProcessRegionClose.java:63) at org.apache.hadoop.hbase.master.HMaster.processToDoQueue(HMaster.java:496) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:431) Caused by: java.lang.NullPointerException at org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:74) ... 3 more 2010-04-22 11:20:20,403 DEBUG org.apache.hadoop.hbase.master.HMaster: Processing todo: ProcessRegionClose of TestTable,-926971162<1271834364111>,1271855470685, false, reassign: true 2010-04-22 11:20:20,403 DEBUG org.apache.hadoop.hbase.master.ProcessRegionClose$1: Exception in RetryableMetaOperation: java.lang.NullPointerException at org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:74) at org.apache.hadoop.hbase.master.ProcessRegionClose.process(ProcessRegionClose.java:63) at org.apache.hadoop.hbase.master.HMaster.processToDoQueue(HMaster.java:496) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:431) 2010-04-22 11:20:20,403 WARN org.apache.hadoop.hbase.master.HMaster: Processing pending operations: ProcessRegionClose of TestTable,-926971162<1271834364111>,1271855470685, false, reassign: true java.lang.RuntimeException: java.lang.NullPointerException at org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:106) at org.apache.hadoop.hbase.master.ProcessRegionClose.process(ProcessRegionClose.java:63) at org.apache.hadoop.hbase.master.HMaster.processToDoQueue(HMaster.java:496) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:431) Caused by: java.lang.NullPointerException at org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:74) ... 3 more 2010-04-22 11:20:20,403 DEBUG org.apache.hadoop.hbase.master.HMaster: Processing todo: ProcessRegionClose of TestTable,-926971162<1271834364111>,1271855470685, false, reassign: true 2010-04-22 11:20:20,403 DEBUG org.apache.hadoop.hbase.master.ProcessRegionClose$1: Exception in RetryableMetaOperation: java.lang.NullPointerException at org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:74) at org.apache.hadoop.hbase.master.ProcessRegionClose.process(ProcessRegionClose.java:63) at org.apache.hadoop.hbase.master.HMaster.processToDoQueue(HMaster.java:496) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:431) 2010-04-22 11:20:20,404 WARN org.apache.hadoop.hbase.master.HMaster: Processing pending operations: ProcessRegionClose of TestTable,-926971162<1271834364111>,1271855470685, false, reassign: true java.lang.RuntimeException: java.lang.NullPointerException at org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:106) at org.apache.hadoop.hbase.master.ProcessRegionClose.process(ProcessRegionClose.java:63) at org.apache.hadoop.hbase.master.HMaster.processToDoQueue(HMaster.java:496) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:431) Caused by: java.lang.NullPointerException at org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:74) ... 3 more 2010-04-22 11:20:20,404 DEBUG org.apache.hadoop.hbase.master.HMaster: Processing todo: ProcessRegionClose of TestTable,-926971162<1271834364111>,1271855470685, false, reassign: true 2010-04-22 11:20:20,404 DEBUG org.apache.hadoop.hbase.master.ProcessRegionClose$1: Exception in RetryableMetaOperation: java.lang.NullPointerException at org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:74) at org.apache.hadoop.hbase.master.ProcessRegionClose.process(ProcessRegionClose.java:63) at org.apache.hadoop.hbase.master.HMaster.processToDoQueue(HMaster.java:496) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:431) 2010-04-22 11:20:20,404 WARN org.apache.hadoop.hbase.master.HMaster: Processing pending operations: ProcessRegionClose of TestTable,-926971162<1271834364111>,1271855470685, false, reassign: true java.lang.RuntimeException: java.lang.NullPointerException at org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:106) at org.apache.hadoop.hbase.master.ProcessRegionClose.process(ProcessRegionClose.java:63) at org.apache.hadoop.hbase.master.HMaster.processToDoQueue(HMaster.java:496) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:431) Caused by: java.lang.NullPointerException at org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:74) ... 3 more