RegionServer status for hbase500.ash1.facebook.com,16020,1435179105838 as of Thu Jun 25 11:58:16 PDT 2015 Version Info: =========================================================== HBase 1.3.0-SNAPSHOT Source code repository git://dev392.prn1.facebook.com/data/users/elliott/hbase revision=41aa8412410fec33242afa40bb7c52c511eb30e2 Compiled by elliott on Wed Jun 24 13:36:22 PDT 2015 From source with checksum 6f3da2b1f6c7e9e641bd046dce051ef2 Hadoop 2.6.0 Source code repository https://git-wip-us.apache.org/repos/asf/hadoop.git revision=e3496499ecb8d220fba99dc5ed4c99c8f9e33bb1 (HEAD, tag: release-2.6.0-rc1, origin/branch-2.6.0, branch-2.6.0) Compiled by acmurthy on 2014-11-13T22:35Z Tasks: =========================================================== Task: Flushing IntegrationTestLoadAndVerify,R\x9B\x1B\xBF\xAE\x08\xD1\xA2,1435179555993.8e2d075f94ce7699f416ec4ced9873cd. Status: RUNNING:Preparing to flush by snapshotting stores in 8e2d075f94ce7699f416ec4ced9873cd Running for 22730s Task: Flushing IntegrationTestLoadAndVerify,\x93\xA385\x81Z\x11\xE6,1435179555993.9f8d0e01a40405b835bf6e5a22a86390. Status: RUNNING:Preparing to flush by snapshotting stores in 9f8d0e01a40405b835bf6e5a22a86390 Running for 22729s Executors: =========================================================== Status for executor: Executor-2-RS_OPEN_META-hbase500:16020 ======================================= 0 events queued, 0 running Status for executor: Executor-3-RS_CLOSE_REGION-hbase500:16020 ======================================= 0 events queued, 0 running Status for executor: Executor-1-RS_OPEN_REGION-hbase500:16020 ======================================= 0 events queued, 0 running Status for executor: Executor-4-RS_CLOSE_META-hbase500:16020 ======================================= 0 events queued, 0 running Status for executor: Executor-6-RS_REGION_REPLICA_FLUSH_OPS-hbase500:16020 ======================================= 0 events queued, 0 running Status for executor: Executor-5-RS_LOG_REPLAY_OPS-hbase500:16020 ======================================= 0 events queued, 0 running Process Thread Dump: 133 active threads Thread 38579 (323584516@qtp-1667534569-46): State: RUNNABLE Blocked count: 188 Waited count: 187 Stack: sun.management.ThreadImpl.getThreadInfo1(Native Method) sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:174) sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:139) org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:165) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:483) org.apache.hadoop.hbase.util.Threads.printThreadInfo(Threads.java:302) org.apache.hadoop.hbase.regionserver.RSDumpServlet.doGet(RSDumpServlet.java:79) javax.servlet.http.HttpServlet.service(HttpServlet.java:707) javax.servlet.http.HttpServlet.service(HttpServlet.java:820) org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:113) org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) org.apache.hadoop.hbase.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:1350) org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) org.apache.hadoop.hbase.http.NoCacheFilter.doFilter(NoCacheFilter.java:49) org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) Thread 37708 (ResponseProcessor for block BP-2061608446-10.188.6.53-1428515260018:blk_1082193112_8466504): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118) java.io.FilterInputStream.read(FilterInputStream.java:83) java.io.FilterInputStream.read(FilterInputStream.java:83) org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2201) org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176) org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:867) Thread 37707 (DataStreamer for file /HBASECI001-HBASE/WALs/hbase500.ash1.facebook.com,16020,1435179105838/hbase500.ash1.facebook.com%2C16020%2C1435179105838.null1.1435233743403 block BP-2061608446-10.188.6.53-1428515260018:blk_1082193112_8466504): State: TIMED_WAITING Blocked count: 3 Waited count: 836 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:564) Thread 37706 (ResponseProcessor for block BP-2061608446-10.188.6.53-1428515260018:blk_1082193107_8466498): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118) java.io.FilterInputStream.read(FilterInputStream.java:83) java.io.FilterInputStream.read(FilterInputStream.java:83) org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2201) org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176) org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:867) Thread 37680 (DataStreamer for file /HBASECI001-HBASE/WALs/hbase500.ash1.facebook.com,16020,1435179105838/hbase500.ash1.facebook.com%2C16020%2C1435179105838.null2.1435233528000 block BP-2061608446-10.188.6.53-1428515260018:blk_1082193107_8466498): State: TIMED_WAITING Blocked count: 28 Waited count: 865 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:564) Thread 37673 (ResponseProcessor for block BP-2061608446-10.188.6.53-1428515260018:blk_1082192017_8465329): State: RUNNABLE Blocked count: 2 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118) java.io.FilterInputStream.read(FilterInputStream.java:83) java.io.FilterInputStream.read(FilterInputStream.java:83) org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2201) org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176) org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:867) Thread 37672 (DataStreamer for file /HBASECI001-HBASE/WALs/hbase500.ash1.facebook.com,16020,1435179105838/hbase500.ash1.facebook.com%2C16020%2C1435179105838.null0.1435233457735 block BP-2061608446-10.188.6.53-1428515260018:blk_1082192017_8465329): State: TIMED_WAITING Blocked count: 4083 Waited count: 4922 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:564) Thread 36063 (IPC Parameter Sending Thread #10): State: TIMED_WAITING Blocked count: 28 Waited count: 17733 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 32001 (RS_LOG_REPLAY_OPS-hbase500:16020-1): State: WAITING Blocked count: 55 Waited count: 61 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73b3203f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 25689 (RS_LOG_REPLAY_OPS-hbase500:16020-0): State: WAITING Blocked count: 72 Waited count: 89 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73b3203f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 23848 (ReplicationExecutor-0): State: WAITING Blocked count: 3 Waited count: 9 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@567f170c Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 3509 (hbase500.ash1.facebook.com,16020,1435179105838_ChoreService_3): State: WAITING Blocked count: 11 Waited count: 11176 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@61da4ab8 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 3506 (hbase500.ash1.facebook.com,16020,1435179105838_ChoreService_2): State: WAITING Blocked count: 24 Waited count: 11363 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@61da4ab8 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 1028 (regionserver/hbase500.ash1.facebook.com/10.188.6.43:16020-shortCompactions-1435179897520): State: WAITING Blocked count: 54064 Waited count: 96147 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@485bd6c1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 1001 (regionserver/hbase500.ash1.facebook.com/10.188.6.43:16020-shortCompactions-1435179883351): State: WAITING Blocked count: 53260 Waited count: 94584 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@485bd6c1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 996 (regionserver/hbase500.ash1.facebook.com/10.188.6.43:16020-shortCompactions-1435179882124): State: WAITING Blocked count: 51944 Waited count: 92546 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@485bd6c1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 165 (RS_OPEN_REGION-hbase500:16020-2): State: WAITING Blocked count: 669 Waited count: 953 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6e2179c7 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 164 (RS_OPEN_REGION-hbase500:16020-1): State: WAITING Blocked count: 738 Waited count: 958 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6e2179c7 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 163 (RS_OPEN_REGION-hbase500:16020-0): State: WAITING Blocked count: 719 Waited count: 983 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6e2179c7 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 162 (org.apache.hadoop.hdfs.PeerCache@48ee2a8f): State: TIMED_WAITING Blocked count: 0 Waited count: 26026 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:244) org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41) org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119) java.lang.Thread.run(Thread.java:745) Thread 146 (SplitLogWorker-hbase500:16020): State: TIMED_WAITING Blocked count: 1598 Waited count: 17328 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.coordination.ZkSplitLogWorkerCoordination.taskLoop(ZkSplitLogWorkerCoordination.java:445) org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:142) java.lang.Thread.run(Thread.java:745) Thread 145 (hbase500:16020Replication Statistics #0): State: TIMED_WAITING Blocked count: 0 Waited count: 266 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 143 (regionserver/hbase500.ash1.facebook.com/10.188.6.43:16020-EventThread): State: WAITING Blocked count: 0 Waited count: 3 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7d40b826 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) Thread 142 (regionserver/hbase500.ash1.facebook.com/10.188.6.43:16020-SendThread(hbasectrl023.ash1.facebook.com:2181)): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) Thread 103 (regionserver/hbase500.ash1.facebook.com/10.188.6.43:16020.leaseChecker): State: TIMED_WAITING Blocked count: 0 Waited count: 7901 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.regionserver.Leases.run(Leases.java:94) java.lang.Thread.run(Thread.java:745) Thread 141 (hbase500.ash1.facebook.com,16020,1435179105838_ChoreService_1): State: TIMED_WAITING Blocked count: 283 Waited count: 11563 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 139 (MemStoreFlusher.1): State: WAITING Blocked count: 139711 Waited count: 239212 Waiting on java.util.concurrent.CountDownLatch$Sync@b9c094a Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231) org.apache.hadoop.hbase.wal.WALKey.getSequenceId(WALKey.java:305) org.apache.hadoop.hbase.regionserver.HRegion.getNextSequenceId(HRegion.java:2422) org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2168) org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2047) org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2011) org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1902) org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:1828) org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:510) org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:471) org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$900(MemStoreFlusher.java:75) org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:259) java.lang.Thread.run(Thread.java:745) Thread 137 (MemStoreFlusher.0): State: WAITING Blocked count: 138931 Waited count: 237448 Waiting on java.util.concurrent.CountDownLatch$Sync@53f41f76 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231) org.apache.hadoop.hbase.wal.WALKey.getSequenceId(WALKey.java:305) org.apache.hadoop.hbase.regionserver.HRegion.getNextSequenceId(HRegion.java:2422) org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2168) org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2047) org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2011) org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1902) org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:1828) org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:510) org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:471) org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$900(MemStoreFlusher.java:75) org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:259) java.lang.Thread.run(Thread.java:745) Thread 96 (regionserver/hbase500.ash1.facebook.com/10.188.6.43:16020.logRoller): State: RUNNABLE Blocked count: 25566 Waited count: 46707 Stack: java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1018) java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328) java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277) org.apache.hadoop.hbase.regionserver.wal.FSHLog$SafePointZigZagLatch.waitSafePoint(FSHLog.java:1613) org.apache.hadoop.hbase.regionserver.wal.FSHLog.replaceWriter(FSHLog.java:799) org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:679) org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:137) java.lang.Thread.run(Thread.java:745) Thread 136 (HBase-Metrics2-1): State: TIMED_WAITING Blocked count: 10 Waited count: 19671 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 134 (sync.4): State: WAITING Blocked count: 7556433 Waited count: 9652692 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@37664813 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1183) java.lang.Thread.run(Thread.java:745) Thread 133 (sync.3): State: WAITING Blocked count: 7557308 Waited count: 9653202 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@698f9c60 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1183) java.lang.Thread.run(Thread.java:745) Thread 132 (sync.2): State: WAITING Blocked count: 7560314 Waited count: 9655925 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f96250a Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1183) java.lang.Thread.run(Thread.java:745) Thread 131 (sync.1): State: WAITING Blocked count: 7562676 Waited count: 9657510 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@59f70edf Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1183) java.lang.Thread.run(Thread.java:745) Thread 130 (sync.0): State: WAITING Blocked count: 7560367 Waited count: 9656420 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5cd4e322 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1183) java.lang.Thread.run(Thread.java:745) Thread 135 (regionserver/hbase500.ash1.facebook.com/10.188.6.43:16020.append-pool3-t1): State: WAITING Blocked count: 157878 Waited count: 9264322 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@49ad0844 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:45) com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:55) com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:123) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 126 (sync.4): State: WAITING Blocked count: 7606311 Waited count: 9717470 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@164da3d2 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1183) java.lang.Thread.run(Thread.java:745) Thread 125 (sync.3): State: WAITING Blocked count: 7606727 Waited count: 9717497 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6dfb9fca Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1183) java.lang.Thread.run(Thread.java:745) Thread 124 (sync.2): State: WAITING Blocked count: 7605480 Waited count: 9716650 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4cb55519 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1183) java.lang.Thread.run(Thread.java:745) Thread 123 (sync.1): State: WAITING Blocked count: 7604070 Waited count: 9715117 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@55d19b2b Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1183) java.lang.Thread.run(Thread.java:745) Thread 122 (sync.0): State: WAITING Blocked count: 7605391 Waited count: 9717027 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2ef9e170 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1183) java.lang.Thread.run(Thread.java:745) Thread 127 (regionserver/hbase500.ash1.facebook.com/10.188.6.43:16020.append-pool2-t1): State: TIMED_WAITING Blocked count: 157821 Waited count: 31852396 Stack: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:460) org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.attainSafePoint(FSHLog.java:1786) org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1761) org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1672) com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 118 (sync.4): State: WAITING Blocked count: 7422678 Waited count: 9497629 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@20677909 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1183) java.lang.Thread.run(Thread.java:745) Thread 117 (sync.3): State: WAITING Blocked count: 7421600 Waited count: 9496242 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@41f4ee14 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1183) java.lang.Thread.run(Thread.java:745) Thread 116 (sync.2): State: WAITING Blocked count: 7422611 Waited count: 9497272 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@373a30ee Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1183) java.lang.Thread.run(Thread.java:745) Thread 115 (sync.1): State: WAITING Blocked count: 7422667 Waited count: 9497123 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7b2e6b4b Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1183) java.lang.Thread.run(Thread.java:745) Thread 114 (sync.0): State: WAITING Blocked count: 7422961 Waited count: 9497626 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@609aea16 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1183) java.lang.Thread.run(Thread.java:745) Thread 119 (regionserver/hbase500.ash1.facebook.com/10.188.6.43:16020.append-pool1-t1): State: WAITING Blocked count: 147671 Waited count: 8962755 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6f83c5f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:45) com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:55) com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:123) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 112 (LeaseRenewer:hadoop@hbaseci001): State: TIMED_WAITING Blocked count: 2607 Waited count: 83444 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:438) org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) java.lang.Thread.run(Thread.java:745) Thread 108 (LruBlockCacheStatsExecutor): State: TIMED_WAITING Blocked count: 0 Waited count: 266 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 106 (regionserver/hbase500.ash1.facebook.com/10.188.6.43:16020.LruBlockCache.EvictionThread): State: TIMED_WAITING Blocked count: 0 Waited count: 7901 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:803) java.lang.Thread.run(Thread.java:745) Thread 105 (IPC Client (1541525668) connection to hbasectrl023.ash1.facebook.com/10.188.6.55:16000 from hadoop): State: TIMED_WAITING Blocked count: 26165 Waited count: 103601 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.waitForWork(RpcClientImpl.java:546) org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.run(RpcClientImpl.java:561) Thread 102 (regionserver/hbase500.ash1.facebook.com/10.188.6.43:16020-longCompactions-1435179107285): State: WAITING Blocked count: 24813 Waited count: 43488 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3de77a9b Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:90) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 101 (regionserver/hbase500.ash1.facebook.com/10.188.6.43:16020-longCompactions-1435179107284): State: WAITING Blocked count: 24395 Waited count: 42524 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3de77a9b Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:90) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 100 (regionserver/hbase500.ash1.facebook.com/10.188.6.43:16020-longCompactions-1435179107284): State: WAITING Blocked count: 24332 Waited count: 42608 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3de77a9b Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:90) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 99 (JvmPauseMonitor): State: TIMED_WAITING Blocked count: 3 Waited count: 156087 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:159) java.lang.Thread.run(Thread.java:745) Thread 98 (regionserver/hbase500.ash1.facebook.com/10.188.6.43:16020-EventThread): State: WAITING Blocked count: 0 Waited count: 3 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5b6d03cf Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) Thread 97 (regionserver/hbase500.ash1.facebook.com/10.188.6.43:16020-SendThread(hbasectrl025.ash1.facebook.com:2181)): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) Thread 22 (regionserver/hbase500.ash1.facebook.com/10.188.6.43:16020): State: TIMED_WAITING Blocked count: 26401 Waited count: 52333 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:92) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:56) org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:952) java.lang.Thread.run(Thread.java:745) Thread 95 (Timer-0): State: TIMED_WAITING Blocked count: 1 Waited count: 2647 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 94 (1548271808@qtp-1667534569-1 - Acceptor0 SelectChannelConnector@0.0.0.0:16030): State: RUNNABLE Blocked count: 973 Waited count: 1 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498) org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192) org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Thread 91 (ReplicationRpcServer.handler=2,queue=0,port=16020): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1c5a76e Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 90 (ReplicationRpcServer.handler=1,queue=0,port=16020): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1c5a76e Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 89 (ReplicationRpcServer.handler=0,queue=0,port=16020): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1c5a76e Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 88 (PriorityRpcServer.handler=19,queue=1,port=16020): State: TIMED_WAITING Blocked count: 1882174 Waited count: 2175968 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:167) org.apache.hadoop.hbase.regionserver.wal.FSHLog.blockOnSync(FSHLog.java:1332) org.apache.hadoop.hbase.regionserver.wal.FSHLog.publishSyncThenBlockOnCompletion(FSHLog.java:1326) org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1460) org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:7792) org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3154) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2802) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2744) org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:696) org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:658) org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2033) org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32594) org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 87 (PriorityRpcServer.handler=18,queue=0,port=16020): State: TIMED_WAITING Blocked count: 1880396 Waited count: 2174263 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:167) org.apache.hadoop.hbase.regionserver.wal.FSHLog.blockOnSync(FSHLog.java:1332) org.apache.hadoop.hbase.regionserver.wal.FSHLog.publishSyncThenBlockOnCompletion(FSHLog.java:1326) org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1460) org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:7792) org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3154) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2802) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2744) org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:696) org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:658) org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2033) org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32594) org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 86 (PriorityRpcServer.handler=17,queue=1,port=16020): State: TIMED_WAITING Blocked count: 1885427 Waited count: 2179106 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:167) org.apache.hadoop.hbase.regionserver.wal.FSHLog.blockOnSync(FSHLog.java:1332) org.apache.hadoop.hbase.regionserver.wal.FSHLog.publishSyncThenBlockOnCompletion(FSHLog.java:1326) org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1460) org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:7792) org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3154) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2802) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2744) org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:696) org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:658) org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2033) org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32594) org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 85 (PriorityRpcServer.handler=16,queue=0,port=16020): State: TIMED_WAITING Blocked count: 1880810 Waited count: 2174431 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:167) org.apache.hadoop.hbase.regionserver.wal.FSHLog.blockOnSync(FSHLog.java:1332) org.apache.hadoop.hbase.regionserver.wal.FSHLog.publishSyncThenBlockOnCompletion(FSHLog.java:1326) org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1460) org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:7792) org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3154) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2802) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2744) org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:696) org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:658) org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2033) org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32594) org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 84 (PriorityRpcServer.handler=15,queue=1,port=16020): State: TIMED_WAITING Blocked count: 1881032 Waited count: 2174796 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:167) org.apache.hadoop.hbase.regionserver.wal.FSHLog.blockOnSync(FSHLog.java:1332) org.apache.hadoop.hbase.regionserver.wal.FSHLog.publishSyncThenBlockOnCompletion(FSHLog.java:1326) org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1460) org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:7792) org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3154) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2802) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2744) org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:696) org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:658) org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2033) org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32594) org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 83 (PriorityRpcServer.handler=14,queue=0,port=16020): State: TIMED_WAITING Blocked count: 1881265 Waited count: 2174612 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:167) org.apache.hadoop.hbase.regionserver.wal.FSHLog.blockOnSync(FSHLog.java:1332) org.apache.hadoop.hbase.regionserver.wal.FSHLog.publishSyncThenBlockOnCompletion(FSHLog.java:1326) org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1460) org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:7792) org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3154) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2802) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2744) org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:696) org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:658) org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2033) org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32594) org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 82 (PriorityRpcServer.handler=13,queue=1,port=16020): State: TIMED_WAITING Blocked count: 1881535 Waited count: 2175253 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:167) org.apache.hadoop.hbase.regionserver.wal.FSHLog.blockOnSync(FSHLog.java:1332) org.apache.hadoop.hbase.regionserver.wal.FSHLog.publishSyncThenBlockOnCompletion(FSHLog.java:1326) org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1460) org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:7792) org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3154) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2802) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2744) org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:696) org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:658) org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2033) org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32594) org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 81 (PriorityRpcServer.handler=12,queue=0,port=16020): State: TIMED_WAITING Blocked count: 1880606 Waited count: 2174013 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:167) org.apache.hadoop.hbase.regionserver.wal.FSHLog.blockOnSync(FSHLog.java:1332) org.apache.hadoop.hbase.regionserver.wal.FSHLog.publishSyncThenBlockOnCompletion(FSHLog.java:1326) org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1460) org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:7792) org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3154) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2802) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2744) org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:696) org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:658) org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2033) org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32594) org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 80 (PriorityRpcServer.handler=11,queue=1,port=16020): State: TIMED_WAITING Blocked count: 1882966 Waited count: 2176704 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:167) org.apache.hadoop.hbase.regionserver.wal.FSHLog.blockOnSync(FSHLog.java:1332) org.apache.hadoop.hbase.regionserver.wal.FSHLog.publishSyncThenBlockOnCompletion(FSHLog.java:1326) org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1460) org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:7792) org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3154) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2802) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2744) org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:696) org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:658) org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2033) org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32594) org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 79 (PriorityRpcServer.handler=10,queue=0,port=16020): State: TIMED_WAITING Blocked count: 1881164 Waited count: 2174741 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:167) org.apache.hadoop.hbase.regionserver.wal.FSHLog.blockOnSync(FSHLog.java:1332) org.apache.hadoop.hbase.regionserver.wal.FSHLog.publishSyncThenBlockOnCompletion(FSHLog.java:1326) org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1460) org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:7792) org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3154) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2802) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2744) org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:696) org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:658) org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2033) org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32594) org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 78 (PriorityRpcServer.handler=9,queue=1,port=16020): State: TIMED_WAITING Blocked count: 1882201 Waited count: 2175988 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:167) org.apache.hadoop.hbase.regionserver.wal.FSHLog.blockOnSync(FSHLog.java:1332) org.apache.hadoop.hbase.regionserver.wal.FSHLog.publishSyncThenBlockOnCompletion(FSHLog.java:1326) org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1460) org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:7792) org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3154) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2802) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2744) org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:696) org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:658) org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2033) org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32594) org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 77 (PriorityRpcServer.handler=8,queue=0,port=16020): State: TIMED_WAITING Blocked count: 1881413 Waited count: 2174935 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:167) org.apache.hadoop.hbase.regionserver.wal.FSHLog.blockOnSync(FSHLog.java:1332) org.apache.hadoop.hbase.regionserver.wal.FSHLog.publishSyncThenBlockOnCompletion(FSHLog.java:1326) org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1460) org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:7792) org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3154) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2802) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2744) org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:696) org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:658) org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2033) org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32594) org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 76 (PriorityRpcServer.handler=7,queue=1,port=16020): State: TIMED_WAITING Blocked count: 1883751 Waited count: 2177500 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:167) org.apache.hadoop.hbase.regionserver.wal.FSHLog.blockOnSync(FSHLog.java:1332) org.apache.hadoop.hbase.regionserver.wal.FSHLog.publishSyncThenBlockOnCompletion(FSHLog.java:1326) org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1460) org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:7792) org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3154) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2802) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2744) org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:696) org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:658) org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2033) org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32594) org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 75 (PriorityRpcServer.handler=6,queue=0,port=16020): State: TIMED_WAITING Blocked count: 1880918 Waited count: 2174369 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:167) org.apache.hadoop.hbase.regionserver.wal.FSHLog.blockOnSync(FSHLog.java:1332) org.apache.hadoop.hbase.regionserver.wal.FSHLog.publishSyncThenBlockOnCompletion(FSHLog.java:1326) org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1460) org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:7792) org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3154) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2802) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2744) org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:696) org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:658) org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2033) org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32594) org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 74 (PriorityRpcServer.handler=5,queue=1,port=16020): State: TIMED_WAITING Blocked count: 1881278 Waited count: 2174914 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:167) org.apache.hadoop.hbase.regionserver.wal.FSHLog.blockOnSync(FSHLog.java:1332) org.apache.hadoop.hbase.regionserver.wal.FSHLog.publishSyncThenBlockOnCompletion(FSHLog.java:1326) org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1460) org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:7792) org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3154) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2802) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2744) org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:696) org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:658) org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2033) org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32594) org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 73 (PriorityRpcServer.handler=4,queue=0,port=16020): State: TIMED_WAITING Blocked count: 1879957 Waited count: 2173664 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:167) org.apache.hadoop.hbase.regionserver.wal.FSHLog.blockOnSync(FSHLog.java:1332) org.apache.hadoop.hbase.regionserver.wal.FSHLog.publishSyncThenBlockOnCompletion(FSHLog.java:1326) org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1460) org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:7792) org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3154) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2802) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2744) org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:696) org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:658) org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2033) org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32594) org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 72 (PriorityRpcServer.handler=3,queue=1,port=16020): State: TIMED_WAITING Blocked count: 1882731 Waited count: 2176163 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:167) org.apache.hadoop.hbase.regionserver.wal.FSHLog.blockOnSync(FSHLog.java:1332) org.apache.hadoop.hbase.regionserver.wal.FSHLog.publishSyncThenBlockOnCompletion(FSHLog.java:1326) org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1460) org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:7792) org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3154) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2802) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2744) org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:696) org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:658) org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2033) org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32594) org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 71 (PriorityRpcServer.handler=2,queue=0,port=16020): State: TIMED_WAITING Blocked count: 1882378 Waited count: 2175995 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:167) org.apache.hadoop.hbase.regionserver.wal.FSHLog.blockOnSync(FSHLog.java:1332) org.apache.hadoop.hbase.regionserver.wal.FSHLog.publishSyncThenBlockOnCompletion(FSHLog.java:1326) org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1460) org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:7792) org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3154) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2802) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2744) org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:696) org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:658) org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2033) org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32594) org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 70 (PriorityRpcServer.handler=1,queue=1,port=16020): State: TIMED_WAITING Blocked count: 1883584 Waited count: 2177341 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:167) org.apache.hadoop.hbase.regionserver.wal.FSHLog.blockOnSync(FSHLog.java:1332) org.apache.hadoop.hbase.regionserver.wal.FSHLog.publishSyncThenBlockOnCompletion(FSHLog.java:1326) org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1460) org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:7792) org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3154) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2802) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2744) org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:696) org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:658) org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2033) org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32594) org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 69 (PriorityRpcServer.handler=0,queue=0,port=16020): State: TIMED_WAITING Blocked count: 1881204 Waited count: 2174749 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:167) org.apache.hadoop.hbase.regionserver.wal.FSHLog.blockOnSync(FSHLog.java:1332) org.apache.hadoop.hbase.regionserver.wal.FSHLog.publishSyncThenBlockOnCompletion(FSHLog.java:1326) org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1460) org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:7792) org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3154) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2802) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2744) org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:696) org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:658) org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2033) org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32594) org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 68 (RW.default.scanRpcServer.handler=29,queue=20,port=16020): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1831b417 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.BoundedPriorityBlockingQueue.take(BoundedPriorityBlockingQueue.java:214) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 67 (RW.default.readRpcServer.handler=28,queue=17,port=16020): State: WAITING Blocked count: 415 Waited count: 420 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@34e39ea8 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.BoundedPriorityBlockingQueue.take(BoundedPriorityBlockingQueue.java:214) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 66 (RW.default.readRpcServer.handler=27,queue=16,port=16020): State: WAITING Blocked count: 1877 Waited count: 1888 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2b065d69 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.BoundedPriorityBlockingQueue.take(BoundedPriorityBlockingQueue.java:214) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 65 (RW.default.readRpcServer.handler=26,queue=15,port=16020): State: WAITING Blocked count: 1196 Waited count: 1204 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@251eb2eb Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.BoundedPriorityBlockingQueue.take(BoundedPriorityBlockingQueue.java:214) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 64 (RW.default.readRpcServer.handler=25,queue=19,port=16020): State: WAITING Blocked count: 2093 Waited count: 2098 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@301a2b53 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.BoundedPriorityBlockingQueue.take(BoundedPriorityBlockingQueue.java:214) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 63 (RW.default.readRpcServer.handler=24,queue=18,port=16020): State: WAITING Blocked count: 1857 Waited count: 1868 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1824f526 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.BoundedPriorityBlockingQueue.take(BoundedPriorityBlockingQueue.java:214) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 62 (RW.default.readRpcServer.handler=23,queue=17,port=16020): State: WAITING Blocked count: 811 Waited count: 817 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@34e39ea8 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.BoundedPriorityBlockingQueue.take(BoundedPriorityBlockingQueue.java:214) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 61 (RW.default.readRpcServer.handler=22,queue=16,port=16020): State: WAITING Blocked count: 1967 Waited count: 1981 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2b065d69 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.BoundedPriorityBlockingQueue.take(BoundedPriorityBlockingQueue.java:214) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 60 (RW.default.readRpcServer.handler=21,queue=15,port=16020): State: WAITING Blocked count: 1115 Waited count: 1122 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@251eb2eb Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.BoundedPriorityBlockingQueue.take(BoundedPriorityBlockingQueue.java:214) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 59 (RW.default.writeRpcServer.handler=20,queue=5,port=16020): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1501e5e Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 58 (RW.default.writeRpcServer.handler=19,queue=4,port=16020): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@51097d75 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 57 (RW.default.writeRpcServer.handler=18,queue=3,port=16020): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@302894b4 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 56 (RW.default.writeRpcServer.handler=17,queue=2,port=16020): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f1aca52 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 55 (RW.default.writeRpcServer.handler=16,queue=1,port=16020): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@e398701 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 54 (RW.default.writeRpcServer.handler=15,queue=0,port=16020): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1441a166 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 53 (RW.default.writeRpcServer.handler=14,queue=14,port=16020): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@60381a6e Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 52 (RW.default.writeRpcServer.handler=13,queue=13,port=16020): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@39a91461 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 51 (RW.default.writeRpcServer.handler=12,queue=12,port=16020): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7bea9dda Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 50 (RW.default.writeRpcServer.handler=11,queue=11,port=16020): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@62b94950 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 49 (RW.default.writeRpcServer.handler=10,queue=10,port=16020): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4272c403 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 48 (RW.default.writeRpcServer.handler=9,queue=9,port=16020): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3309da30 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 47 (RW.default.writeRpcServer.handler=8,queue=8,port=16020): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1c2295c6 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 46 (RW.default.writeRpcServer.handler=7,queue=7,port=16020): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3479acdb Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 45 (RW.default.writeRpcServer.handler=6,queue=6,port=16020): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@60ff348b Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 44 (RW.default.writeRpcServer.handler=5,queue=5,port=16020): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1501e5e Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 43 (RW.default.writeRpcServer.handler=4,queue=4,port=16020): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@51097d75 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 42 (RW.default.writeRpcServer.handler=3,queue=3,port=16020): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@302894b4 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 41 (RW.default.writeRpcServer.handler=2,queue=2,port=16020): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f1aca52 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 40 (RW.default.writeRpcServer.handler=1,queue=1,port=16020): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@e398701 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 39 (RW.default.writeRpcServer.handler=0,queue=0,port=16020): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1441a166 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 23 (RpcServer.listener,port=16020): State: BLOCKED Blocked count: 2274 Waited count: 0 Blocked on org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader@313474c9 Blocked by 31 (RpcServer.reader=7,bindAddress=0.0.0.0,port=16020) Stack: org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.registerChannel(RpcServer.java:668) org.apache.hadoop.hbase.ipc.RpcServer$Listener.doAccept(RpcServer.java:832) org.apache.hadoop.hbase.ipc.RpcServer$Listener.run(RpcServer.java:741) Thread 35 (RpcServer.responder): State: RUNNABLE Blocked count: 76 Waited count: 92 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.hbase.ipc.RpcServer$Responder.doRunLoop(RpcServer.java:972) org.apache.hadoop.hbase.ipc.RpcServer$Responder.run(RpcServer.java:919) Thread 38 (main-EventThread): State: WAITING Blocked count: 833 Waited count: 976 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1d5abafe Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) Thread 37 (main-SendThread(hbasectrl024.ash1.facebook.com:2181)): State: RUNNABLE Blocked count: 46 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) Thread 34 (Timer for 'HBase' metrics system): State: TIMED_WAITING Blocked count: 1 Waited count: 7935 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 33 (RpcServer.reader=9,bindAddress=0.0.0.0,port=16020): State: WAITING Blocked count: 270 Waited count: 4172 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73c14be6 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:350) org.apache.hadoop.hbase.ipc.BalancedQueueRpcExecutor.dispatch(BalancedQueueRpcExecutor.java:77) org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.dispatch(SimpleRpcScheduler.java:197) org.apache.hadoop.hbase.ipc.RpcServer$Connection.processRequest(RpcServer.java:1853) org.apache.hadoop.hbase.ipc.RpcServer$Connection.processOneRpc(RpcServer.java:1753) org.apache.hadoop.hbase.ipc.RpcServer$Connection.process(RpcServer.java:1612) org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1592) org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:856) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:641) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:617) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 32 (RpcServer.reader=8,bindAddress=0.0.0.0,port=16020): State: WAITING Blocked count: 280 Waited count: 2972 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73c14be6 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:350) org.apache.hadoop.hbase.ipc.BalancedQueueRpcExecutor.dispatch(BalancedQueueRpcExecutor.java:77) org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.dispatch(SimpleRpcScheduler.java:197) org.apache.hadoop.hbase.ipc.RpcServer$Connection.processRequest(RpcServer.java:1853) org.apache.hadoop.hbase.ipc.RpcServer$Connection.processOneRpc(RpcServer.java:1753) org.apache.hadoop.hbase.ipc.RpcServer$Connection.process(RpcServer.java:1612) org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1592) org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:856) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:641) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:617) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 31 (RpcServer.reader=7,bindAddress=0.0.0.0,port=16020): State: WAITING Blocked count: 283 Waited count: 3316 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73c14be6 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:350) org.apache.hadoop.hbase.ipc.BalancedQueueRpcExecutor.dispatch(BalancedQueueRpcExecutor.java:77) org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.dispatch(SimpleRpcScheduler.java:197) org.apache.hadoop.hbase.ipc.RpcServer$Connection.processRequest(RpcServer.java:1853) org.apache.hadoop.hbase.ipc.RpcServer$Connection.processOneRpc(RpcServer.java:1753) org.apache.hadoop.hbase.ipc.RpcServer$Connection.process(RpcServer.java:1612) org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1592) org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:856) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:641) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:617) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 30 (RpcServer.reader=6,bindAddress=0.0.0.0,port=16020): State: WAITING Blocked count: 268 Waited count: 3272 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73c14be6 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:350) org.apache.hadoop.hbase.ipc.BalancedQueueRpcExecutor.dispatch(BalancedQueueRpcExecutor.java:77) org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.dispatch(SimpleRpcScheduler.java:197) org.apache.hadoop.hbase.ipc.RpcServer$Connection.processRequest(RpcServer.java:1853) org.apache.hadoop.hbase.ipc.RpcServer$Connection.processOneRpc(RpcServer.java:1753) org.apache.hadoop.hbase.ipc.RpcServer$Connection.process(RpcServer.java:1612) org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1592) org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:856) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:641) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:617) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 29 (RpcServer.reader=5,bindAddress=0.0.0.0,port=16020): State: WAITING Blocked count: 301 Waited count: 4432 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2ace6293 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:350) org.apache.hadoop.hbase.ipc.BalancedQueueRpcExecutor.dispatch(BalancedQueueRpcExecutor.java:77) org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.dispatch(SimpleRpcScheduler.java:197) org.apache.hadoop.hbase.ipc.RpcServer$Connection.processRequest(RpcServer.java:1853) org.apache.hadoop.hbase.ipc.RpcServer$Connection.processOneRpc(RpcServer.java:1753) org.apache.hadoop.hbase.ipc.RpcServer$Connection.process(RpcServer.java:1612) org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1592) org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:856) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:641) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:617) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 28 (RpcServer.reader=4,bindAddress=0.0.0.0,port=16020): State: WAITING Blocked count: 293 Waited count: 4332 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73c14be6 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:350) org.apache.hadoop.hbase.ipc.BalancedQueueRpcExecutor.dispatch(BalancedQueueRpcExecutor.java:77) org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.dispatch(SimpleRpcScheduler.java:197) org.apache.hadoop.hbase.ipc.RpcServer$Connection.processRequest(RpcServer.java:1853) org.apache.hadoop.hbase.ipc.RpcServer$Connection.processOneRpc(RpcServer.java:1753) org.apache.hadoop.hbase.ipc.RpcServer$Connection.process(RpcServer.java:1612) org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1592) org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:856) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:641) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:617) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 27 (RpcServer.reader=3,bindAddress=0.0.0.0,port=16020): State: WAITING Blocked count: 286 Waited count: 4368 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73c14be6 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:350) org.apache.hadoop.hbase.ipc.BalancedQueueRpcExecutor.dispatch(BalancedQueueRpcExecutor.java:77) org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.dispatch(SimpleRpcScheduler.java:197) org.apache.hadoop.hbase.ipc.RpcServer$Connection.processRequest(RpcServer.java:1853) org.apache.hadoop.hbase.ipc.RpcServer$Connection.processOneRpc(RpcServer.java:1753) org.apache.hadoop.hbase.ipc.RpcServer$Connection.process(RpcServer.java:1612) org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1592) org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:856) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:641) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:617) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 26 (RpcServer.reader=2,bindAddress=0.0.0.0,port=16020): State: WAITING Blocked count: 290 Waited count: 3784 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73c14be6 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:350) org.apache.hadoop.hbase.ipc.BalancedQueueRpcExecutor.dispatch(BalancedQueueRpcExecutor.java:77) org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.dispatch(SimpleRpcScheduler.java:197) org.apache.hadoop.hbase.ipc.RpcServer$Connection.processRequest(RpcServer.java:1853) org.apache.hadoop.hbase.ipc.RpcServer$Connection.processOneRpc(RpcServer.java:1753) org.apache.hadoop.hbase.ipc.RpcServer$Connection.process(RpcServer.java:1612) org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1592) org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:856) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:641) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:617) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 25 (RpcServer.reader=1,bindAddress=0.0.0.0,port=16020): State: WAITING Blocked count: 279 Waited count: 3634 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73c14be6 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:350) org.apache.hadoop.hbase.ipc.BalancedQueueRpcExecutor.dispatch(BalancedQueueRpcExecutor.java:77) org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.dispatch(SimpleRpcScheduler.java:197) org.apache.hadoop.hbase.ipc.RpcServer$Connection.processRequest(RpcServer.java:1853) org.apache.hadoop.hbase.ipc.RpcServer$Connection.processOneRpc(RpcServer.java:1753) org.apache.hadoop.hbase.ipc.RpcServer$Connection.process(RpcServer.java:1612) org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1592) org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:856) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:641) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:617) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 24 (RpcServer.reader=0,bindAddress=0.0.0.0,port=16020): State: WAITING Blocked count: 309 Waited count: 4792 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73c14be6 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:350) org.apache.hadoop.hbase.ipc.BalancedQueueRpcExecutor.dispatch(BalancedQueueRpcExecutor.java:77) org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.dispatch(SimpleRpcScheduler.java:197) org.apache.hadoop.hbase.ipc.RpcServer$Connection.processRequest(RpcServer.java:1853) org.apache.hadoop.hbase.ipc.RpcServer$Connection.processOneRpc(RpcServer.java:1753) org.apache.hadoop.hbase.ipc.RpcServer$Connection.process(RpcServer.java:1612) org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1592) org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:856) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:641) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:617) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 5 (Signal Dispatcher): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: Thread 3 (Finalizer): State: WAITING Blocked count: 10641 Waited count: 10449 Waiting on java.lang.ref.ReferenceQueue$Lock@1ca3b418 Stack: java.lang.Object.wait(Native Method) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:142) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:158) java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209) Thread 2 (Reference Handler): State: WAITING Blocked count: 6663 Waited count: 6585 Waiting on java.lang.ref.Reference$Lock@58cbafc2 Stack: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157) Thread 1 (main): State: WAITING Blocked count: 7 Waited count: 6 Waiting on java.lang.Thread@15de9036 Stack: java.lang.Object.wait(Native Method) java.lang.Thread.join(Thread.java:1245) java.lang.Thread.join(Thread.java:1319) org.apache.hadoop.hbase.util.HasThread.join(HasThread.java:89) org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:66) org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:87) org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126) org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2650) Stacks: =========================================================== RS Configuration: =========================================================== dfs.journalnode.rpc-address0.0.0.0:8485hdfs-default.xml io.storefile.bloom.block.size131072hbase-default.xml dfs.client.failover.proxy.provider.hbaseci001org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProviderhdfs-site.xml yarn.ipc.rpc.classorg.apache.hadoop.yarn.ipc.HadoopYarnProtoRPCyarn-default.xml mapreduce.job.maxtaskfailures.per.tracker3mapred-default.xml yarn.client.max-cached-nodemanagers-proxies0yarn-default.xml ha.zookeeper.parent-node/hadoop-hacore-site.xml hbase.rest.threads.min2hbase-default.xml hbase.rs.cacheblocksonwritefalsehbase-default.xml ha.health-monitor.connect-retry-interval.ms1000core-default.xml hbase.regionserver.global.memstore.size.min.range0.1hbase-site.xml yarn.resourcemanager.work-preserving-recovery.enabledfalseyarn-default.xml dfs.client.mmap.cache.size256hdfs-default.xml mapreduce.reduce.markreset.buffer.percent0.0mapred-default.xml dfs.datanode.data.dirfile:///mnt/d0/HBASECI001-DFS/,file:///mnt/d1/HBASECI001-DFS/,file:///mnt/d2/HBASECI001-DFS/,file:///mnt/d3/HBASECI001-DFS/,file:///mnt/d4/HBASECI001-DFS/,file:///mnt/d5/HBASECI001-DFS/,file:///mnt/d6/HBASECI001-DFS/,file:///mnt/d7/HBASECI001-DFS/,file:///mnt/d8/HBASECI001-DFS/,file:///mnt/d9/HBASECI001-DFS/,file:///mnt/d10/HBASECI001-DFS/,file:///mnt/d11/HBASECI001-DFS/hdfs-site.xml mapreduce.jobhistory.max-age-ms604800000mapred-default.xml dfs.namenode.lazypersist.file.scrub.interval.sec300hdfs-default.xml mapreduce.job.ubertask.enablefalsemapred-default.xml dfs.namenode.delegation.token.renew-interval86400000hdfs-default.xml yarn.nodemanager.log-aggregation.compression-typenoneyarn-default.xml dfs.namenode.replication.considerLoadtruehdfs-default.xml mapreduce.job.complete.cancel.delegation.tokenstruemapred-default.xml mapreduce.jobhistory.datestring.cache.size200000mapred-default.xml hadoop.security.kms.client.authentication.retry-count1core-default.xml hadoop.ssl.enabled.protocolsTLSv1core-default.xml hbase.status.multicast.address.ip226.1.1.3hbase-default.xml dfs.namenode.retrycache.heap.percent0.03fhdfs-default.xml yarn.resourcemanager.scheduler.address${yarn.resourcemanager.hostname}:8030yarn-default.xml dfs.namenode.logging.levelinfohdfs-default.xml dfs.client.file-block-storage-locations.num-threads10hdfs-default.xml dfs.datanode.balance.bandwidthPerSec1048576hdfs-default.xml yarn.resourcemanager.proxy-user-privileges.enabledfalseyarn-default.xml mapreduce.reduce.shuffle.fetch.retry.enabled${yarn.nodemanager.recovery.enabled}mapred-default.xml io.mapfile.bloom.error.rate0.005core-default.xml yarn.nodemanager.resourcemanager.minimum.versionNONEyarn-default.xml yarn.resourcemanager.nodemanagers.heartbeat-interval-ms1000yarn-default.xml dfs.secondary.namenode.kerberos.internal.spnego.principal${dfs.web.authentication.kerberos.principal}hdfs-default.xml yarn.nodemanager.delete.debug-delay-sec0yarn-default.xml dfs.client.read.shortcircuit.streams.cache.size256hdfs-default.xml dfs.image.transfer.bandwidthPerSec0hdfs-default.xml yarn.scheduler.maximum-allocation-vcores32yarn-default.xml hfile.block.bloom.cacheonwritefalsehbase-default.xml hbase.zookeeper.quorumhbasectrl022.ash1.facebook.com,hbasectrl026.ash1.facebook.com,hbasectrl023.ash1.facebook.com,hbasectrl024.ash1.facebook.com,hbasectrl025.ash1.facebook.comhbase-site.xml hbase.http.staticuser.userdr.stackhbase-default.xml yarn.timeline-service.address${yarn.timeline-service.hostname}:10200yarn-default.xml yarn.nodemanager.disk-health-checker.min-free-space-per-disk-mb0yarn-default.xml mapreduce.job.hdfs-servers${fs.defaultFS}yarn-default.xml mapreduce.task.profile.reduce.params${mapreduce.task.profile.params}mapred-default.xml hbase.zookeeper.property.syncLimit5hbase-default.xml dfs.namenode.fs-limits.min-block-size1048576hdfs-default.xml ftp.stream-buffer-size4096core-default.xml dfs.client.use.legacy.blockreader.localfalsehdfs-default.xml dfs.short.circuit.shared.memory.watcher.interrupt.check.ms60000hdfs-default.xml dfs.datanode.directoryscan.threads1hdfs-default.xml fs.s3a.buffer.dir${hadoop.tmp.dir}/s3acore-default.xml yarn.client.application-client-protocol.poll-interval-ms200yarn-default.xml yarn.timeline-service.leveldb-timeline-store.path/hay/yarn/timelineyarn-site.xml mapreduce.job.split.metainfo.maxsize10000000mapred-default.xml dfs.namenode.edits.noeditlogchannelflushfalsehdfs-default.xml s3native.bytes-per-checksum512core-default.xml hbase.rest.filter.classesorg.apache.hadoop.hbase.rest.filter.GzipFilterhbase-default.xml yarn.client.failover-retries-on-socket-timeouts0yarn-default.xml dfs.namenode.startup.delay.block.deletion.sec0hdfs-default.xml dfs.webhdfs.user.provider.user.pattern^[A-Za-z_][A-Za-z0-9._-]*[$]?$hdfs-default.xml mapreduce.tasktracker.tasks.sleeptimebeforesigkill5000mapred-default.xml yarn.timeline-service.client.retry-interval-ms1000yarn-default.xml dfs.encrypt.data.transfer.cipher.key.bitlength128hdfs-default.xml yarn.resourcemanager.webapp.https.address.rm4hbasectrl025.ash1.facebook.com:8090yarn-site.xml yarn.resourcemanager.webapp.https.address.rm3hbasectrl024.ash1.facebook.com:8090yarn-site.xml yarn.resourcemanager.webapp.https.address.rm2hbasectrl023.ash1.facebook.com:8090yarn-site.xml yarn.resourcemanager.webapp.https.address.rm1hbasectrl026.ash1.facebook.com:8090yarn-site.xml yarn.resourcemanager.webapp.https.address.rm0hbasectrl022.ash1.facebook.com:8090yarn-site.xml hadoop.http.authentication.typesimplecore-default.xml dfs.namenode.path.based.cache.refresh.interval.ms30000hdfs-default.xml mapreduce.local.clientfactory.class.nameorg.apache.hadoop.mapred.LocalClientFactorymapred-default.xml dfs.datanode.cache.revocation.timeout.ms900000hdfs-default.xml ipc.client.connection.maxidletime10000core-default.xml dfs.namenode.safemode.threshold-pct0.999fhdfs-default.xml hfile.block.cache.size0.6hbase-site.xml fs.s3a.multipart.purge.age86400core-default.xml dfs.namenode.num.checkpoints.retained2hdfs-default.xml hbase.hregion.memstore.mslab.enabledtruehbase-default.xml yarn.timeline-service.bind-host0.0.0.0yarn-site.xml mapreduce.job.ubertask.maxmaps9mapred-default.xml dfs.namenode.stale.datanode.interval30000hdfs-default.xml yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage90.0yarn-default.xml mapreduce.tasktracker.http.address0.0.0.0:50060mapred-default.xml mapreduce.ifile.readahead.bytes4194304mapred-default.xml mapreduce.jobhistory.admin.addresshbasectrl025.ash1.facebook.com:10033mapred-site.xml s3.client-write-packet-size65536core-default.xml hbase.master.port16000hbase-default.xml dfs.block.access.token.lifetime600hdfs-default.xml yarn.app.mapreduce.am.resource.cpu-vcores1mapred-default.xml mapreduce.input.lineinputformat.linespermap1mapred-default.xml hbase.regionserver.checksum.verifytruehbase-default.xml dfs.namenode.num.extra.edits.retained1000000hdfs-default.xml hbase.security.visibility.mutations.checkauthsfalsehbase-default.xml mapreduce.reduce.shuffle.input.buffer.percent0.70mapred-default.xml hadoop.http.staticuser.userdr.whocore-default.xml mapreduce.reduce.maxattempts4mapred-default.xml hadoop.security.group.mapping.ldap.search.filter.user(&(objectClass=user)(sAMAccountName={0}))core-default.xml mapreduce.jobhistory.admin.acl*mapred-default.xml dfs.client.contextdefaulthdfs-default.xml mapreduce.map.maxattempts4mapred-default.xml yarn.resourcemanager.zk-retry-interval-ms1000yarn-default.xml mapreduce.jobhistory.cleaner.interval-ms86400000mapred-default.xml dfs.datanode.drop.cache.behind.readsfalsehdfs-default.xml hbase.server.versionfile.writeattempts3hbase-default.xml dfs.permissions.superusergrouproothdfs-site.xml hbase.zookeeper.useMultiTruehbase-site.xml fs.s3n.block.size67108864core-default.xml hadoop.registry.system.aclssasl:yarn@, sasl:mapred@, sasl:mapred@hdfs@yarn-default.xml dfs.namenode.list.cache.pools.num.responses100hdfs-default.xml yarn.resourcemanager.resource-tracker.address.rm4hbasectrl025.ash1.facebook.com:8031yarn-site.xml yarn.resourcemanager.resource-tracker.address.rm3hbasectrl024.ash1.facebook.com:8031yarn-site.xml yarn.resourcemanager.resource-tracker.address.rm2hbasectrl023.ash1.facebook.com:8031yarn-site.xml yarn.resourcemanager.resource-tracker.address.rm1hbasectrl026.ash1.facebook.com:8031yarn-site.xml hbase.zookeeper.leaderport3888hbase-default.xml yarn.resourcemanager.resource-tracker.address.rm0hbasectrl022.ash1.facebook.com:8031yarn-site.xml dfs.datanode.slow.io.warning.threshold.ms300hdfs-default.xml hbase.master.info.port16010programatically yarn.nodemanager.bind-host0.0.0.0yarn-site.xml dfs.namenode.fs-limits.max-blocks-per-file1048576hdfs-default.xml yarn.nodemanager.vmem-check-enabledtrueyarn-default.xml hadoop.security.authenticationsimplecore-default.xml mapreduce.reduce.cpu.vcores1mapred-default.xml net.topology.node.switch.mapping.implorg.apache.hadoop.net.ScriptBasedMappingcore-default.xml fs.s3.sleepTimeSeconds10core-default.xml yarn.timeline-service.ttl-ms604800000yarn-default.xml yarn.resourcemanager.keytab/etc/krb5.keytabyarn-default.xml yarn.resourcemanager.container.liveness-monitor.interval-ms600000yarn-default.xml mapreduce.jobtracker.heartbeats.in.second100mapred-default.xml yarn.app.mapreduce.am.scheduler.heartbeat.interval-ms1000mapred-default.xml yarn.app.mapreduce.client-am.ipc.max-retries-on-timeouts3mapred-default.xml yarn.nodemanager.linux-container-executor.cgroups.hierarchy/hadoop-yarnyarn-default.xml s3.bytes-per-checksum512core-default.xml hbase.regionserver.dns.nameserverdefaulthbase-default.xml hadoop.ssl.require.client.certfalsecore-default.xml yarn.resourcemanager.ha.rm-idsrm0,rm1,rm2,rm3,rm4yarn-site.xml dfs.journalnode.http-address0.0.0.0:8480hdfs-default.xml mapreduce.output.fileoutputformat.compressfalsemapred-default.xml dfs.ha.automatic-failover.enabledTruehdfs-site.xml hbase.ipc.server.callqueue.read.ratio0.3hbase-site.xml hbase.cluster.distributedTruehbase-site.xml hbase.rootdirhdfs://hbaseci001/HBASECI001-HBASE/programatically yarn.resourcemanager.webapp.delegation-token-auth-filter.enabledtrueyarn-default.xml mapreduce.shuffle.max.threads0mapred-default.xml dfs.namenode.invalidate.work.pct.per.iteration0.32fhdfs-default.xml s3native.client-write-packet-size65536core-default.xml dfs.client.block.write.replace-datanode-on-failure.policyDEFAULThdfs-default.xml mapreduce.client.submit.file.replication10mapred-default.xml yarn.app.mapreduce.am.job.committer.commit-window10000mapred-default.xml yarn.nodemanager.sleep-delay-before-sigkill.ms250yarn-default.xml yarn.nodemanager.env-whitelistJAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,HADOOP_YARN_HOMEyarn-default.xml dfs.namenode.acls.enabledfalsehdfs-default.xml dfs.namenode.secondary.http-address0.0.0.0:50090hdfs-default.xml mapreduce.map.speculativeFalsemapred-site.xml mapreduce.job.speculative.slowtaskthreshold1.0mapred-default.xml mapreduce.task.tmp.dir./tmpmapred-default.xml yarn.nodemanager.linux-container-executor.cgroups.mountfalseyarn-default.xml hbase.auth.token.max.lifetime604800000hbase-default.xml hbase.regionserver.msginterval3000hbase-default.xml mapreduce.tasktracker.http.threads40mapred-default.xml mapreduce.jobhistory.http.policyHTTP_ONLYmapred-default.xml hbase.ipc.client.fallback-to-simple-auth-allowedfalsehbase-default.xml fs.s3a.paging.maximum5000core-default.xml hbase.rest.threads.max100hbase-default.xml fs.s3.buffer.dir${hadoop.tmp.dir}/s3core-default.xml hbase.snapshot.enabledtruehbase-default.xml hbase.dynamic.jars.dir${hbase.rootdir}/libhbase-default.xml hbase.defaults.for.version1.3.0-SNAPSHOThbase-default.xml io.native.lib.availabletruecore-default.xml mapreduce.jobhistory.done-dir${yarn.app.mapreduce.am.staging-dir}/history/donemapred-default.xml hbase.regions.slop0.2hbase-default.xml hadoop.registry.zk.retry.interval.ms1000yarn-default.xml dfs.namenode.avoid.write.stale.datanodeTruehdfs-site.xml dfs.namenode.checkpoint.txns1000000hdfs-default.xml hadoop.ssl.hostname.verifierDEFAULTcore-default.xml zookeeper.znode.rootserverroot-region-serverhbase-default.xml mapreduce.task.timeout600000mapred-default.xml hbase.client.max.perserver.tasks5hbase-default.xml yarn.nodemanager.disk-health-checker.interval-ms120000yarn-default.xml dfs.journalnode.https-address0.0.0.0:8481hdfs-default.xml hadoop.security.groups.cache.secs300core-default.xml mapreduce.input.fileinputformat.split.minsize0mapred-default.xml dfs.datanode.sync.behind.writesTruehdfs-site.xml zookeeper.session.timeout90000hbase-default.xml ipc.server.tcpnodelayTruecore-site.xml mapreduce.shuffle.port13562mapred-default.xml hadoop.rpc.protectionauthenticationcore-default.xml dfs.client.https.keystore.resourcessl-client.xmlhdfs-default.xml dfs.namenode.list.encryption.zones.num.responses100hdfs-default.xml yarn.client.failover-proxy-providerorg.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvideryarn-site.xml mapreduce.jobtracker.retiredjobs.cache.size1000mapred-default.xml hbase.balancer.period300000hbase-default.xml ipc.client.tcpnodelayTruecore-site.xml dfs.ha.tail-edits.period60hdfs-default.xml dfs.datanode.drop.cache.behind.writesfalsehdfs-default.xml fs.s3.maxRetries4core-default.xml mapreduce.jobtracker.addresslocalmapred-default.xml hadoop.http.authentication.kerberos.principalHTTP/_HOST@LOCALHOSTcore-default.xml nfs.server.port2049hdfs-default.xml yarn.resourcemanager.webapp.address${yarn.resourcemanager.hostname}:8088yarn-default.xml mapreduce.task.profile.reduces0-2mapred-default.xml yarn.timeline-service.client.max-retries30yarn-default.xml yarn.resourcemanager.am.max-attempts2yarn-default.xml hbase.normalizer.enabledfalsehbase-default.xml hbase.hstore.blockingWaitTime90000hbase-default.xml nfs.dump.dir/tmp/.hdfs-nfshdfs-default.xml hbase.client.pause100hbase-default.xml hbase.client.write.buffer2097152hbase-default.xml dfs.bytes-per-checksum512hdfs-default.xml mapreduce.job.end-notification.max.retry.interval5000mapred-default.xml ipc.client.connect.retry.interval1000core-default.xml fs.s3a.multipart.size104857600core-default.xml yarn.app.mapreduce.am.command-opts-server -Xmx3g -XX:+UseG1GC -XX:MaxGCPauseMillis=150yarn-site.xml yarn.nodemanager.process-kill-wait.ms2000yarn-default.xml hbase.rpc.timeout60000hbase-default.xml hbase.metrics.exposeOperationTimestruehbase-default.xml dfs.namenode.safemode.min.datanodes0hdfs-default.xml hbase.thrift.maxWorkerThreads1000hbase-default.xml dfs.namenode.write.stale.datanode.ratio0.5fhdfs-default.xml hadoop.jetty.logs.serve.aliasestruecore-default.xml mapreduce.reduce.shuffle.fetch.retry.timeout-ms30000mapred-default.xml fs.du.interval600000core-default.xml mapreduce.tasktracker.dns.nameserverdefaultmapred-default.xml hbase.master.catalog.timeout600000hbase-default.xml hbase.regionserver.global.memstore.size.max.range0.7hbase-site.xml hfile.block.cache.size.max.range0.7hbase-site.xml hadoop.security.random.device.file.path/dev/urandomcore-default.xml mapreduce.task.merge.progress.records10000mapred-default.xml hbase.region.replica.replication.enabledfalsehbase-default.xml dfs.webhdfs.enabledtruehdfs-default.xml hadoop.registry.securefalseyarn-default.xml hadoop.ssl.client.confssl-client.xmlcore-default.xml mapreduce.job.counters.max120mapred-default.xml yarn.nodemanager.localizer.fetch.thread-count4yarn-default.xml io.mapfile.bloom.size1048576core-default.xml yarn.nodemanager.localizer.client.thread-count5yarn-default.xml fs.automatic.closetruecore-default.xml mapreduce.task.profilefalsemapred-default.xml dfs.namenode.edit.log.autoroll.multiplier.threshold2.0hdfs-default.xml mapreduce.task.combine.progress.records10000mapred-default.xml mapreduce.shuffle.ssl.file.buffer.size65536mapred-default.xml fs.swift.implorg.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemcore-default.xml yarn.app.mapreduce.am.container.log.backups0mapred-default.xml hbase.hstore.bytes.per.checksum16384hbase-default.xml dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction0.75fhdfs-default.xml hbase.hstore.flusher.count2hbase-default.xml dfs.namenode.backup.address0.0.0.0:50100hdfs-default.xml dfs.client.https.need-authfalsehdfs-default.xml mapreduce.app-submission.cross-platformfalsemapred-default.xml yarn.timeline-service.ttl-enabletrueyarn-default.xml dfs.user.home.dir.prefix/userhdfs-default.xml yarn.nodemanager.container-monitor.procfs-tree.smaps-based-rss.enabledfalseyarn-default.xml yarn.nodemanager.keytab/etc/krb5.keytabyarn-default.xml dfs.namenode.xattrs.enabledtruehdfs-default.xml dfs.client.write.exclude.nodes.cache.expiry.interval.millis600000hdfs-default.xml mapreduce.jobtracker.restart.recoverfalsemapred-default.xml mapreduce.map.skip.proc.count.autoincrtruemapred-default.xml dfs.namenode.datanode.registration.ip-hostname-checktruehdfs-default.xml dfs.image.transfer.chunksize65536hdfs-default.xml hadoop.security.instrumentation.requires.adminfalsecore-default.xml io.compression.codec.bzip2.librarysystem-nativecore-default.xml dfs.namenode.name.dir.restoreTruehdfs-site.xml hbase.client.retries.number350programatically dfs.namenode.resource.checked.volumes.minimum1hdfs-default.xml hadoop.ssl.keystores.factory.classorg.apache.hadoop.security.ssl.FileBasedKeyStoresFactorycore-default.xml dfs.namenode.list.cache.directives.num.responses100hdfs-default.xml hbase.status.multicast.address.port16100hbase-default.xml fs.ftp.host0.0.0.0core-default.xml hbase.hstore.checksum.algorithmCRC32Chbase-default.xml s3.blocksize67108864core-default.xml s3native.stream-buffer-size4096core-default.xml mapreduce.jobtracker.taskschedulerorg.apache.hadoop.mapred.JobQueueTaskSchedulermapred-default.xml dfs.datanode.dns.nameserverdefaulthdfs-default.xml yarn.nodemanager.resource.memory-mb8588yarn-site.xml mapreduce.task.userlog.limit.kb0mapred-default.xml hadoop.security.crypto.codec.classes.aes.ctr.nopaddingorg.apache.hadoop.crypto.OpensslAesCtrCryptoCodec,org.apache.hadoop.crypto.JceAesCtrCryptoCodeccore-default.xml mapreduce.reduce.speculativeFalsemapred-site.xml yarn.resourcemanager.admin.address.rm4hbasectrl025.ash1.facebook.com:8033yarn-site.xml yarn.nodemanager.container-monitor.interval-ms3000yarn-default.xml yarn.resourcemanager.admin.address.rm3hbasectrl024.ash1.facebook.com:8033yarn-site.xml yarn.resourcemanager.admin.address.rm2hbasectrl023.ash1.facebook.com:8033yarn-site.xml yarn.resourcemanager.admin.address.rm1hbasectrl026.ash1.facebook.com:8033yarn-site.xml yarn.resourcemanager.admin.address.rm0hbasectrl022.ash1.facebook.com:8033yarn-site.xml net.topology.script.file.name/opt/hbase/bin/locationforcore-site.xml dfs.replication.max512hdfs-default.xml dfs.replication3hdfs-default.xml dfs.ha.fencing.methodsshell(/bin/true)core-site.xml yarn.client.failover-retries0yarn-default.xml yarn.nodemanager.resource.cpu-vcores8yarn-default.xml mapreduce.jobhistory.recovery.enableTruemapred-site.xml hbase.server.thread.wakefrequency10000hbase-default.xml nfs.exports.allowed.hosts* rwcore-default.xml hbase.lease.recovery.timeout900000hbase-default.xml hbase.coordinated.state.manager.classorg.apache.hadoop.hbase.coordination.ZkCoordinatedStateManagerhbase-default.xml mapreduce.reduce.shuffle.memory.limit.percent0.25mapred-default.xml file.replication3core-site.xml mapreduce.job.reduce.shuffle.consumer.plugin.classorg.apache.hadoop.mapreduce.task.reduce.Shufflemapred-default.xml hfile.format.version3hbase-default.xml mapreduce.job.jvm.numtasks1mapred-default.xml dfs.datanode.fsdatasetcache.max.threads.per.volume4hdfs-default.xml mapreduce.am.max-attempts2mapred-default.xml mapreduce.shuffle.connection-keep-alive.timeout5mapred-default.xml hadoop.fuse.timer.period5hdfs-default.xml mapreduce.job.reduces50mapred-site.xml hbase.thrift.minWorkerThreads16hbase-default.xml hbase.zookeeper.dns.interfacedefaulthbase-default.xml yarn.app.mapreduce.am.job.task.listener.thread-count30mapred-default.xml yarn.resourcemanager.store.classorg.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStoreyarn-site.xml mapreduce.job.speculative.slownodethreshold1.0mapred-default.xml s3native.replication3core-default.xml mapreduce.tasktracker.reduce.tasks.maximum2mapred-default.xml hbase.snapshot.restore.failsafe.namehbase-failsafe-{snapshot.name}-{restore.timestamp}hbase-default.xml fs.permissions.umask-mode022core-default.xml mapreduce.cluster.local.dir${hadoop.tmp.dir}/mapred/localmapred-default.xml mapreduce.client.output.filterFAILEDmapred-default.xml yarn.nodemanager.pmem-check-enabledtrueyarn-default.xml dfs.client.failover.connection.retries.on.timeouts0hdfs-default.xml mapreduce.jobtracker.instrumentationorg.apache.hadoop.mapred.JobTrackerMetricsInstmapred-default.xml ftp.replication3core-default.xml hbase.hstore.blockingStoreFiles25hbase-site.xml hadoop.security.group.mapping.ldap.search.attr.membermembercore-default.xml hbase.regionserver.hlog.reader.implorg.apache.hadoop.hbase.regionserver.wal.ProtobufLogReaderhbase-default.xml dfs.namenode.replication.work.multiplier.per.iteration2hdfs-default.xml yarn.resourcemanager.resource-tracker.address${yarn.resourcemanager.hostname}:8031yarn-default.xml mapreduce.tasktracker.outofband.heartbeatfalsemapred-default.xml hbase.master.info.bindAddress0.0.0.0hbase-default.xml dfs.namenode.edits.dir/hay/dfs/hdfs-site.xml yarn.resourcemanager.scheduler.monitor.enablefalseyarn-default.xml fs.trash.checkpoint.interval0core-default.xml hadoop.registry.zk.retry.times5yarn-default.xml dfs.client.read.shortcircuit.streams.cache.expiry.ms300000hdfs-default.xml yarn.timeline-service.leveldb-timeline-store.start-time-write-cache-size10000yarn-default.xml s3.stream-buffer-size4096core-default.xml fs.s3a.connection.maximum15core-default.xml file.client-write-packet-size65536core-default.xml mapreduce.tasktracker.healthchecker.script.timeout600000mapred-default.xml hbase.status.listener.classorg.apache.hadoop.hbase.client.ClusterStatusListener$MulticastListenerhbase-default.xml dfs.namenode.fs-limits.max-directory-items1048576hdfs-default.xml mapreduce.tasktracker.taskcontrollerorg.apache.hadoop.mapred.DefaultTaskControllermapred-default.xml dfs.namenode.path.based.cache.block.map.allocation.percent0.25hdfs-default.xml fs.s3a.implorg.apache.hadoop.fs.s3a.S3AFileSystemcore-default.xml dfs.namenode.checkpoint.dirfile://${hadoop.tmp.dir}/dfs/namesecondaryhdfs-default.xml hbase.regionserver.dns.interfacedefaulthbase-default.xml yarn.nodemanager.remote-app-log-dir${fs.defaultFS}/yarn-logsyarn-site.xml mapreduce.reduce.shuffle.retry-delay.max.ms60000mapred-default.xml io.map.index.interval128core-default.xml dfs.namenode.replication.interval3hdfs-default.xml dfs.client.block.write.replace-datanode-on-failure.enabletruehdfs-default.xml hbase.rest.port8080hbase-default.xml hbase.regionserver.handler.count30hbase-default.xml hadoop.ssl.server.confssl-server.xmlcore-default.xml dfs.nameserviceshbaseci001hdfs-site.xml hadoop.rpc.socket.factory.class.defaultorg.apache.hadoop.net.StandardSocketFactorycore-default.xml yarn.app.mapreduce.client.max-retries3mapred-default.xml hbase.wal.regiongrouping.numgroups3hbase-site.xml yarn.nodemanager.address${yarn.nodemanager.hostname}:8140yarn-site.xml hbase.ipc.server.callqueue.scan.ratio0.2hbase-site.xml dfs.datanode.max.transfer.threads4096hdfs-default.xml ha.failover-controller.graceful-fence.rpc-timeout.ms5000core-default.xml dfs.datanode.ipc.address0.0.0.0:50020hdfs-default.xml yarn.resourcemanager.delayed.delegation-token.removal-interval-ms30000yarn-default.xml yarn.timeline-service.enabledfalseyarn-default.xml dfs.client.cached.conn.retry3hdfs-default.xml dfs.namenode.backup.http-address0.0.0.0:50105hdfs-default.xml mapreduce.tasktracker.report.address127.0.0.1:0mapred-default.xml hbase.bulkload.retries.number10hbase-default.xml dfs.namenode.checkpoint.period3600hdfs-default.xml mapreduce.task.attempt.idhb_rs_hbase500.ash1.facebook.com,16020,1435179105838programatically hbase.hregion.max.filesize53687091200hbase-site.xml dfs.datanode.shared.file.descriptor.paths/dev/shm,/tmphdfs-default.xml hbase.master.loadbalancer.classorg.apache.hadoop.hbase.master.balancer.StochasticLoadBalancerhbase-default.xml dfs.http.policyHTTP_ONLYhdfs-default.xml hadoop.security.groups.cache.warn.after.ms5000core-default.xml dfs.namenode.fs-limits.max-xattrs-per-inode32hdfs-default.xml yarn.resourcemanager.zk-aclworld:anyone:rwcdayarn-default.xml mapreduce.job.speculative.speculativecap0.1mapred-default.xml dfs.namenode.support.allow.formatFalsehdfs-site.xml dfs.namenode.checkpoint.max-retries3hdfs-default.xml zookeeper.znode.acl.parentaclhbase-default.xml hbase.status.publisher.classorg.apache.hadoop.hbase.master.ClusterStatusPublisher$MulticastPublisherhbase-default.xml yarn.resourcemanager.fs.state-store.retry-policy-spec2000, 500yarn-default.xml hbase.tmp.dir${java.io.tmpdir}/hbase-${user.name}hbase-default.xml dfs.namenode.decommission.nodes.per.interval5hdfs-default.xml hbase.wal.providermultiwalhbase-site.xml mapreduce.job.committer.setup.cleanup.neededtruemapred-default.xml dfs.datanode.cache.revocation.polling.ms500hdfs-default.xml mapreduce.job.end-notification.retry.attempts0mapred-default.xml yarn.resourcemanager.state-store.max-completed-applications${yarn.resourcemanager.max-completed-applications}yarn-default.xml hbase.http.max.threads10hbase-default.xml mapreduce.map.output.compressTruemapred-site.xml hbase.client.localityCheck.threadPoolSize2hbase-default.xml mapreduce.jobhistory.cleaner.enabletruemapred-default.xml io.seqfile.local.dir${hadoop.tmp.dir}/io/localcore-default.xml dfs.blockreport.split.threshold1000000hdfs-default.xml mapreduce.reduce.shuffle.read.timeout180000mapred-default.xml mapreduce.job.queuenamedefaultmapred-default.xml ipc.client.connect.max.retries10core-default.xml io.seqfile.lazydecompresstruecore-default.xml yarn.app.mapreduce.am.staging-dir/tmp/hadoop-yarn/stagingmapred-default.xml yarn.nodemanager.linux-container-executor.resources-handler.classorg.apache.hadoop.yarn.server.nodemanager.util.DefaultLCEResourcesHandleryarn-default.xml yarn.timeline-service.leveldb-timeline-store.read-cache-size104857600yarn-default.xml hbase.master.distributed.log.replaytruehbase-default.xml io.file.buffer.size65536core-site.xml yarn.resourcemanager.am-rm-tokens.master-key-rolling-interval-secs86400yarn-default.xml ha.zookeeper.parent-znode/hadoop-hacore-default.xml mapreduce.tasktracker.indexcache.mb10mapred-default.xml tfile.io.chunk.size1048576core-default.xml yarn.resourcemanager.work-preserving-recovery.scheduling-wait-ms10000yarn-default.xml yarn.timeline-service.keytab/etc/krb5.keytabyarn-default.xml yarn.acl.enablefalseyarn-default.xml hbase.regionserver.regionSplitLimit1000hbase-default.xml hadoop.security.group.mapping.ldap.directory.search.timeout10000core-default.xml mapreduce.job.token.tracking.ids.enabledfalsemapred-default.xml hbase.thrift.maxQueuedRequests1000hbase-default.xml mapreduce.map.output.compress.codecorg.apache.hadoop.io.compress.Lz4Codecmapred-site.xml s3.replication3core-default.xml hadoop.registry.zk.root/registryyarn-default.xml tfile.fs.input.buffer.size262144core-default.xml yarn.timeline-service.http-authentication.typesimpleyarn-default.xml ha.failover-controller.graceful-fence.connection.retries1core-default.xml hbase.master.balancer.stochastic.localityCost0hbase-site.xml net.topology.script.number.args100core-default.xml fs.s3n.multipart.uploads.block.size67108864core-default.xml hfile.block.index.cacheonwritefalsehbase-default.xml yarn.nodemanager.recovery.dir/mnt/d3/HBASECI001-YARN/recoveryyarn-site.xml hadoop.ssl.enabledfalsecore-default.xml yarn.timeline-service.handler-thread-count10yarn-default.xml hbase.config.read.zookeeper.configfalsehbase-default.xml hbase.column.max.version1hbase-default.xml dfs.client.read.shortcircuit.buffer.size131072programatically dfs.namenode.reject-unresolved-dn-topology-mappingfalsehdfs-default.xml mapreduce.jobhistory.recovery.store.classorg.apache.hadoop.mapreduce.v2.hs.HistoryServerFileSystemStateStoreServicemapred-default.xml yarn.nodemanager.log.retain-seconds10800yarn-default.xml yarn.resourcemanager.admin.address${yarn.resourcemanager.hostname}:8033yarn-default.xml yarn.resourcemanager.recovery.enabledfalseyarn-default.xml dfs.client.slow.io.warning.threshold.ms30000hdfs-default.xml yarn.resourcemanager.ha.automatic-failover.zk-base-path/yarn-leader-electionyarn-default.xml fs.AbstractFileSystem.viewfs.implorg.apache.hadoop.fs.viewfs.ViewFscore-default.xml mapreduce.tasktracker.dns.interfacedefaultmapred-default.xml mapreduce.jobtracker.handler.count10mapred-default.xml dfs.blockreport.initialDelay0hdfs-default.xml fs.AbstractFileSystem.hdfs.implorg.apache.hadoop.fs.Hdfscore-default.xml dfs.namenode.retrycache.expirytime.millis600000hdfs-default.xml dfs.client.failover.sleep.max.millis15000hdfs-default.xml hbase.regionserver.thread.compaction.large3hbase-site.xml hbase.zookeeper.property.clientPort2181hbase-default.xml yarn.resourcemanager.max-completed-applications10000yarn-default.xml yarn.nodemanager.log-dirs/mnt/d0/HBASECI001-YARN/logs,/mnt/d1/HBASECI001-YARN/logs,/mnt/d2/HBASECI001-YARN/logs,/mnt/d3/HBASECI001-YARN/logs,/mnt/d4/HBASECI001-YARN/logs,/mnt/d5/HBASECI001-YARN/logs,/mnt/d6/HBASECI001-YARN/logs,/mnt/d7/HBASECI001-YARN/logs,/mnt/d8/HBASECI001-YARN/logs,/mnt/d9/HBASECI001-YARN/logs,/mnt/d10/HBASECI001-YARN/logs,/mnt/d11/HBASECI001-YARN/logsyarn-site.xml dfs.client.failover.sleep.base.millis500hdfs-default.xml yarn.nodemanager.linux-container-executor.nonsecure-mode.user-pattern^[_.A-Za-z0-9][-@_.A-Za-z0-9]{0,255}?[$]?$yarn-default.xml hbase.rest.readonlyfalsehbase-default.xml dfs.default.chunk.view.size32768hdfs-default.xml dfs.client.read.shortcircuitfalsehdfs-default.xml ftp.blocksize67108864core-default.xml mapreduce.job.acl-modify-job mapred-default.xml zookeeper.znode.parent/hbasehbase-default.xml fs.defaultFShdfs://hbaseci001programatically hbase.rpc.shortoperation.timeout10000hbase-default.xml hadoop.http.filter.initializersorg.apache.hadoop.http.lib.StaticUserWebFiltercore-default.xml fs.s3n.multipart.copy.block.size5368709120core-default.xml mapreduce.map.java.opts-server -Xmx3g -XX:+UseG1GC -XX:MaxGCPauseMillis=500mapred-site.xml yarn.resourcemanager.connect.max-wait.ms900000yarn-default.xml dfs.datanode.synconcloseTruehdfs-site.xml hadoop.security.group.mapping.ldap.sslfalsecore-default.xml dfs.namenode.max.extra.edits.segments.retained10000hdfs-default.xml dfs.namenode.https-address0.0.0.0:50470hdfs-default.xml yarn.nodemanager.aux-servicesmapreduce_shuffleyarn-site.xml yarn.resourcemanager.bind-host0.0.0.0yarn-site.xml yarn.resourcemanager.admin.client.thread-count1yarn-default.xml hadoop.security.kms.client.encrypted.key.cache.size500core-default.xml ipc.client.kill.max10core-default.xml hadoop.security.group.mapping.ldap.search.filter.group(objectClass=group)core-default.xml fs.AbstractFileSystem.file.implorg.apache.hadoop.fs.local.LocalFscore-default.xml hadoop.http.authentication.kerberos.keytab${user.home}/hadoop.keytabcore-default.xml yarn.client.nodemanager-connect.max-wait-ms900000yarn-default.xml mapreduce.job.map.output.collector.classorg.apache.hadoop.mapred.MapTask$MapOutputBuffermapred-default.xml dfs.namenode.path.based.cache.retry.interval.ms30000hdfs-default.xml yarn.log.dir/logsyarn-site.xml hadoop.security.uid.cache.secs14400core-default.xml mapreduce.map.cpu.vcores1mapred-default.xml yarn.log-aggregation.retain-check-interval-seconds-1yarn-default.xml mapreduce.map.log.levelINFOmapred-default.xml mapred.child.java.opts-Xmx200mmapred-default.xml hfile.index.block.max.size131072hbase-default.xml hbase.client.scanner.timeout.period60000hbase-default.xml hadoop.registry.zk.session.timeout.ms60000yarn-default.xml yarn.nodemanager.local-cache.max-files-per-directory8192yarn-default.xml dfs.https.server.keystore.resourcessl-server.xmlhdfs-default.xml mapreduce.jobtracker.taskcache.levels2mapred-default.xml hbase.regionserver.global.memstore.size0.2hbase-site.xml dfs.datanode.handler.count10hdfs-default.xml s3native.blocksize67108864core-default.xml dfs.datanode.fsdataset.volume.choosing.policyorg.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicyhdfs-site.xml mapreduce.client.completion.pollinterval5000mapred-default.xml hbase.hstore.compactionThreshold3hbase-default.xml dfs.stream-buffer-size4096hdfs-default.xml dfs.namenode.delegation.key.update-interval86400000hdfs-default.xml mapreduce.job.maps100mapred-site.xml hbase.master.logcleaner.ttl600000hbase-default.xml mapreduce.job.acl-view-job mapred-default.xml dfs.namenode.enable.retrycachetruehdfs-default.xml yarn.resourcemanager.connect.retry-interval.ms1000yarn-site.xml dfs.namenode.shared.edits.dirqjournal://hbasectrl022.ash1.facebook.com:8485;hbasectrl026.ash1.facebook.com:8485;hbasectrl023.ash1.facebook.com:8485;hbasectrl024.ash1.facebook.com:8485;hbasectrl025.ash1.facebook.com:8485/hbaseci001hdfs-site.xml yarn.timeline-service.leveldb-timeline-store.ttl-interval-ms300000yarn-default.xml fs.s3a.multipart.threshold2147483647core-default.xml dfs.namenode.decommission.interval30hdfs-default.xml hbase.rootdir.perms700hbase-default.xml hbase.hregion.majorcompaction604800000hbase-default.xml mapreduce.shuffle.max.connections0mapred-default.xml yarn.log-aggregation-enableTrueyarn-site.xml dfs.client-write-packet-size65536hdfs-default.xml dfs.client.file-block-storage-locations.timeout.millis1000hdfs-default.xml mapreduce.jobtracker.expire.trackers.interval600000mapred-default.xml dfs.client.block.write.retries3hdfs-default.xml mapreduce.task.io.sort.factor10mapred-default.xml hbase.hregion.memstore.flush.size134217728hbase-default.xml ha.health-monitor.sleep-after-disconnect.ms1000core-default.xml ha.zookeeper.session-timeout.ms5000core-default.xml yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-userstrueyarn-default.xml dfs.support.appendtruehdfs-default.xml mapreduce.input.fileinputformat.list-status.num-threads1mapred-default.xml io.skip.checksum.errorsfalsecore-default.xml hbase.ipc.client.tcpnodelaytruehbase-default.xml hbase.regionserver.optionalcacheflushinterval3600000hbase-default.xml yarn.resourcemanager.scheduler.client.thread-count50yarn-default.xml dfs.namenode.safemode.extension10000hdfs-site.xml hbase.normalizer.period1800000hbase-default.xml mapreduce.jobhistory.move.thread-count3mapred-default.xml yarn.resourcemanager.zk-state-store.parent-path/rmstoreyarn-default.xml ipc.client.idlethreshold4000core-default.xml hbase.regionserver.port16020hbase-site.xml dfs.namenode.accesstime.precision3600000hdfs-default.xml mapreduce.task.profile.params-agentlib:hprof=cpu=samples,heap=sites,force=n,thread=y,verbose=n,file=%smapred-default.xml hbase.hstore.time.to.purge.deletes0hbase-default.xml hbase.regionserver.logroll.errors.tolerated2hbase-default.xml mapreduce.jobhistory.keytab/etc/security/keytab/jhs.service.keytabmapred-default.xml hbase.hstore.compaction.max10hbase-default.xml dfs.datanode.hdfs-blocks-metadata.enabledfalsehdfs-default.xml yarn.scheduler.minimum-allocation-mb4294yarn-site.xml yarn.resourcemanager.container-tokens.master-key-rolling-interval-secs86400yarn-default.xml hbase.master.infoserver.redirecttruehbase-default.xml mapreduce.reduce.shuffle.fetch.retry.interval-ms1000mapred-default.xml hadoop.user.group.static.mapping.overridesdr.who=;core-default.xml hadoop.security.kms.client.encrypted.key.cache.low-watermark0.3fcore-default.xml hbase.hregion.percolumnfamilyflush.size.lower.bound16777216hbase-default.xml fs.s3a.connection.ssl.enabledtruecore-default.xml dfs.datanode.directoryscan.interval21600hdfs-default.xml hbase.zookeeper.property.initLimit10hbase-default.xml yarn.resourcemanager.scheduler.monitor.policiesorg.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicyyarn-default.xml ipc.server.listen.queue.size128core-default.xml rpc.metrics.quantile.enablefalsecore-default.xml yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds-1yarn-default.xml mapreduce.jobtracker.persist.jobstatus.dir/jobtracker/jobsInfomapred-default.xml yarn.client.nodemanager-client-async.thread-pool-max-size500yarn-default.xml hadoop.security.group.mappingorg.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallbackcore-default.xml yarn.resourcemanager.system-metrics-publisher.enabledfalseyarn-default.xml dfs.namenode.name.dir/hay/dfs/hdfs-site.xml hbase.coprocessor.abortonerrortruehbase-default.xml yarn.am.liveness-monitor.expiry-interval-ms600000yarn-default.xml yarn.nm.liveness-monitor.expiry-interval-ms600000yarn-default.xml hbase.hstore.compaction.kv.max10hbase-default.xml hbase.hregion.preclose.flush.size5242880hbase-default.xml ftp.bytes-per-checksum512core-default.xml dfs.namenode.max.objects0hdfs-default.xml ha.zookeeper.quorumhbasectrl022.ash1.facebook.com:2181,hbasectrl026.ash1.facebook.com:2181,hbasectrl023.ash1.facebook.com:2181,hbasectrl024.ash1.facebook.com:2181,hbasectrl025.ash1.facebook.com:2181core-site.xml hbase.master.hfilecleaner.pluginsorg.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleanerhbase-default.xml mapreduce.job.emit-timeline-datafalsemapred-default.xml hbase.regionserver.thread.compaction.small3hbase-site.xml hbase.metrics.showTableNametruehbase-default.xml mapreduce.map.memory.mb5120mapred-site.xml yarn.client.nodemanager-connect.retry-interval-ms10000yarn-default.xml dfs.namenode.edits.journal-plugin.qjournalorg.apache.hadoop.hdfs.qjournal.client.QuorumJournalManagerhdfs-default.xml mapreduce.tasktracker.healthchecker.interval60000mapred-default.xml nfs.wtmax1048576hdfs-default.xml yarn.timeline-service.leveldb-timeline-store.start-time-read-cache-size10000yarn-default.xml hbase.lease.recovery.dfs.timeout64000hbase-default.xml hadoop.registry.zk.connection.timeout.ms15000yarn-default.xml yarn.resourcemanager.address${yarn.resourcemanager.hostname}:8032yarn-default.xml dfs.cachereport.intervalMsec10000hdfs-default.xml mapreduce.task.skip.start.attempts2mapred-default.xml yarn.resourcemanager.zk-timeout-ms10000yarn-default.xml dfs.namenode.checkpoint.edits.dir${dfs.namenode.checkpoint.dir}hdfs-default.xml hbase.server.scanner.max.result.size104857600hbase-default.xml hadoop.hdfs.configuration.version1hdfs-default.xml mapreduce.map.skip.maxrecords0mapred-default.xml yarn.resourcemanager.system-metrics-publisher.dispatcher.pool-size10yarn-default.xml dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold10737418240hdfs-default.xml nfs.allow.insecure.portstruehdfs-default.xml mapreduce.jobtracker.system.dir${hadoop.tmp.dir}/mapred/systemmapred-default.xml yarn.timeline-service.hostnamehbasectrl025.ash1.facebook.comyarn-site.xml hadoop.registry.rm.enabledfalseyarn-default.xml mapreduce.job.reducer.preempt.delay.sec0mapred-default.xml hbase.zookeeper.dns.nameserverdefaulthbase-default.xml hbase.ipc.server.callqueue.handler.factor0.7hbase-site.xml mapreduce.shuffle.ssl.enabledfalsemapred-default.xml yarn.nodemanager.vmem-pmem-ratio2.1yarn-default.xml yarn.nodemanager.container-manager.thread-count20yarn-default.xml dfs.encrypt.data.transferfalsehdfs-default.xml dfs.block.access.key.update.interval600hdfs-default.xml yarn.resourcemanager.cluster-idhbaseci001yarn-site.xml hbase.bulkload.staging.dir${hbase.fs.tmp.dir}hbase-default.xml hadoop.tmp.dir/haycore-site.xml dfs.namenode.audit.loggersdefaulthdfs-default.xml fs.AbstractFileSystem.har.implorg.apache.hadoop.fs.HarFscore-default.xml yarn.nodemanager.localizer.cache.target-size-mb10240yarn-default.xml yarn.http.policyHTTP_ONLYyarn-default.xml hbase.regionserver.logroll.period3600000hbase-default.xml dfs.client.short.circuit.replica.stale.threshold.ms1800000hdfs-default.xml yarn.timeline-service.webapp.https.address${yarn.timeline-service.hostname}:8190yarn-default.xml mapreduce.jobtracker.persist.jobstatus.hours1mapred-default.xml tfile.fs.output.buffer.size262144core-default.xml hbase.hregion.memstore.block.multiplier4hbase-default.xml dfs.namenode.checkpoint.check.period60hdfs-default.xml dfs.datanode.dns.interfacedefaulthdfs-default.xml fs.ftp.host.port21core-default.xml mapreduce.task.io.sort.mb1024mapred-site.xml dfs.namenode.inotify.max.events.per.rpc1000hdfs-default.xml hadoop.security.group.mapping.ldap.search.attr.group.namecncore-default.xml dfs.namenode.avoid.read.stale.datanodeTruehdfs-site.xml mapreduce.output.fileoutputformat.compress.typeRECORDmapred-default.xml hbase.storescanner.parallel.seek.enablefalsehbase-default.xml hbase.regionserver.storefile.refresh.period0hbase-default.xml mapreduce.reduce.skip.proc.count.autoincrtruemapred-default.xml hbase.dfs.client.read.shortcircuit.buffer.size131072hbase-default.xml file.bytes-per-checksum512core-default.xml mapreduce.job.userlog.retain.hours24mapred-default.xml dfs.datanode.http.address0.0.0.0:50075hdfs-default.xml dfs.image.compressfalsehdfs-default.xml mapreduce.reduce.java.opts-server -Xmx3g -XX:+UseG1GC -XX:MaxGCPauseMillis=500mapred-site.xml dfs.journalnode.edits.dir/hay/journalhdfs-site.xml yarn.resourcemanager.zk-addresshbasectrl022.ash1.facebook.com,hbasectrl026.ash1.facebook.com,hbasectrl023.ash1.facebook.com,hbasectrl024.ash1.facebook.com,hbasectrl025.ash1.facebook.comyarn-site.xml ha.health-monitor.check-interval.ms1000core-default.xml dfs.permissions.enabledtruehdfs-default.xml hbase.thrift.htablepool.size.max1000hbase-default.xml yarn.resourcemanager.resource-tracker.client.thread-count50yarn-default.xml dfs.client.domain.socket.data.trafficfalsehdfs-default.xml dfs.image.compression.codecorg.apache.hadoop.io.compress.DefaultCodechdfs-default.xml dfs.datanode.address0.0.0.0:50010hdfs-default.xml dfs.block.access.token.enablefalsehdfs-default.xml mapreduce.reduce.input.buffer.percent0.0mapred-default.xml hbase.client.scanner.caching2147483647hbase-default.xml yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usagefalseyarn-default.xml mapreduce.tasktracker.local.dir.minspacestart0mapred-default.xml dfs.blockreport.intervalMsec21600000hdfs-default.xml hbase.snapshot.restore.take.failsafe.snapshottruehbase-default.xml ha.health-monitor.rpc-timeout.ms45000core-default.xml dfs.client.failover.connection.retries0hdfs-default.xml dfs.namenode.kerberos.internal.spnego.principal${dfs.web.authentication.kerberos.principal}hdfs-default.xml hadoop.policy.filehbase-policy.xmlhbase-default.xml hbase.fs.tmp.dir/user/${user.name}/hbase-staginghbase-default.xml yarn.scheduler.maximum-allocation-mb8588yarn-site.xml mapreduce.task.files.preserve.failedtasksfalsemapred-default.xml hbase.status.publishedfalsehbase-default.xml yarn.nodemanager.delete.thread-count4yarn-default.xml dfs.https.enablefalsehdfs-default.xml mapreduce.output.fileoutputformat.compress.codecorg.apache.hadoop.io.compress.DefaultCodecmapred-default.xml dfs.namenode.rpc-address.hbaseci001.nn1hbasectrl026.ash1.facebook.com:9000hdfs-site.xml dfs.namenode.rpc-address.hbaseci001.nn0hbasectrl022.ash1.facebook.com:9000hdfs-site.xml map.sort.classorg.apache.hadoop.util.QuickSortmapred-default.xml mapreduce.job.classloaderfalsemapred-default.xml hadoop.registry.zk.retry.ceiling.ms60000yarn-default.xml mapreduce.jobtracker.tasktracker.maxblacklists4mapred-default.xml io.seqfile.compress.blocksize1000000core-default.xml dfs.blocksize268435456hdfs-site.xml hbase.client.scanner.max.result.size2097152hbase-default.xml mapreduce.task.profile.maps0-2mapred-default.xml mapreduce.jobtracker.staging.root.dir${hadoop.tmp.dir}/mapred/stagingmapred-default.xml yarn.nodemanager.localizer.cache.cleanup.interval-ms600000yarn-default.xml mapreduce.jobtracker.http.address0.0.0.0:50030mapred-default.xml hbase.regionserver.info.bindAddress0.0.0.0hbase-default.xml dfs.client.mmap.cache.timeout.ms3600000hdfs-default.xml yarn.timeline-service.generic-application-history.fs-history-store.uri/hay/yarn/timeline/generic-historyyarn-site.xml hadoop.security.java.secure.random.algorithmSHA1PRNGcore-default.xml fs.client.resolve.remote.symlinkstruecore-default.xml hbase.master.logcleaner.pluginsorg.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleanerhbase-default.xml hbase.master.info.port.orig16010programatically hbase.data.umask.enablefalsehbase-default.xml hbase.coprocessor.user.enabledtruehbase-default.xml mapreduce.tasktracker.local.dir.minspacekill0mapred-default.xml nfs.mountd.port4242hdfs-default.xml yarn.nodemanager.disk-health-checker.min-healthy-disks0.25yarn-default.xml mapreduce.tasktracker.taskmemorymanager.monitoringinterval5000mapred-default.xml hbase.local.dir${hbase.tmp.dir}/local/hbase-default.xml dfs.namenode.resource.du.reserved104857600hdfs-default.xml mapreduce.job.end-notification.retry.interval1000mapred-default.xml mapreduce.jobhistory.loadedjobs.cache.size5mapred-default.xml dfs.client.datanode-restart.timeout30hdfs-default.xml yarn.nodemanager.local-dirs/mnt/d0/HBASECI001-YARN/nm-local-dir,/mnt/d1/HBASECI001-YARN/nm-local-dir,/mnt/d2/HBASECI001-YARN/nm-local-dir,/mnt/d3/HBASECI001-YARN/nm-local-dir,/mnt/d4/HBASECI001-YARN/nm-local-dir,/mnt/d5/HBASECI001-YARN/nm-local-dir,/mnt/d6/HBASECI001-YARN/nm-local-dir,/mnt/d7/HBASECI001-YARN/nm-local-dir,/mnt/d8/HBASECI001-YARN/nm-local-dir,/mnt/d9/HBASECI001-YARN/nm-local-dir,/mnt/d10/HBASECI001-YARN/nm-local-dir,/mnt/d11/HBASECI001-YARN/nm-local-diryarn-site.xml hbase.table.lock.enabletruehbase-default.xml dfs.datanode.block.id.layout.upgrade.threads12hdfs-default.xml hbase.storescanner.parallel.seek.threads10hbase-default.xml hadoop.registry.jaas.contextClientyarn-default.xml yarn.timeline-service.webapp.address${yarn.timeline-service.hostname}:8188yarn-default.xml hbase.online.schema.update.enabletruehbase-default.xml mapreduce.jobhistory.addresshbasectrl025.ash1.facebook.com:10020mapred-site.xml mapreduce.jobtracker.persist.jobstatus.activetruemapred-default.xml file.blocksize67108864core-default.xml dfs.datanode.readahead.bytes4193404hdfs-default.xml hbase.zookeeper.property.dataDir/hay/zookeeperhbase-site.xml dfs.namenode.http-address0.0.0.0:50070hdfs-default.xml hadoop.work.around.non.threadsafe.getpwuidfalsecore-default.xml yarn.resourcemanager.configuration.provider-classorg.apache.hadoop.yarn.LocalConfigurationProvideryarn-default.xml yarn.nodemanager.recovery.enabledTrueyarn-site.xml yarn.resourcemanager.hostname0.0.0.0yarn-default.xml yarn.resourcemanager.webapp.address.rm4hbasectrl025.ash1.facebook.com:8088yarn-site.xml yarn.resourcemanager.webapp.address.rm3hbasectrl024.ash1.facebook.com:8088yarn-site.xml hbase.regionserver.handler.abort.on.error.percent0.5hbase-default.xml yarn.resourcemanager.webapp.address.rm2hbasectrl023.ash1.facebook.com:8088yarn-site.xml yarn.resourcemanager.webapp.address.rm1hbasectrl026.ash1.facebook.com:8088yarn-site.xml yarn.resourcemanager.webapp.address.rm0hbasectrl022.ash1.facebook.com:8088yarn-site.xml fs.s3n.multipart.uploads.enabledfalsecore-default.xml hbase.security.exec.permission.checksfalsehbase-default.xml hbase.master.normalizer.classorg.apache.hadoop.hbase.master.normalizer.SimpleRegionNormalizerhbase-default.xml dfs.namenode.fs-limits.max-component-length255hdfs-default.xml hbase.regionserver.info.port.autofalsehbase-default.xml ha.failover-controller.cli-check.rpc-timeout.ms20000core-default.xml hbase.auth.key.update.interval86400000hbase-default.xml ftp.client-write-packet-size65536core-default.xml mapreduce.reduce.shuffle.parallelcopies5mapred-default.xml mapreduce.jobhistory.principaljhs/_HOST@REALM.TLDmapred-default.xml hadoop.http.authentication.simple.anonymous.allowedtruecore-default.xml yarn.log-aggregation.retain-seconds604800yarn-site.xml hbase.regionserver.thrift.framedfalsehbase-default.xml hbase.zookeeper.property.maxClientCnxns300hbase-default.xml mapreduce.client.genericoptionsparser.usedtrueprogramatically yarn.timeline-service.http-authentication.simple.anonymous.allowedtrueyarn-default.xml dfs.namenode.secondary.https-address0.0.0.0:50091hdfs-default.xml mapreduce.job.ubertask.maxreduces1mapred-default.xml hbase.assignment.usezkFalsehbase-site.xml yarn.nodemanager.health-checker.interval-ms600000yarn-default.xml dfs.namenode.fs-limits.max-xattr-size16384hdfs-default.xml fs.s3a.multipart.purgefalsecore-default.xml hadoop.security.kms.client.encrypted.key.cache.num.refill.threads2core-default.xml hbase.server.compactchecker.interval.multiplier1000hbase-default.xml yarn.timeline-service.store-classorg.apache.hadoop.yarn.server.timeline.LeveldbTimelineStoreyarn-default.xml mapreduce.shuffle.transfer.buffer.size131072mapred-default.xml yarn.resourcemanager.zk-num-retries1000yarn-default.xml mapreduce.jobtracker.jobhistory.task.numberprogresssplits12mapred-default.xml yarn.nodemanager.webapp.address${yarn.nodemanager.hostname}:8042yarn-default.xml yarn.app.mapreduce.client-am.ipc.max-retries3mapred-default.xml ha.failover-controller.new-active.rpc-timeout.ms60000core-default.xml mapreduce.jobhistory.client.thread-count10mapred-default.xml fs.trash.interval0core-default.xml hbase.client.max.perregion.tasks1hbase-default.xml mapreduce.reduce.skip.maxgroups0mapred-default.xml hfile.block.cache.size.min.range0.1hbase-site.xml mapreduce.reduce.memory.mb4096mapred-site.xml yarn.nodemanager.health-checker.script.timeout-ms1200000yarn-default.xml dfs.datanode.du.reserved0hdfs-default.xml dfs.namenode.resource.check.interval5000hdfs-default.xml mapreduce.client.progressmonitor.pollinterval1000mapred-default.xml yarn.nodemanager.hostname0.0.0.0yarn-default.xml yarn.resourcemanager.ha.enabledTrueyarn-site.xml dfs.ha.log-roll.period120hdfs-default.xml yarn.scheduler.minimum-allocation-vcores1yarn-default.xml dfs.client.block.write.replace-datanode-on-failure.best-effortfalsehdfs-default.xml yarn.app.mapreduce.am.container.log.limit.kb0mapred-default.xml hadoop.http.authentication.signature.secret.file${user.home}/hadoop-http-auth-signature-secretcore-default.xml mapreduce.jobhistory.move.interval-ms180000mapred-default.xml dfs.ha.namenodes.hbaseci001nn0,nn1hdfs-site.xml yarn.nodemanager.container-executor.classorg.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutoryarn-default.xml hadoop.security.authorizationfalsecore-default.xml dfs.storage.policy.enabledtruehdfs-default.xml dfs.datanode.https.address0.0.0.0:50475hdfs-default.xml yarn.nodemanager.localizer.address${yarn.nodemanager.hostname}:8040yarn-default.xml mapreduce.jobhistory.recovery.store.fs.uri${fs.defaultFS}/mapred/history/recoverystoremapred-site.xml dfs.namenode.replication.min1hdfs-default.xml mapreduce.shuffle.connection-keep-alive.enablefalsemapred-default.xml hadoop.common.configuration.version0.23.0core-default.xml yarn.app.mapreduce.task.container.log.backups0mapred-default.xml hadoop.security.groups.negative-cache.secs30core-default.xml yarn.resourcemanager.hostname.rm4hbasectrl025.ash1.facebook.comyarn-site.xml yarn.resourcemanager.hostname.rm3hbasectrl024.ash1.facebook.comyarn-site.xml mapreduce.ifile.readaheadtruemapred-default.xml yarn.nodemanager.resource.percentage-physical-cpu-limit100yarn-default.xml yarn.resourcemanager.hostname.rm2hbasectrl023.ash1.facebook.comyarn-site.xml yarn.resourcemanager.hostname.rm1hbasectrl026.ash1.facebook.comyarn-site.xml mapreduce.job.max.split.locations10mapred-default.xml dfs.datanode.max.locked.memory0hdfs-default.xml yarn.resourcemanager.hostname.rm0hbasectrl022.ash1.facebook.comyarn-site.xml hadoop.registry.zk.quorumlocalhost:2181yarn-default.xml mapreduce.jobhistory.joblist.cache.size20000mapred-default.xml mapreduce.job.end-notification.max.attempts5mapred-default.xml dfs.image.transfer.timeout60000hdfs-default.xml dfs.client.read.shortcircuit.skip.checksumfalsehdfs-default.xml nfs.rtmax1048576hdfs-default.xml dfs.namenode.edit.log.autoroll.check.interval.ms300000hdfs-default.xml mapreduce.reduce.shuffle.connect.timeout180000mapred-default.xml dfs.datanode.failed.volumes.tolerated3hdfs-site.xml mapreduce.jobhistory.webapp.addresshbasectrl025.ash1.facebook.com:19888mapred-site.xml fs.s3a.connection.timeout5000core-default.xml dfs.client.mmap.retry.timeout.ms300000hdfs-default.xml dfs.datanode.data.dir.perm700hdfs-default.xml hadoop.http.authentication.token.validity36000core-default.xml ipc.client.connect.max.retries.on.timeouts45core-default.xml yarn.nodemanager.docker-container-executor.exec-name/usr/bin/dockeryarn-default.xml yarn.app.mapreduce.am.job.committer.cancel-timeout60000mapred-default.xml dfs.ha.fencing.ssh.connect-timeout30000core-default.xml hbase.data.umask000hbase-default.xml mapreduce.reduce.log.levelINFOmapred-default.xml mapreduce.reduce.shuffle.merge.percent0.66mapred-default.xml ipc.client.fallback-to-simple-auth-allowedfalsecore-default.xml io.serializationsorg.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerializationcore-default.xml hbase.regionserver.hlog.writer.implorg.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriterhbase-default.xml fs.s3.block.size67108864core-default.xml yarn.nodemanager.linux-container-executor.nonsecure-mode.local-usernobodyyarn-default.xml hadoop.kerberos.kinit.commandkinitcore-default.xml hadoop.security.kms.client.encrypted.key.cache.expiry43200000core-default.xml yarn.resourcemanager.fs.state-store.uri${hadoop.tmp.dir}/yarn/system/rmstoreyarn-default.xml hbase.regionserver.region.split.policyorg.apache.hadoop.hbase.regionserver.IncreasingToUpperBoundRegionSplitPolicyhbase-default.xml hbase.cells.scanned.per.heartbeat.check10000hbase-default.xml yarn.admin.acl*yarn-default.xml dfs.namenode.delegation.token.max-lifetime604800000hdfs-default.xml mapreduce.reduce.merge.inmem.threshold1000mapred-default.xml net.topology.implorg.apache.hadoop.net.NetworkTopologycore-default.xml yarn.resourcemanager.ha.automatic-failover.enabledtrueyarn-default.xml dfs.datanode.use.datanode.hostnamefalsehdfs-default.xml dfs.heartbeat.interval3hdfs-default.xml yarn.resourcemanager.scheduler.classorg.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduleryarn-default.xml io.map.index.skip0core-default.xml dfs.namenode.handler.count24hdfs-site.xml yarn.resourcemanager.webapp.https.address${yarn.resourcemanager.hostname}:8090yarn-default.xml yarn.nodemanager.admin-envMALLOC_ARENA_MAX=$MALLOC_ARENA_MAXyarn-default.xml hbase.client.max.total.tasks100hbase-default.xml yarn.resourcemanager.address.rm4hbasectrl025.ash1.facebook.com:8032yarn-site.xml yarn.resourcemanager.address.rm3hbasectrl024.ash1.facebook.com:8032yarn-site.xml hadoop.security.crypto.cipher.suiteAES/CTR/NoPaddingcore-default.xml yarn.resourcemanager.address.rm2hbasectrl023.ash1.facebook.com:8032yarn-site.xml mapreduce.task.profile.map.params${mapreduce.task.profile.params}mapred-default.xml mapreduce.jobtracker.jobhistory.block.size3145728mapred-default.xml yarn.resourcemanager.address.rm1hbasectrl026.ash1.facebook.com:8032yarn-site.xml yarn.resourcemanager.address.rm0hbasectrl022.ash1.facebook.com:8032yarn-site.xml hbase.zookeeper.peerport2888hbase-default.xml hadoop.security.crypto.buffer.size8192core-default.xml hbase.table.max.rowsize1073741824hbase-default.xml yarn.nodemanager.aux-services.mapreduce_shuffle.classorg.apache.hadoop.mapred.ShuffleHandleryarn-default.xml hbase.regionserver.ipc.address0.0.0.0hbase-site.xml hbase.master.executor.serverops.threads18hbase-site.xml mapreduce.cluster.acls.enabledfalsemapred-default.xml hbase.regionserver.info.port16030programatically hbase.coprocessor.enabledtruehbase-default.xml hbase.hregion.majorcompaction.jitter0.50hbase-default.xml fs.har.impl.disable.cachetruecore-default.xml mapreduce.tasktracker.map.tasks.maximum2mapred-default.xml ipc.client.connect.timeout20000core-default.xml yarn.timeline-service.generic-application-history.enabledTrueyarn-site.xml yarn.nodemanager.remote-app-log-dir-suffixlogsyarn-default.xml fs.df.interval60000core-default.xml hbase.regionserver.thrift.framed.max_frame_size_in_mb2hbase-default.xml hbase.http.filter.initializersorg.apache.hadoop.hbase.http.lib.StaticUserWebFilterhbase-default.xml hadoop.util.hash.typemurmurcore-default.xml mapreduce.jobhistory.minicluster.fixed.portsfalsemapred-default.xml mapreduce.jobtracker.jobhistory.lru.cache.size5mapred-default.xml dfs.client.failover.max.attempts15hdfs-default.xml dfs.client.use.datanode.hostnamefalsehdfs-default.xml ha.zookeeper.aclworld:anyone:rwcdacore-default.xml hbase.replication.rpc.codecorg.apache.hadoop.hbase.codec.KeyValueCodecWithTagshbase-default.xml mapreduce.jobtracker.maxtasks.perjob-1mapred-default.xml mapreduce.map.sort.spill.percent0.80mapred-default.xml file.stream-buffer-size4096core-default.xml hbase.meta.replicas.usefalseprogramatically yarn.resourcemanager.ha.automatic-failover.embeddedtrueyarn-default.xml hbase.regionserver.catalog.timeout600000hbase-default.xml hbase.security.authenticationsimplehbase-default.xml dfs.host.excludes/hay/excludeshdfs-site.xml yarn.resourcemanager.nodemanager.minimum.versionNONEyarn-default.xml hadoop.fuse.connection.timeout300hdfs-default.xml hbase.client.keyvalue.maxsize10485760hbase-default.xml hbase.regionserver.thrift.compactfalsehbase-default.xml mapreduce.tasktracker.instrumentationorg.apache.hadoop.mapred.TaskTrackerMetricsInstmapred-default.xml io.seqfile.sorter.recordlimit1000000core-default.xml yarn.app.mapreduce.am.resource.mb4294yarn-site.xml mapreduce.framework.nameyarnmapred-site.xml mapreduce.job.reduce.slowstart.completedmaps0.05mapred-default.xml yarn.resourcemanager.client.thread-count50yarn-default.xml mapreduce.cluster.temp.dir${hadoop.tmp.dir}/mapred/tempmapred-default.xml dfs.client.mmap.enabledtruehdfs-default.xml mapreduce.jobhistory.intermediate-done-dir${yarn.app.mapreduce.am.staging-dir}/history/done_intermediatemapred-default.xml hbase.defaults.for.version.skipfalsehbase-default.xml fs.s3a.attempts.maximum10core-default.xml hbase.rest.support.proxyuserfalsehbase-default.xml Logs =========================================================== RS Queue: =========================================================== Compaction/Split Queue summary: compaction_queue=(0:0), split_queue=0, merge_queue=0 Compaction/Split Queue dump: LargeCompation Queue: SmallCompation Queue: Split Queue: Region Merge Queue: Flush Queue summary: flush_queue=121 Flush Queue Queue dump: Flush Queue: [flush region IntegrationTestLoadAndVerify,{\x84{\x84{\x84z\x10,1435179555993.4b07b14b3f5d8c3b76dc88f9ef7dd74f.] [flush region IntegrationTestLoadAndVerify,)\xB1\xBB\xFA\xE0\x8D)4,1435179555993.e612b5d28e3bcf0cdb5c02ec3926c4c6.] [flush region IntegrationTestLoadAndVerify,0\x19\x0B\x86\xC2b/\x88,1435179555993.99203590f70b76d41a0b13eb0adcd620.] [flush region IntegrationTestLoadAndVerify,\xFF\x92\xB6n$$\xFC\x90,1435179555993.79f299500f1edd2a257a4dd07495b6b3.] [flush region IntegrationTestLoadAndVerify,\xCF\xB0O\xB0O\xB0M>,1435179555993.b544afe91a8bc3af53f494a6e0e41498.] [flush region IntegrationTestLoadAndVerify,}\xB9"Kko\xFC>,1435179555993.7b9682d5a892ab32d330c78c06114580.] [flush region IntegrationTestLoadAndVerify,\x84\x8D\xBBi) \x02\xFE,1435179555993.4b60ac4b51a8c5921f4818c19ed172de.] [flush region IntegrationTestLoadAndVerify,\x96\xB2r (\xFB\x94\xEC,1435179555993.4f0c728f968e16afc809878490cac4c9.] [flush region IntegrationTestLoadAndVerify,R\x9B\x1B\xBF\xAE\x08\xD1\xA2,1435179555993.8e2d075f94ce7699f416ec4ced9873cd.] [flush region IntegrationTestLoadAndVerify,\x93\xA385\x81Z\x11\xE6,1435179555993.9f8d0e01a40405b835bf6e5a22a86390.] [flush region IntegrationTestLoadAndVerify,m\xB6\xDBm\xB6\xDBll,1435179555993.12f2d9906b37df3d5404d9d01181655d.] [flush region IntegrationTestLoadAndVerify,\xEF\x90o\x90o\x90l\xBE,1435179555993.ce6440e2c6ce8d3bbbe6fa9b44fc35e3.] [flush region IntegrationTestLoadAndVerify,rD\x96\xD6\xDF\xFBp\xEC,1435179555993.65338019d46599c2b8cfbd9af4513d77.] [flush region IntegrationTestLoadAndVerify,\xBDyb\x0B\xAB0;>,1435179555993.0244ec5cf506fac9314a920402f8c8dd.] [flush region IntegrationTestLoadAndVerify,"\xA6~\x144\xEF\xA2>,1435179555993.4a40f98099b6ebe01e3b85bed10e4fb6.] [flush region IntegrationTestLoadAndVerify,YKG\x02"o\xD8>,1435179555993.7478960fc1dc9eefff0ca7f6abc387fc.] [flush region IntegrationTestLoadAndVerify,$\xDB$\xDB$\xDB$l,1435179555993.92d809bf48bdb959ed19a30cf7d9d257.] [flush region IntegrationTestLoadAndVerify,\x14\xFDK\xD8\xB9\x8F\x94\xBE,1435179555993.7ec0c641350f222bcc91d8c716c67340.] [flush region IntegrationTestLoadAndVerify,t\x9D\xABy\x19/\xF3>,1435179555993.f9ae44b012d8984bd747b80a3dec2c1f.] [flush region IntegrationTestLoadAndVerify,\xA8\xFB\x96\xB2r &\xFE,1435179555993.cfce718bcf9ef2046b3ac0a75837f6dc.] [flush region IntegrationTestLoadAndVerify,\xA6\xB4\xB8\xFD\xDD\x90$\xBE,1435179555993.be0dd87c6d0520eefb5691103fd14de2.] [flush region IntegrationTestLoadAndVerify,\xED[\xC8\xC9\x7F\xA4\xEA\x90,1435179555993.6303482ae32fec271310518bff96d8d8.] [flush region IntegrationTestLoadAndVerify,\xE6t\xF8\xBE\x1DPc\xBE,1435179555993.37678922d1c8f8ca5e7ac3578a885b9b.] [flush region IntegrationTestLoadAndVerify,\x9D\xABy\x19/\xF4\x9B\xD0,1435179555993.c86a77ff72bbf384cf2b8c41939ddb45.] [flush region IntegrationTestLoadAndVerify,\x8BbT\x86\xE6\xD0\x09\xBE,1435179555993.41dc2d9d4bc48ad7ebaf270746c554bc.] [flush region IntegrationTestLoadAndVerify,9F\xB9F\xB9F\xB8\x9A,1435179555993.3f1426537b4e60ebd7def35c3699de0b.] [flush region IntegrationTestLoadAndVerify,7H\xB7H\xB7H\xB6\xA2,1435179555993.0e3209a97bf9eb42b58c63615c23f79e.] [flush region IntegrationTestLoadAndVerify,\xAD\x9B\x89\x09?\xE4\xAB\x90,1435179555993.77f2adf6a45b90361d5a565eab341890.] [flush region IntegrationTestLoadAndVerify,\xAF\xE2f\xBD\xD4t\xAD\xD0,1435179555993.6368be638fff74ea2d2e32f0be895260.] [flush region IntegrationTestLoadAndVerify,D\xBBD\xBBD\xBBC\xEC,1435179555993.85b7b59ec7222414cabbb8b7346885fa.] [flush region IntegrationTestLoadAndVerify,\xC6pk\x02\xB4'D\x1A,1435179555993.2960a6997c59da3f77a72ece369bc53e.] [flush region IntegrationTestLoadAndVerify,\xBF\xC0?\xC0?\xC0=~,1435179555993.2dbf22f465a447f30a0e7153da033e82.] [flush region IntegrationTestLoadAndVerify,\xDDk\xB8\xD9o\xB4\xDA\xD0,1435179555993.ac897dfaf8f16910867f981546e072fa.] [flush region IntegrationTestLoadAndVerify,2\xA8\xC4\xF1\xE9\x842\x10,1435179555993.a607b498542273feae004f7109ac19a9.] [flush region IntegrationTestLoadAndVerify,\x06gO\x8B\xE1\xD5\x06T,1435179555993.de8fdf06a317dd23040caac5be84d1f7.] [flush region IntegrationTestLoadAndVerify,\xBE\xC1>\xC1>\xC1<\x82,1435179555993.905e52a029a0ebbeb75ae38ffbc9cccf.] [flush region IntegrationTestLoadAndVerify,\x07fP\x8A\xE2\xD4\x07P,1435179555993.ce1cb90226ce571311eb99c288f44086.] [flush region IntegrationTestLoadAndVerify,\xF4\x0B\xF4\x0B\xF4\x0B\xF1,,1435179555993.0e763c8c29452ac9efbbf4e3edb1f1fa.] [flush region IntegrationTestLoadAndVerify,\x19\xD3\xE2\xF8uA\x99\x86,1435179555993.6ac8f973af3d229a1f31999ce0cc2be0.] [flush region IntegrationTestLoadAndVerify,\xDF\xA0_\xA0_\xA0\x5C\xFE,1435179555993.12a2cd5b2a34d90efca4ded1e83201c9.] [flush region IntegrationTestLoadAndVerify,4\xCB4\xCB4\xCB4,,1435179555993.4bd893ac3d26df9f71f513c8521eebe6.] [flush region IntegrationTestLoadAndVerify,\xFA\xE0\x8D)\xB1\xBB\xF7\xEC,1435179555993.9d7c10b19d7455942043f870930a0f46.] [flush region IntegrationTestLoadAndVerify,\x86\xC2b0\x19\x0B\x85,,1435179555993.6c46305087dc5f0e0711d194e817b73e.] [flush region IntegrationTestLoadAndVerify,\xABT\xABT\xABT\xA9P,1435179555993.52511d47633b978fd4c2fe7a6c128e34.] [flush region IntegrationTestLoadAndVerify,\xFD^\x0F\xA749zb,1435179555993.850c1655f58b3304f4dbd33bbdbfb678.] [flush region IntegrationTestLoadAndVerify,\xB6\xA4\xC8\xED\xED\x804~,1435179555993.64d4566ef4c97d17d807db99ef84cbd2.] [flush region IntegrationTestLoadAndVerify,V\xF22_\xE9;U\xEC,1435179555993.23e13bf1c50773d1e129a106ada7dc25.] [flush region IntegrationTestLoadAndVerify,\xD8\xCB\xC6\x82\xA1\xF0V>,1435179555993.180fe31b5b0a75df79474b7fb5470c5a.] [flush region IntegrationTestLoadAndVerify,[\xB6\x92\x92\x00H\xDA\xA2,1435179555993.3fd09d04b25acc4826410ebb63326cc2.] [flush region IntegrationTestLoadAndVerify,\x02\xC6^4\x15\x0F\x82\xBE,1435179555993.9a6daaa0fcc58f68604b1bf6b46a3000.] [flush region IntegrationTestLoadAndVerify,\x09\xBFe-\x1C\x08\x89\xA2,1435179555993.2e8dbaa579b4c96d803203d808742a25.] [flush region IntegrationTestLoadAndVerify,y\x98\xB0t\x1E*\xF8*,1435179555993.0831a911059551a337d0849ba3fc2caf.] [flush region IntegrationTestLoadAndVerify,\xA4m\xDBII\x00"~,1435179555993.ea29488b4cf824014a6936f00b8cddb7.] [flush region IntegrationTestLoadAndVerify,\x12\xC8\xA5\x11\xC9\xA4\x12\x90,1435179555993.f654b679149186e961613d76fbc1dc09.] [flush region IntegrationTestLoadAndVerify,;\xB2\x04\xD6\x97\x1F\xBA\xFE,1435179555993.9eb160c1fc819e5af4305434e3cd52e3.] [flush region IntegrationTestLoadAndVerify,\x05V\x17\x9F<1\x85F,1435179555993.6be4b5eb180684e23c70b913aa37b008.] [flush region IntegrationTestLoadAndVerify,\xBBV\xF22_\xE99",1435179555993.33afdf14162730b65818c89e94e4905f.] [flush region IntegrationTestLoadAndVerify,\xB8\xFD\xDD\x90&\xB4\xB6\xD0,1435179555993.c586339e35e5db2feee5f47db7d9fb9d.] [flush region IntegrationTestLoadAndVerify,\x8D\xCD\xA0\x16\xC4\xA9\x0C",1435179555993.56b1c76354cd609fb488e101419049ec.] [flush region IntegrationTestLoadAndVerify,\xC8\xED\xED\x806\xA4\xC6\x90,1435179555993.3b92b9f847f14cfb4aa43af6e841e77f.] [flush region IntegrationTestLoadAndVerify,\xE8\xCE\x0D`V\x84\xE6\x10,1435179555993.24dc72fb1a08d113509ddacf1dd109c8.] [flush region IntegrationTestLoadAndVerify,\xDA%\xDA%\xDA%\xD7\x94,1435179555993.4320db62b70ea350573ec1e92c7356fb.] [flush region IntegrationTestLoadAndVerify,\x0B\xF4\x0B\xF4\x0B\xF4\x0B\xD0,1435179555993.bd8a6f99caeb2c40b97ce52571bda241.] [flush region IntegrationTestLoadAndVerify,\x99\x0B\x86\xC2b0\x17>,1435179555993.27c2a17769b0e9c91626c555bb653392.] [flush region IntegrationTestLoadAndVerify,\x8F\xF0\x0F\xF0\x0F\xF0\x0E>,1435179555993.4aa14fc4b1e0b0b748e68d5df2e40ef9.] [flush region IntegrationTestLoadAndVerify,\xD4>\x0B\x19x\xD0Q\xBE,1435179555993.90cbaec3ea9c6c476e9f53476ef7f559.] [flush region IntegrationTestLoadAndVerify,f\xD0\x0BbT\x86\xE5\x9A,1435179555993.7368554d837f1e7c7dc9fb5a36d8a9d4.] [flush region IntegrationTestLoadAndVerify,K\xEA\xF0}9\xA1\xCB\x06,1435179555993.6653805cd4fd8f2be55a00a1026702cd.] [flush region IntegrationTestLoadAndVerify,\x1B\xD1\xE4\xF6w?\x9B~,1435179555993.decf2f47c05577d7e412f1859398c2f3.] [flush region IntegrationTestLoadAndVerify,=\xF8\xE2\x8B+\xAF\xBD>,1435179555993.d9d969c534770c971b59e2d7638edce2.] [flush region IntegrationTestLoadAndVerify,\x926\xED\xA4\xA4\x80\x10~,1435179555993.7140b8cebca96a81ac45428476ae1565.] [flush region IntegrationTestLoadAndVerify,\x00\x126\xED\xA4\xA4\x80\x12,1435179555993.b74fcdd73616347c2b87b3c9e661d16f.] [flush region IntegrationTestLoadAndVerify,\x7F\xFF\xFF\xFF\xFF\xFF\xFE~,1435179555993.4c253454501e686b2939cccc1fbfeb6a.] [flush region IntegrationTestLoadAndVerify,\x89\x09?\xE4\xAD\x9B\x87l,1435179555993.3145d4fe291d00f810f94f28a4273613.] [flush region IntegrationTestLoadAndVerify,]\xD9\x02kK\x8F\xDC\xBE,1435179555993.ca49c0bda385627db6835f0d976e0898.] [flush region IntegrationTestLoadAndVerify,N\x9F\x17\xC3\xAA\x0C\xCD\xB2,1435179555993.61d2f57afaa00ddb8293849785eb73c0.] [flush region IntegrationTestLoadAndVerify,PB\x07\x1Dt\xD4OP,1435179555993.91c18147c890f7afd11ab0671f4775c2.] [flush region IntegrationTestLoadAndVerify,`\x1F\xE0\x1F\xE0\x1F\xDE\xFE,1435179555993.191ff5eb197ae940aece710bae3a2c7f.] [flush region IntegrationTestLoadAndVerify,\xDB$\xDB$\xDB$\xD8\x90,1435179555993.b09a1dcd8173c10781b2a78c0616ca0c.] [flush region IntegrationTestLoadAndVerify,\xFF\x5C\x11\xA567|Z,1435179555993.0e8e6c35c36cfdc8a436775938db0346.] [flush region IntegrationTestLoadAndVerify,II\x00$m\xDBHl,1435179555993.4f950e3e2ff318a726b427eae7b9d52c.] [flush region IntegrationTestLoadAndVerify,'\xC5\xF0\xEA\x833\xA7N,1435179555993.e013402ff008281c3aff1cfeb88fedf7.] [flush region IntegrationTestLoadAndVerify,p\x0F\xF0\x0F\xF0\x0F\xEE\xBE,1435179555993.7763019bf0e6755b5b6a6b4b557e0066.] [flush region IntegrationTestLoadAndVerify,\xF8\xAB\xE6b\xC1\xD0u\xBE,1435179555993.849a4d930e66ac641e6c85faeeab7b12.] [flush region IntegrationTestLoadAndVerify,\xD1\xE4\xF6w?\x9B\xCFl,1435179555993.7e212be1a211ba2d4be14f86c2d98433.] [flush region IntegrationTestLoadAndVerify,\xC1\xE2\xAF\x99\x8B\x07?\x9A,1435179555993.b7508c0821051eb0d23f2633d165fb07.] [flush region IntegrationTestLoadAndVerify,\xF1\xD7ME\x04 n\xFE,1435179555993.7bf375cc57317ed095add39c77cef6b0.] [flush region IntegrationTestLoadAndVerify,\xB4p"&\xFD\x94\xB2P,1435179555993.ceb5aeda5e6c1b22ca146fbd5378dd34.] [flush region IntegrationTestLoadAndVerify,v\xE4\x89-\xAD\xBF\xF5~,1435179555993.b2f2bde86d86922b419debc759b5a71c.] [flush region IntegrationTestLoadAndVerify,G\x14Y]}\xEF\xC6>,1435179555993.424cbf1b5ec4bf8a3935defe6cb8abea.] [flush region IntegrationTestLoadAndVerify,\xCB"\x94G&\x90H\xBE,1435179555993.e291cb210a23e943445f630bcf1c25a4.] [flush region IntegrationTestLoadAndVerify,\x0E\x16{\xCDW;\x0D\xEC,1435179555993.6dc50927104642a07676f4b02e255dbe.] [flush region IntegrationTestLoadAndVerify,+\xE6b\xC1\xD0x\xABb,1435179555993.324fb2f483a5c2127793ff596158e351.] [flush region IntegrationTestLoadAndVerify,\xE1\xE7=T\xF40_>,1435179555993.318e64f5d10cb1cb4371c35d8c983825.] [flush region IntegrationTestLoadAndVerify,f\xF4y=\x9D\xCF\xE5\xBE,1435179555993.b8ca942870cf0918118dc71b96ecde73.] [flush region IntegrationTestLoadAndVerify,\x17D)\x8DN\x1F\x96\xFE,1435179555993.1daa84ccf1c79d253b655eecffb32fbb.] [flush region IntegrationTestLoadAndVerify,\x07\x1Dt\xD4PB\x07\x08,1435179555993.6c22bdcb400cd682231a3b17493a6768.] [flush region IntegrationTestLoadAndVerify, q\xD7ME\x04 \x10,1435179555993.e8327e102b8810fab093df74ec1a9860.] [flush region IntegrationTestLoadAndVerify,\xA294\x82Y\x14\xA0P,1435179555993.6b5408e2a98f977b49b3499a29c5cccd.] [flush region IntegrationTestLoadAndVerify,\xA2\xB8\xB5\x01\xD9\x94 \xCE,1435179555993.9865d345300d1d28b2ca8961d93c8b0e.] [flush region IntegrationTestLoadAndVerify,\xB2)Dri\x04\xB0\x10,1435179555993.4e91b88f6497d2e398fee750b3451dec.] [flush region IntegrationTestLoadAndVerify,k\x824\xA6\xC6\xEF\xEA>,1435179555993.741e9be4173db2f24b565b3bfff3f873.] [flush region IntegrationTestLoadAndVerify,\x82kK\x8F\xDD\xD9\x00\xE2,1435179555993.01e3b324ac9ed4a1305f98bb9fc9acf3.] [flush region IntegrationTestLoadAndVerify,\x94k\x94k\x94k\x92\xAC,1435179555993.af97415e7d84ac6c37bb981106d957b2.] [flush region IntegrationTestLoadAndVerify,\x15X^|\xF0\xC6\x15\x18,1435179555993.176087e404d89e61912ea8ed54695e9a.] [flush region IntegrationTestLoadAndVerify,b\x8B+\xAF\xBD\xF8\xE1b,1435179555993.7893be6a0575fd87117e5ab5e297c73f.] [flush region IntegrationTestLoadAndVerify,\xC4\x84\x9F\xF2V\xCD\xC24,1435179555993.ffe7173d67f18af35738dca1e4e5f300.] [flush region IntegrationTestLoadAndVerify,i;V\xF22_\xE7\xFE,1435179555993.ddc19c9d338f492acee56d57880eed33.] [flush region IntegrationTestLoadAndVerify,y=\x9D\xCF\xE6\xF4w\xD0,1435179555993.025e2a9b43cdc0dce266cde51646edc4.] [flush region IntegrationTestLoadAndVerify,\xF6w?\x9B\xD1\xE4\xF3\x90,1435179555993.9a7a5c6975dfd85bf2045fd1edeb7d77.] [flush region IntegrationTestLoadAndVerify,\xCD{\xA8\xE9_\xC4\xCB\x10,1435179555993.24e4bf960d65e7dde33deb1e62b54569.] [flush region IntegrationTestLoadAndVerify,d\x89-\xAD\xBF\xF6\xE3Z,1435179555993.9fb53c01031ec8a16610e3b170d58a19.] [flush region IntegrationTestLoadAndVerify,\x9F\xF2V\xCD\xC4\x84\x9E\x10,1435179555993.c3b4b6ff429473446a78ba44f804e28b.] [flush region IntegrationTestLoadAndVerify,B\x86\x9D\xF4T\xCF\xC1\xBE,1435179555993.78459f05b80e3cdbd3e5667d8f921879.] [flush region IntegrationTestLoadAndVerify,\xD6\x97\x1F\xBB\xB2\x04\xD4\x10,1435179555993.ad576e0d9564b513c4b15ec3478c4e68.] [flush region IntegrationTestLoadAndVerify,\xE4@Q\xF7-d\xE1\x90,1435179555993.334499b0e1a3682b1172a85c63578b84.] [flush region IntegrationTestLoadAndVerify,-\xF6\x9B\xADw\x1B-l,1435179555993.bbb0d0c97217a61a1887c4dd00fa074a.] [flush region IntegrationTestLoadAndVerify,\x10]Y\x81\xEB\xCB\x10,,1435179555993.ce00a9bc6e112076278d09f9ca564874.] [flush region IntegrationTestLoadAndVerify,\x1Ea\x9Ea\x9Ea\x9E\x06,1435179555993.ba84e091b17d32c305a5d3b81f6b9e70.] [flush region IntegrationTestLoadAndVerify,\x9Bd\x9Bd\x9Bd\x99\x90,1435179555993.17c5b5256bbc7f99397c5707e605acbf.] [flush region IntegrationTestLoadAndVerify,\xEB\x02\xB4'Fph>,1435179555993.3af3ee4830aff4383789a15edde8b4e5.]