RegionServer status for hbase4537.frc3.facebook.com,16020,1440656248801 as of Thu Aug 27 12:03:41 PDT 2015 Version Info: =========================================================== HBase 1.2.0-SNAPSHOT Source code repository git://dev2022.prn2.facebook.com/data/users/sfllaw/hbase revision=8489fccf3d830214f479c7263fc395f01b25abbd Compiled by sfllaw on Tue Aug 25 14:17:41 PDT 2015 From source with checksum 6926cdb75a8617c66ef05801fb892d13 Hadoop 2.6.0 Source code repository https://git-wip-us.apache.org/repos/asf/hadoop.git revision=e3496499ecb8d220fba99dc5ed4c99c8f9e33bb1 (HEAD, tag: release-2.6.0-rc1, origin/branch-2.6.0, branch-2.6.0) Compiled by acmurthy on 2014-11-13T22:35Z Tasks: =========================================================== Task: Compacting evts in mytable,38cf001b:gCNaR20NqRAx,1437586774027.8364b228aae1db4fd387fb43a84c0887. Status: RUNNING:Compacting store evts Running for 33187s Task: Compacting evts in mytable,52e64cf2:gAu75CpwU2IH,1437586870140.8529b87cf90caa664f8c8f76c29f66d7. Status: RUNNING:Compacting store evts Running for 33117s Task: Flushing mytable,0e9c40df:gHRMnoeRoHfi,1437586617090.cfd4f0e2a31879aaf47392fffb22454d. Status: ABORTED:Preparing to flush by snapshotting stores in cfd4f0e2a31879aaf47392fffb22454d Completed 21s ago Ran for 60s Task: Flushing mytable,43226c56:gELoKpZaukZu,1437586808965.471189bf69582d8d028c205abe907099. Status: ABORTED:Preparing to flush by snapshotting stores in 471189bf69582d8d028c205abe907099 Completed 17s ago Ran for 60s Task: Flushing mytable,3f4bc8,1437586791858.8277dad8e0af2c9a4a85454cc61fc3f7. Status: RUNNING:Preparing to flush by snapshotting stores in 8277dad8e0af2c9a4a85454cc61fc3f7 Running for 21s Task: Flushing mytable,9767b73d:gGMi5MBknVQn,1437587109978.6ff8e150920337728aae90a5d6506505. Status: RUNNING:Preparing to flush by snapshotting stores in 6ff8e150920337728aae90a5d6506505 Running for 17s Executors: =========================================================== Status for executor: Executor-4-RS_CLOSE_META-hbase4537:16020 ======================================= 0 events queued, 0 running Status for executor: Executor-2-RS_OPEN_META-hbase4537:16020 ======================================= 0 events queued, 0 running Status for executor: Executor-3-RS_CLOSE_REGION-hbase4537:16020 ======================================= 0 events queued, 0 running Status for executor: Executor-1-RS_OPEN_REGION-hbase4537:16020 ======================================= 0 events queued, 0 running Status for executor: Executor-6-RS_REGION_REPLICA_FLUSH_OPS-hbase4537:16020 ======================================= 0 events queued, 0 running Status for executor: Executor-5-RS_LOG_REPLAY_OPS-hbase4537:16020 ======================================= 0 events queued, 0 running Process Thread Dump: 116 active threads Thread 5331 (Timer for 'HBase' metrics system): State: TIMED_WAITING Blocked count: 0 Waited count: 27 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 5299 (1631730183@qtp-4073506-2): State: RUNNABLE Blocked count: 252 Waited count: 254 Stack: sun.management.ThreadImpl.getThreadInfo1(Native Method) sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:174) sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:139) org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:165) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:497) org.apache.hadoop.hbase.util.Threads.printThreadInfo(Threads.java:302) org.apache.hadoop.hbase.regionserver.RSDumpServlet.doGet(RSDumpServlet.java:79) javax.servlet.http.HttpServlet.service(HttpServlet.java:707) javax.servlet.http.HttpServlet.service(HttpServlet.java:820) org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:113) org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) org.apache.hadoop.hbase.http.ClickjackingPreventionFilter.doFilter(ClickjackingPreventionFilter.java:48) org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) org.apache.hadoop.hbase.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:1354) org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) Thread 4122 (ResponseProcessor for block BP-1735829752-10.210.49.21-1437433901380:blk_1105356126_31664161): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118) java.io.FilterInputStream.read(FilterInputStream.java:83) java.io.FilterInputStream.read(FilterInputStream.java:83) org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2201) org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176) org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:867) Thread 4121 (DataStreamer for file /FBTRACE004-HBASE/WALs/hbase4537.frc3.facebook.com,16020,1440656248801/hbase4537.frc3.facebook.com%2C16020%2C1440656248801.default.1440669198699 block BP-1735829752-10.210.49.21-1437433901380:blk_1105356126_31664161): State: TIMED_WAITING Blocked count: 3 Waited count: 1106 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:564) Thread 168 (regionserver/hbase4537.frc3.facebook.com/10.210.81.27:16020-shortCompactions-1440656347829): State: WAITING Blocked count: 5879 Waited count: 9589 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@393a50fd Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 167 (regionserver/hbase4537.frc3.facebook.com/10.210.81.27:16020-shortCompactions-1440656347827): State: WAITING Blocked count: 6167 Waited count: 10241 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@393a50fd Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 166 (regionserver/hbase4537.frc3.facebook.com/10.210.81.27:16020-shortCompactions-1440656347827): State: TIMED_WAITING Blocked count: 5829 Waited count: 42464 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:167) org.apache.hadoop.hbase.regionserver.wal.FSHLog.blockOnSync(FSHLog.java:1332) org.apache.hadoop.hbase.regionserver.wal.FSHLog.publishSyncThenBlockOnCompletion(FSHLog.java:1326) org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1445) org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeCompactionMarker(WALUtil.java:64) org.apache.hadoop.hbase.regionserver.HStore.writeCompactionWalRecord(HStore.java:1288) org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1227) org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1814) org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:525) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 165 (regionserver/hbase4537.frc3.facebook.com/10.210.81.27:16020-shortCompactions-1440656347827): State: WAITING Blocked count: 6047 Waited count: 10394 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@393a50fd Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 158 (IPC Parameter Sending Thread #1): State: TIMED_WAITING Blocked count: 49 Waited count: 53070 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 143 (RS_OPEN_REGION-hbase4537:16020-2): State: WAITING Blocked count: 102 Waited count: 136 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@70d0107d Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 142 (RS_OPEN_REGION-hbase4537:16020-0): State: WAITING Blocked count: 145 Waited count: 141 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@70d0107d Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 141 (RS_OPEN_REGION-hbase4537:16020-1): State: WAITING Blocked count: 126 Waited count: 177 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@70d0107d Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 140 (org.apache.hadoop.hdfs.PeerCache@fbf634a): State: TIMED_WAITING Blocked count: 0 Waited count: 15291 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:244) org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41) org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119) java.lang.Thread.run(Thread.java:745) Thread 136 (SplitLogWorker-hbase4537:16020): State: TIMED_WAITING Blocked count: 2 Waited count: 9196 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.coordination.ZkSplitLogWorkerCoordination.taskLoop(ZkSplitLogWorkerCoordination.java:445) org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:142) java.lang.Thread.run(Thread.java:745) Thread 135 (hbase4537:16020Replication Statistics #0): State: TIMED_WAITING Blocked count: 0 Waited count: 154 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 133 (regionserver/hbase4537.frc3.facebook.com/10.210.81.27:16020-EventThread): State: WAITING Blocked count: 0 Waited count: 3 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@37b4540e Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) Thread 132 (regionserver/hbase4537.frc3.facebook.com/10.210.81.27:16020-SendThread(hbasectrl340.frc3.facebook.com:2181)): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) Thread 109 (regionserver/hbase4537.frc3.facebook.com/10.210.81.27:16020.leaseChecker): State: TIMED_WAITING Blocked count: 0 Waited count: 4597 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.regionserver.Leases.run(Leases.java:94) java.lang.Thread.run(Thread.java:745) Thread 131 (hbase4537.frc3.facebook.com,16020,1440656248801_ChoreService_1): State: TIMED_WAITING Blocked count: 205 Waited count: 10278 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 129 (MemStoreFlusher.1): State: TIMED_WAITING Blocked count: 10431 Waited count: 20451 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037) java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328) java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277) org.apache.hadoop.hbase.wal.WALKey.getSequenceId(WALKey.java:319) org.apache.hadoop.hbase.regionserver.HRegion.getNextSequenceId(HRegion.java:2439) org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2185) org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2064) org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2028) org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1919) org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:1845) org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:510) org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:471) org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$900(MemStoreFlusher.java:75) org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:259) java.lang.Thread.run(Thread.java:745) Thread 127 (MemStoreFlusher.0): State: TIMED_WAITING Blocked count: 10996 Waited count: 21252 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037) java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328) java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277) org.apache.hadoop.hbase.wal.WALKey.getSequenceId(WALKey.java:319) org.apache.hadoop.hbase.regionserver.HRegion.getNextSequenceId(HRegion.java:2439) org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2185) org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2064) org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2028) org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1919) org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:1845) org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:510) org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:471) org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$900(MemStoreFlusher.java:75) org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:259) java.lang.Thread.run(Thread.java:745) Thread 100 (regionserver/hbase4537.frc3.facebook.com/10.210.81.27:16020.logRoller): State: RUNNABLE Blocked count: 4686 Waited count: 8306 Stack: java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1018) java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328) java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277) org.apache.hadoop.hbase.regionserver.wal.FSHLog$SafePointZigZagLatch.waitSafePoint(FSHLog.java:1613) org.apache.hadoop.hbase.regionserver.wal.FSHLog.replaceWriter(FSHLog.java:799) org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:679) org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:137) java.lang.Thread.run(Thread.java:745) Thread 126 (HBase-Metrics2-1): State: TIMED_WAITING Blocked count: 0 Waited count: 17992 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 124 (sync.4): State: WAITING Blocked count: 6203067 Waited count: 7870319 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@493a0ad9 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1183) java.lang.Thread.run(Thread.java:745) Thread 123 (sync.3): State: WAITING Blocked count: 6205967 Waited count: 7873073 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3c3341af Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1183) java.lang.Thread.run(Thread.java:745) Thread 122 (sync.2): State: WAITING Blocked count: 6203221 Waited count: 7869302 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56b08788 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1183) java.lang.Thread.run(Thread.java:745) Thread 121 (sync.1): State: WAITING Blocked count: 6203852 Waited count: 7870914 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@20aec560 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1183) java.lang.Thread.run(Thread.java:745) Thread 120 (sync.0): State: WAITING Blocked count: 6202849 Waited count: 7869307 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@395b7f6e Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1183) java.lang.Thread.run(Thread.java:745) Thread 125 (regionserver/hbase4537.frc3.facebook.com/10.210.81.27:16020.append-pool1-t1): State: TIMED_WAITING Blocked count: 239951 Waited count: 38793849 Stack: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:460) org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.attainSafePoint(FSHLog.java:1786) org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1761) org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1672) com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 118 (LeaseRenewer:hadoop@fbtrace004): State: TIMED_WAITING Blocked count: 1528 Waited count: 49017 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:438) org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) java.lang.Thread.run(Thread.java:745) Thread 114 (LruBlockCacheStatsExecutor): State: TIMED_WAITING Blocked count: 0 Waited count: 154 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 112 (regionserver/hbase4537.frc3.facebook.com/10.210.81.27:16020.LruBlockCache.EvictionThread): State: TIMED_WAITING Blocked count: 0 Waited count: 4598 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:803) java.lang.Thread.run(Thread.java:745) Thread 111 (IPC Client (1893960929) connection to hbasectrl341.frc3.facebook.com/10.210.66.46:16000 from hadoop): State: TIMED_WAITING Blocked count: 15322 Waited count: 61206 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.waitForWork(RpcClientImpl.java:565) org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.run(RpcClientImpl.java:587) Thread 108 (regionserver/hbase4537.frc3.facebook.com/10.210.81.27:16020-longCompactions-1440656250329): State: WAITING Blocked count: 2915 Waited count: 4603 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2e627ac4 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:90) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 107 (regionserver/hbase4537.frc3.facebook.com/10.210.81.27:16020-longCompactions-1440656250328): State: TIMED_WAITING Blocked count: 2929 Waited count: 38158 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:167) org.apache.hadoop.hbase.regionserver.wal.FSHLog.blockOnSync(FSHLog.java:1332) org.apache.hadoop.hbase.regionserver.wal.FSHLog.publishSyncThenBlockOnCompletion(FSHLog.java:1326) org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1445) org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeCompactionMarker(WALUtil.java:64) org.apache.hadoop.hbase.regionserver.HStore.writeCompactionWalRecord(HStore.java:1288) org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1227) org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1814) org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:525) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 106 (regionserver/hbase4537.frc3.facebook.com/10.210.81.27:16020-longCompactions-1440656250327): State: WAITING Blocked count: 3190 Waited count: 5639 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2e627ac4 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:90) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 105 (regionserver/hbase4537.frc3.facebook.com/10.210.81.27:16020-longCompactions-1440656250327): State: WAITING Blocked count: 3196 Waited count: 4520 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2e627ac4 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:90) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 104 (regionserver/hbase4537.frc3.facebook.com/10.210.81.27:16020-longCompactions-1440656250325): State: WAITING Blocked count: 2962 Waited count: 5203 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2e627ac4 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:90) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 103 (JvmPauseMonitor): State: TIMED_WAITING Blocked count: 0 Waited count: 91907 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:159) java.lang.Thread.run(Thread.java:745) Thread 102 (regionserver/hbase4537.frc3.facebook.com/10.210.81.27:16020-EventThread): State: WAITING Blocked count: 0 Waited count: 3 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7b391e18 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) Thread 101 (regionserver/hbase4537.frc3.facebook.com/10.210.81.27:16020-SendThread(hbasectrl342.frc3.facebook.com:2181)): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) Thread 25 (regionserver/hbase4537.frc3.facebook.com/10.210.81.27:16020): State: TIMED_WAITING Blocked count: 15435 Waited count: 30752 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:92) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:56) org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:954) java.lang.Thread.run(Thread.java:745) Thread 99 (Timer-0): State: TIMED_WAITING Blocked count: 1 Waited count: 1534 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 98 (559050604@qtp-4073506-1 - Acceptor0 SelectChannelConnector@0.0.0.0:16030): State: RUNNABLE Blocked count: 9 Waited count: 1 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498) org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192) org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Thread 95 (ReplicationRpcServer.handler=2,queue=0,port=16020): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@462f39a Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 94 (ReplicationRpcServer.handler=1,queue=0,port=16020): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@462f39a Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 93 (ReplicationRpcServer.handler=0,queue=0,port=16020): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@462f39a Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 92 (PriorityRpcServer.handler=19,queue=1,port=16020): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3798e4d0 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 91 (PriorityRpcServer.handler=18,queue=0,port=16020): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1009a6ed Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 90 (PriorityRpcServer.handler=17,queue=1,port=16020): State: WAITING Blocked count: 3 Waited count: 7 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3798e4d0 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 89 (PriorityRpcServer.handler=16,queue=0,port=16020): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1009a6ed Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 88 (PriorityRpcServer.handler=15,queue=1,port=16020): State: WAITING Blocked count: 3 Waited count: 7 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3798e4d0 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 87 (PriorityRpcServer.handler=14,queue=0,port=16020): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1009a6ed Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 86 (PriorityRpcServer.handler=13,queue=1,port=16020): State: WAITING Blocked count: 4 Waited count: 7 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3798e4d0 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 85 (PriorityRpcServer.handler=12,queue=0,port=16020): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1009a6ed Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 84 (PriorityRpcServer.handler=11,queue=1,port=16020): State: WAITING Blocked count: 3 Waited count: 7 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3798e4d0 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 83 (PriorityRpcServer.handler=10,queue=0,port=16020): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1009a6ed Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 82 (PriorityRpcServer.handler=9,queue=1,port=16020): State: WAITING Blocked count: 4 Waited count: 7 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3798e4d0 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 81 (PriorityRpcServer.handler=8,queue=0,port=16020): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1009a6ed Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 80 (PriorityRpcServer.handler=7,queue=1,port=16020): State: WAITING Blocked count: 3 Waited count: 7 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3798e4d0 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 79 (PriorityRpcServer.handler=6,queue=0,port=16020): State: WAITING Blocked count: 3 Waited count: 7 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1009a6ed Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 78 (PriorityRpcServer.handler=5,queue=1,port=16020): State: WAITING Blocked count: 3 Waited count: 7 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3798e4d0 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 77 (PriorityRpcServer.handler=4,queue=0,port=16020): State: WAITING Blocked count: 3 Waited count: 7 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1009a6ed Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 76 (PriorityRpcServer.handler=3,queue=1,port=16020): State: WAITING Blocked count: 22 Waited count: 7 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3798e4d0 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 75 (PriorityRpcServer.handler=2,queue=0,port=16020): State: WAITING Blocked count: 3 Waited count: 7 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1009a6ed Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 74 (PriorityRpcServer.handler=1,queue=1,port=16020): State: WAITING Blocked count: 4 Waited count: 7 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3798e4d0 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 73 (PriorityRpcServer.handler=0,queue=0,port=16020): State: WAITING Blocked count: 3 Waited count: 7 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1009a6ed Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 72 (RW.default.scanRpcServer.handler=29,queue=20,port=16020): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@71d2dc87 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.BoundedPriorityBlockingQueue.take(BoundedPriorityBlockingQueue.java:214) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 71 (RW.default.readRpcServer.handler=28,queue=17,port=16020): State: WAITING Blocked count: 662 Waited count: 23780 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@623f1f04 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.BoundedPriorityBlockingQueue.take(BoundedPriorityBlockingQueue.java:214) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 70 (RW.default.readRpcServer.handler=27,queue=16,port=16020): State: WAITING Blocked count: 670 Waited count: 23962 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1271d9f7 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.BoundedPriorityBlockingQueue.take(BoundedPriorityBlockingQueue.java:214) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 69 (RW.default.readRpcServer.handler=26,queue=15,port=16020): State: WAITING Blocked count: 635 Waited count: 23947 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f122b9a Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.BoundedPriorityBlockingQueue.take(BoundedPriorityBlockingQueue.java:214) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 68 (RW.default.readRpcServer.handler=25,queue=19,port=16020): State: WAITING Blocked count: 1307 Waited count: 34548 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2843f01b Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.BoundedPriorityBlockingQueue.take(BoundedPriorityBlockingQueue.java:214) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 67 (RW.default.readRpcServer.handler=24,queue=18,port=16020): State: WAITING Blocked count: 1293 Waited count: 33623 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@329e0d28 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.BoundedPriorityBlockingQueue.take(BoundedPriorityBlockingQueue.java:214) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 66 (RW.default.readRpcServer.handler=23,queue=17,port=16020): State: WAITING Blocked count: 677 Waited count: 23766 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@623f1f04 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.BoundedPriorityBlockingQueue.take(BoundedPriorityBlockingQueue.java:214) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 65 (RW.default.readRpcServer.handler=22,queue=16,port=16020): State: WAITING Blocked count: 626 Waited count: 23478 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1271d9f7 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.BoundedPriorityBlockingQueue.take(BoundedPriorityBlockingQueue.java:214) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 64 (RW.default.readRpcServer.handler=21,queue=15,port=16020): State: WAITING Blocked count: 598 Waited count: 23601 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f122b9a Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.BoundedPriorityBlockingQueue.take(BoundedPriorityBlockingQueue.java:214) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:127) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 63 (RW.default.writeRpcServer.handler=20,queue=5,port=16020): State: TIMED_WAITING Blocked count: 312072 Waited count: 559345 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:167) org.apache.hadoop.hbase.regionserver.wal.FSHLog.blockOnSync(FSHLog.java:1332) org.apache.hadoop.hbase.regionserver.wal.FSHLog.publishSyncThenBlockOnCompletion(FSHLog.java:1326) org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1460) org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:7851) org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3171) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2819) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2761) org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:696) org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:658) org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2031) org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32594) org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2117) org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 62 (RW.default.writeRpcServer.handler=19,queue=4,port=16020): State: TIMED_WAITING Blocked count: 313067 Waited count: 560644 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:167) org.apache.hadoop.hbase.regionserver.wal.FSHLog.blockOnSync(FSHLog.java:1332) org.apache.hadoop.hbase.regionserver.wal.FSHLog.publishSyncThenBlockOnCompletion(FSHLog.java:1326) org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1460) org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:7851) org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3171) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2819) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2761) org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:696) org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:658) org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2031) org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32594) org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2117) org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 61 (RW.default.writeRpcServer.handler=18,queue=3,port=16020): State: TIMED_WAITING Blocked count: 312839 Waited count: 560098 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:167) org.apache.hadoop.hbase.regionserver.wal.FSHLog.blockOnSync(FSHLog.java:1332) org.apache.hadoop.hbase.regionserver.wal.FSHLog.publishSyncThenBlockOnCompletion(FSHLog.java:1326) org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1460) org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:7851) org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3171) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2819) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2761) org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:696) org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:658) org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2031) org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32594) org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2117) org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 60 (RW.default.writeRpcServer.handler=17,queue=2,port=16020): State: TIMED_WAITING Blocked count: 313190 Waited count: 560894 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:167) org.apache.hadoop.hbase.regionserver.wal.FSHLog.blockOnSync(FSHLog.java:1332) org.apache.hadoop.hbase.regionserver.wal.FSHLog.publishSyncThenBlockOnCompletion(FSHLog.java:1326) org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1460) org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:7851) org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3171) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2819) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2761) org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:696) org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:658) org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2031) org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32594) org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2117) org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 59 (RW.default.writeRpcServer.handler=16,queue=1,port=16020): State: TIMED_WAITING Blocked count: 313808 Waited count: 561798 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:167) org.apache.hadoop.hbase.regionserver.wal.FSHLog.blockOnSync(FSHLog.java:1332) org.apache.hadoop.hbase.regionserver.wal.FSHLog.publishSyncThenBlockOnCompletion(FSHLog.java:1326) org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1460) org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:7851) org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3171) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2819) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2761) org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:696) org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:658) org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2031) org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32594) org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2117) org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 58 (RW.default.writeRpcServer.handler=15,queue=0,port=16020): State: TIMED_WAITING Blocked count: 312804 Waited count: 559970 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:167) org.apache.hadoop.hbase.regionserver.wal.FSHLog.blockOnSync(FSHLog.java:1332) org.apache.hadoop.hbase.regionserver.wal.FSHLog.publishSyncThenBlockOnCompletion(FSHLog.java:1326) org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1460) org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:7851) org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3171) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2819) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2761) org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:696) org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:658) org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2031) org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32594) org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2117) org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 57 (RW.default.writeRpcServer.handler=14,queue=14,port=16020): State: TIMED_WAITING Blocked count: 622361 Waited count: 1036742 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:167) org.apache.hadoop.hbase.regionserver.wal.FSHLog.blockOnSync(FSHLog.java:1332) org.apache.hadoop.hbase.regionserver.wal.FSHLog.publishSyncThenBlockOnCompletion(FSHLog.java:1326) org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1460) org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:7851) org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3171) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2819) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2761) org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:696) org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:658) org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2031) org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32594) org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2117) org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 56 (RW.default.writeRpcServer.handler=13,queue=13,port=16020): State: TIMED_WAITING Blocked count: 625104 Waited count: 1040722 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:167) org.apache.hadoop.hbase.regionserver.wal.FSHLog.blockOnSync(FSHLog.java:1332) org.apache.hadoop.hbase.regionserver.wal.FSHLog.publishSyncThenBlockOnCompletion(FSHLog.java:1326) org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1460) org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:7851) org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3171) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2819) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2761) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2765) org.apache.hadoop.hbase.regionserver.HRegion.doBatchMutate(HRegion.java:3495) org.apache.hadoop.hbase.regionserver.HRegion.checkAndMutate(HRegion.java:3380) org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2102) org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32584) org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2117) org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 55 (RW.default.writeRpcServer.handler=12,queue=12,port=16020): State: TIMED_WAITING Blocked count: 624224 Waited count: 1039282 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:167) org.apache.hadoop.hbase.regionserver.wal.FSHLog.blockOnSync(FSHLog.java:1332) org.apache.hadoop.hbase.regionserver.wal.FSHLog.publishSyncThenBlockOnCompletion(FSHLog.java:1326) org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1460) org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:7851) org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3171) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2819) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2761) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2765) org.apache.hadoop.hbase.regionserver.HRegion.doBatchMutate(HRegion.java:3495) org.apache.hadoop.hbase.regionserver.HRegion.checkAndMutate(HRegion.java:3380) org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2102) org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32584) org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2117) org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 54 (RW.default.writeRpcServer.handler=11,queue=11,port=16020): State: WAITING Blocked count: 622559 Waited count: 1004506 Waiting on java.util.LinkedList@2a4ae5c5 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.MultiVersionConsistencyControl.waitForPreviousTransactionsComplete(MultiVersionConsistencyControl.java:224) org.apache.hadoop.hbase.regionserver.MultiVersionConsistencyControl.waitForPreviousTransactionsComplete(MultiVersionConsistencyControl.java:203) org.apache.hadoop.hbase.regionserver.HRegion.checkAndMutate(HRegion.java:3307) org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2102) org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32584) org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2117) org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 53 (RW.default.writeRpcServer.handler=10,queue=10,port=16020): State: WAITING Blocked count: 623791 Waited count: 1005714 Waiting on java.util.LinkedList@2a4ae5c5 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.MultiVersionConsistencyControl.waitForPreviousTransactionsComplete(MultiVersionConsistencyControl.java:224) org.apache.hadoop.hbase.regionserver.MultiVersionConsistencyControl.waitForPreviousTransactionsComplete(MultiVersionConsistencyControl.java:203) org.apache.hadoop.hbase.regionserver.HRegion.checkAndMutate(HRegion.java:3307) org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2102) org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32584) org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2117) org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 52 (RW.default.writeRpcServer.handler=9,queue=9,port=16020): State: TIMED_WAITING Blocked count: 623065 Waited count: 1037796 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:167) org.apache.hadoop.hbase.regionserver.wal.FSHLog.blockOnSync(FSHLog.java:1332) org.apache.hadoop.hbase.regionserver.wal.FSHLog.publishSyncThenBlockOnCompletion(FSHLog.java:1326) org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1460) org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:7851) org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3171) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2819) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2761) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2765) org.apache.hadoop.hbase.regionserver.HRegion.doBatchMutate(HRegion.java:3495) org.apache.hadoop.hbase.regionserver.HRegion.checkAndMutate(HRegion.java:3380) org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2102) org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32584) org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2117) org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 51 (RW.default.writeRpcServer.handler=8,queue=8,port=16020): State: TIMED_WAITING Blocked count: 626835 Waited count: 1042533 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:167) org.apache.hadoop.hbase.regionserver.wal.FSHLog.blockOnSync(FSHLog.java:1332) org.apache.hadoop.hbase.regionserver.wal.FSHLog.publishSyncThenBlockOnCompletion(FSHLog.java:1326) org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1460) org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:7851) org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3171) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2819) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2761) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2765) org.apache.hadoop.hbase.regionserver.HRegion.doBatchMutate(HRegion.java:3495) org.apache.hadoop.hbase.regionserver.HRegion.checkAndMutate(HRegion.java:3380) org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2102) org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32584) org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2117) org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 50 (RW.default.writeRpcServer.handler=7,queue=7,port=16020): State: TIMED_WAITING Blocked count: 623983 Waited count: 1039419 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:167) org.apache.hadoop.hbase.regionserver.wal.FSHLog.blockOnSync(FSHLog.java:1332) org.apache.hadoop.hbase.regionserver.wal.FSHLog.publishSyncThenBlockOnCompletion(FSHLog.java:1326) org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1460) org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:7851) org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3171) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2819) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2761) org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:696) org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:658) org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2031) org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32594) org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2117) org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 49 (RW.default.writeRpcServer.handler=6,queue=6,port=16020): State: TIMED_WAITING Blocked count: 623486 Waited count: 1038028 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:167) org.apache.hadoop.hbase.regionserver.wal.FSHLog.blockOnSync(FSHLog.java:1332) org.apache.hadoop.hbase.regionserver.wal.FSHLog.publishSyncThenBlockOnCompletion(FSHLog.java:1326) org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1460) org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:7851) org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3171) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2819) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2761) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2765) org.apache.hadoop.hbase.regionserver.HRegion.doBatchMutate(HRegion.java:3495) org.apache.hadoop.hbase.regionserver.HRegion.checkAndMutate(HRegion.java:3380) org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2102) org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32584) org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2117) org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 48 (RW.default.writeRpcServer.handler=5,queue=5,port=16020): State: TIMED_WAITING Blocked count: 312639 Waited count: 559721 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:167) org.apache.hadoop.hbase.regionserver.wal.FSHLog.blockOnSync(FSHLog.java:1332) org.apache.hadoop.hbase.regionserver.wal.FSHLog.publishSyncThenBlockOnCompletion(FSHLog.java:1326) org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1460) org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:7851) org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3171) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2819) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2761) org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:696) org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:658) org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2031) org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32594) org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2117) org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 47 (RW.default.writeRpcServer.handler=4,queue=4,port=16020): State: TIMED_WAITING Blocked count: 313267 Waited count: 560954 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:167) org.apache.hadoop.hbase.regionserver.wal.FSHLog.blockOnSync(FSHLog.java:1332) org.apache.hadoop.hbase.regionserver.wal.FSHLog.publishSyncThenBlockOnCompletion(FSHLog.java:1326) org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1460) org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:7851) org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3171) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2819) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2761) org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:696) org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:658) org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2031) org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32594) org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2117) org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 46 (RW.default.writeRpcServer.handler=3,queue=3,port=16020): State: TIMED_WAITING Blocked count: 312723 Waited count: 560697 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:167) org.apache.hadoop.hbase.regionserver.wal.FSHLog.blockOnSync(FSHLog.java:1332) org.apache.hadoop.hbase.regionserver.wal.FSHLog.publishSyncThenBlockOnCompletion(FSHLog.java:1326) org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1460) org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:7851) org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3171) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2819) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2761) org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:696) org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:658) org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2031) org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32594) org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2117) org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 45 (RW.default.writeRpcServer.handler=2,queue=2,port=16020): State: TIMED_WAITING Blocked count: 312926 Waited count: 560673 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:167) org.apache.hadoop.hbase.regionserver.wal.FSHLog.blockOnSync(FSHLog.java:1332) org.apache.hadoop.hbase.regionserver.wal.FSHLog.publishSyncThenBlockOnCompletion(FSHLog.java:1326) org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1460) org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:7851) org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3171) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2819) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2761) org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:696) org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:658) org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2031) org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32594) org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2117) org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 44 (RW.default.writeRpcServer.handler=1,queue=1,port=16020): State: TIMED_WAITING Blocked count: 313077 Waited count: 561289 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:167) org.apache.hadoop.hbase.regionserver.wal.FSHLog.blockOnSync(FSHLog.java:1332) org.apache.hadoop.hbase.regionserver.wal.FSHLog.publishSyncThenBlockOnCompletion(FSHLog.java:1326) org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1460) org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:7851) org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3171) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2819) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2761) org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:696) org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:658) org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2031) org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32594) org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2117) org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 43 (RW.default.writeRpcServer.handler=0,queue=0,port=16020): State: TIMED_WAITING Blocked count: 312287 Waited count: 559662 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:167) org.apache.hadoop.hbase.regionserver.wal.FSHLog.blockOnSync(FSHLog.java:1332) org.apache.hadoop.hbase.regionserver.wal.FSHLog.publishSyncThenBlockOnCompletion(FSHLog.java:1326) org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1460) org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:7851) org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3171) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2819) org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2761) org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:696) org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:658) org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2031) org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32594) org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2117) org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) java.lang.Thread.run(Thread.java:745) Thread 26 (RpcServer.listener,port=16020): State: BLOCKED Blocked count: 147 Waited count: 0 Blocked on org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader@5231b221 Blocked by 34 (RpcServer.reader=7,bindAddress=0.0.0.0,port=16020) Stack: org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.registerChannel(RpcServer.java:668) org.apache.hadoop.hbase.ipc.RpcServer$Listener.doAccept(RpcServer.java:832) org.apache.hadoop.hbase.ipc.RpcServer$Listener.run(RpcServer.java:741) Thread 38 (RpcServer.responder): State: RUNNABLE Blocked count: 114 Waited count: 164 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.hbase.ipc.RpcServer$Responder.doRunLoop(RpcServer.java:972) org.apache.hadoop.hbase.ipc.RpcServer$Responder.run(RpcServer.java:919) Thread 42 (main-EventThread): State: WAITING Blocked count: 0 Waited count: 3 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@11cdf6f0 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) Thread 41 (main-SendThread(hbasectrl338.frc3.facebook.com:2181)): State: RUNNABLE Blocked count: 1 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) Thread 39 (Thread-5): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method) org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52) org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:459) java.lang.Thread.run(Thread.java:745) Thread 36 (RpcServer.reader=9,bindAddress=0.0.0.0,port=16020): State: WAITING Blocked count: 18 Waited count: 27 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3647e1a5 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:350) org.apache.hadoop.hbase.ipc.RWQueueRpcExecutor.dispatch(RWQueueRpcExecutor.java:173) org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.dispatch(SimpleRpcScheduler.java:201) org.apache.hadoop.hbase.ipc.RpcServer$Connection.processRequest(RpcServer.java:1856) org.apache.hadoop.hbase.ipc.RpcServer$Connection.processOneRpc(RpcServer.java:1753) org.apache.hadoop.hbase.ipc.RpcServer$Connection.process(RpcServer.java:1612) org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1592) org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:856) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:641) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:617) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 35 (RpcServer.reader=8,bindAddress=0.0.0.0,port=16020): State: WAITING Blocked count: 27 Waited count: 16 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e899362 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:350) org.apache.hadoop.hbase.ipc.RWQueueRpcExecutor.dispatch(RWQueueRpcExecutor.java:173) org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.dispatch(SimpleRpcScheduler.java:201) org.apache.hadoop.hbase.ipc.RpcServer$Connection.processRequest(RpcServer.java:1856) org.apache.hadoop.hbase.ipc.RpcServer$Connection.processOneRpc(RpcServer.java:1753) org.apache.hadoop.hbase.ipc.RpcServer$Connection.process(RpcServer.java:1612) org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1592) org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:856) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:641) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:617) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 34 (RpcServer.reader=7,bindAddress=0.0.0.0,port=16020): State: WAITING Blocked count: 26 Waited count: 23 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e899362 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:350) org.apache.hadoop.hbase.ipc.RWQueueRpcExecutor.dispatch(RWQueueRpcExecutor.java:173) org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.dispatch(SimpleRpcScheduler.java:201) org.apache.hadoop.hbase.ipc.RpcServer$Connection.processRequest(RpcServer.java:1856) org.apache.hadoop.hbase.ipc.RpcServer$Connection.processOneRpc(RpcServer.java:1753) org.apache.hadoop.hbase.ipc.RpcServer$Connection.process(RpcServer.java:1612) org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1592) org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:856) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:641) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:617) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 33 (RpcServer.reader=6,bindAddress=0.0.0.0,port=16020): State: WAITING Blocked count: 29 Waited count: 18 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3647e1a5 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:350) org.apache.hadoop.hbase.ipc.RWQueueRpcExecutor.dispatch(RWQueueRpcExecutor.java:173) org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.dispatch(SimpleRpcScheduler.java:201) org.apache.hadoop.hbase.ipc.RpcServer$Connection.processRequest(RpcServer.java:1856) org.apache.hadoop.hbase.ipc.RpcServer$Connection.processOneRpc(RpcServer.java:1753) org.apache.hadoop.hbase.ipc.RpcServer$Connection.process(RpcServer.java:1612) org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1592) org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:856) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:641) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:617) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 32 (RpcServer.reader=5,bindAddress=0.0.0.0,port=16020): State: WAITING Blocked count: 22 Waited count: 18 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e899362 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:350) org.apache.hadoop.hbase.ipc.RWQueueRpcExecutor.dispatch(RWQueueRpcExecutor.java:173) org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.dispatch(SimpleRpcScheduler.java:201) org.apache.hadoop.hbase.ipc.RpcServer$Connection.processRequest(RpcServer.java:1856) org.apache.hadoop.hbase.ipc.RpcServer$Connection.processOneRpc(RpcServer.java:1753) org.apache.hadoop.hbase.ipc.RpcServer$Connection.process(RpcServer.java:1612) org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1592) org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:856) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:641) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:617) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 31 (RpcServer.reader=4,bindAddress=0.0.0.0,port=16020): State: WAITING Blocked count: 24 Waited count: 18 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3647e1a5 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:350) org.apache.hadoop.hbase.ipc.RWQueueRpcExecutor.dispatch(RWQueueRpcExecutor.java:173) org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.dispatch(SimpleRpcScheduler.java:201) org.apache.hadoop.hbase.ipc.RpcServer$Connection.processRequest(RpcServer.java:1856) org.apache.hadoop.hbase.ipc.RpcServer$Connection.processOneRpc(RpcServer.java:1753) org.apache.hadoop.hbase.ipc.RpcServer$Connection.process(RpcServer.java:1612) org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1592) org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:856) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:641) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:617) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 30 (RpcServer.reader=3,bindAddress=0.0.0.0,port=16020): State: WAITING Blocked count: 31 Waited count: 32 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e899362 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:350) org.apache.hadoop.hbase.ipc.RWQueueRpcExecutor.dispatch(RWQueueRpcExecutor.java:173) org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.dispatch(SimpleRpcScheduler.java:201) org.apache.hadoop.hbase.ipc.RpcServer$Connection.processRequest(RpcServer.java:1856) org.apache.hadoop.hbase.ipc.RpcServer$Connection.processOneRpc(RpcServer.java:1753) org.apache.hadoop.hbase.ipc.RpcServer$Connection.process(RpcServer.java:1612) org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1592) org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:856) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:641) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:617) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 29 (RpcServer.reader=2,bindAddress=0.0.0.0,port=16020): State: WAITING Blocked count: 24 Waited count: 26 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3647e1a5 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:350) org.apache.hadoop.hbase.ipc.RWQueueRpcExecutor.dispatch(RWQueueRpcExecutor.java:173) org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.dispatch(SimpleRpcScheduler.java:201) org.apache.hadoop.hbase.ipc.RpcServer$Connection.processRequest(RpcServer.java:1856) org.apache.hadoop.hbase.ipc.RpcServer$Connection.processOneRpc(RpcServer.java:1753) org.apache.hadoop.hbase.ipc.RpcServer$Connection.process(RpcServer.java:1612) org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1592) org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:856) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:641) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:617) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 28 (RpcServer.reader=1,bindAddress=0.0.0.0,port=16020): State: WAITING Blocked count: 25 Waited count: 26 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e899362 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:350) org.apache.hadoop.hbase.ipc.RWQueueRpcExecutor.dispatch(RWQueueRpcExecutor.java:173) org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.dispatch(SimpleRpcScheduler.java:201) org.apache.hadoop.hbase.ipc.RpcServer$Connection.processRequest(RpcServer.java:1856) org.apache.hadoop.hbase.ipc.RpcServer$Connection.processOneRpc(RpcServer.java:1753) org.apache.hadoop.hbase.ipc.RpcServer$Connection.process(RpcServer.java:1612) org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1592) org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:856) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:641) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:617) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 27 (RpcServer.reader=0,bindAddress=0.0.0.0,port=16020): State: WAITING Blocked count: 29 Waited count: 36 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3647e1a5 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:350) org.apache.hadoop.hbase.ipc.RWQueueRpcExecutor.dispatch(RWQueueRpcExecutor.java:173) org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.dispatch(SimpleRpcScheduler.java:201) org.apache.hadoop.hbase.ipc.RpcServer$Connection.processRequest(RpcServer.java:1856) org.apache.hadoop.hbase.ipc.RpcServer$Connection.processOneRpc(RpcServer.java:1753) org.apache.hadoop.hbase.ipc.RpcServer$Connection.process(RpcServer.java:1612) org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1592) org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:856) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:641) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:617) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 5 (Signal Dispatcher): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: Thread 3 (Finalizer): State: WAITING Blocked count: 16660 Waited count: 5781 Waiting on java.lang.ref.ReferenceQueue$Lock@23e84203 Stack: java.lang.Object.wait(Native Method) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164) java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209) Thread 2 (Reference Handler): State: WAITING Blocked count: 2617 Waited count: 771 Waiting on java.lang.ref.Reference$Lock@19932c16 Stack: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157) Thread 1 (main): State: WAITING Blocked count: 9 Waited count: 6 Waiting on java.lang.Thread@17560025 Stack: java.lang.Object.wait(Native Method) java.lang.Thread.join(Thread.java:1245) java.lang.Thread.join(Thread.java:1319) org.apache.hadoop.hbase.util.HasThread.join(HasThread.java:89) org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:66) org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:87) org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126) org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2653) Stacks: =========================================================== RS Configuration: =========================================================== dfs.journalnode.rpc-address0.0.0.0:8485hdfs-default.xml io.storefile.bloom.block.size131072hbase-default.xml yarn.ipc.rpc.classorg.apache.hadoop.yarn.ipc.HadoopYarnProtoRPCyarn-default.xml mapreduce.job.maxtaskfailures.per.tracker3mapred-default.xml yarn.client.max-cached-nodemanagers-proxies0yarn-default.xml ha.zookeeper.parent-node/hadoop-hacore-site.xml hbase.rest.threads.min2hbase-default.xml hbase.rs.cacheblocksonwritefalsehbase-default.xml ha.health-monitor.connect-retry-interval.ms1000core-default.xml hbase.regionserver.global.memstore.size.min.range0.1hbase-site.xml yarn.resourcemanager.work-preserving-recovery.enabledfalseyarn-default.xml dfs.client.mmap.cache.size256hdfs-default.xml mapreduce.reduce.markreset.buffer.percent0.0mapred-default.xml dfs.datanode.data.dirfile:///mnt/d0/FBTRACE004-DFS/,file:///mnt/d1/FBTRACE004-DFS/,file:///mnt/d2/FBTRACE004-DFS/,file:///mnt/d3/FBTRACE004-DFS/,file:///mnt/d4/FBTRACE004-DFS/,file:///mnt/d5/FBTRACE004-DFS/,file:///mnt/d6/FBTRACE004-DFS/,file:///mnt/d7/FBTRACE004-DFS/,file:///mnt/d8/FBTRACE004-DFS/,file:///mnt/d9/FBTRACE004-DFS/,file:///mnt/d10/FBTRACE004-DFS/,file:///mnt/d11/FBTRACE004-DFS/,file:///mnt/d12/FBTRACE004-DFS/,file:///mnt/d13/FBTRACE004-DFS/,file:///mnt/d14/FBTRACE004-DFS/hdfs-site.xml mapreduce.jobhistory.max-age-ms604800000mapred-default.xml dfs.namenode.lazypersist.file.scrub.interval.sec300hdfs-default.xml mapreduce.job.ubertask.enablefalsemapred-default.xml dfs.namenode.delegation.token.renew-interval86400000hdfs-default.xml yarn.nodemanager.log-aggregation.compression-typenoneyarn-default.xml dfs.namenode.replication.considerLoadtruehdfs-default.xml mapreduce.job.complete.cancel.delegation.tokenstruemapred-default.xml mapreduce.jobhistory.datestring.cache.size200000mapred-default.xml hadoop.security.kms.client.authentication.retry-count1core-default.xml hadoop.ssl.enabled.protocolsTLSv1core-default.xml hbase.status.multicast.address.ip226.1.1.3hbase-default.xml dfs.namenode.retrycache.heap.percent0.03fhdfs-default.xml yarn.resourcemanager.scheduler.address${yarn.resourcemanager.hostname}:8030yarn-default.xml dfs.namenode.logging.levelinfohdfs-default.xml dfs.client.file-block-storage-locations.num-threads10hdfs-default.xml dfs.datanode.balance.bandwidthPerSec1048576hdfs-default.xml yarn.resourcemanager.proxy-user-privileges.enabledfalseyarn-default.xml mapreduce.reduce.shuffle.fetch.retry.enabled${yarn.nodemanager.recovery.enabled}mapred-default.xml io.mapfile.bloom.error.rate0.005core-default.xml yarn.nodemanager.resourcemanager.minimum.versionNONEyarn-default.xml yarn.resourcemanager.nodemanagers.heartbeat-interval-ms1000yarn-default.xml dfs.secondary.namenode.kerberos.internal.spnego.principal${dfs.web.authentication.kerberos.principal}hdfs-default.xml yarn.nodemanager.delete.debug-delay-sec0yarn-default.xml dfs.client.read.shortcircuit.streams.cache.size256hdfs-default.xml dfs.image.transfer.bandwidthPerSec0hdfs-default.xml yarn.scheduler.maximum-allocation-vcores32yarn-default.xml hfile.block.bloom.cacheonwritefalsehbase-default.xml hbase.zookeeper.quorumhbasectrl338.frc3.facebook.com,hbasectrl339.frc3.facebook.com,hbasectrl340.frc3.facebook.com,hbasectrl341.frc3.facebook.com,hbasectrl342.frc3.facebook.comhbase-site.xml hbase.http.staticuser.userdr.stackhbase-default.xml yarn.timeline-service.address${yarn.timeline-service.hostname}:10200yarn-default.xml yarn.nodemanager.disk-health-checker.min-free-space-per-disk-mb0yarn-default.xml mapreduce.job.hdfs-servers${fs.defaultFS}yarn-default.xml mapreduce.task.profile.reduce.params${mapreduce.task.profile.params}mapred-default.xml hbase.zookeeper.property.syncLimit5hbase-default.xml dfs.namenode.fs-limits.min-block-size1048576hdfs-default.xml ftp.stream-buffer-size4096core-default.xml dfs.client.use.legacy.blockreader.localfalsehdfs-default.xml dfs.short.circuit.shared.memory.watcher.interrupt.check.ms60000hdfs-default.xml dfs.datanode.directoryscan.threads1hdfs-default.xml fs.s3a.buffer.dir${hadoop.tmp.dir}/s3acore-default.xml yarn.client.application-client-protocol.poll-interval-ms200yarn-default.xml yarn.timeline-service.leveldb-timeline-store.path/hay/yarn/timelineyarn-site.xml mapreduce.job.split.metainfo.maxsize10000000mapred-default.xml dfs.namenode.edits.noeditlogchannelflushfalsehdfs-default.xml s3native.bytes-per-checksum512core-default.xml hbase.rest.filter.classesorg.apache.hadoop.hbase.rest.filter.GzipFilterhbase-default.xml yarn.client.failover-retries-on-socket-timeouts0yarn-default.xml dfs.namenode.startup.delay.block.deletion.sec0hdfs-default.xml dfs.webhdfs.user.provider.user.pattern^[A-Za-z_][A-Za-z0-9._-]*[$]?$hdfs-default.xml mapreduce.tasktracker.tasks.sleeptimebeforesigkill5000mapred-default.xml yarn.timeline-service.client.retry-interval-ms1000yarn-default.xml dfs.encrypt.data.transfer.cipher.key.bitlength128hdfs-default.xml yarn.resourcemanager.webapp.https.address.rm4hbasectrl342.frc3.facebook.com:8090yarn-site.xml yarn.resourcemanager.webapp.https.address.rm3hbasectrl341.frc3.facebook.com:8090yarn-site.xml yarn.resourcemanager.webapp.https.address.rm2hbasectrl340.frc3.facebook.com:8090yarn-site.xml yarn.resourcemanager.webapp.https.address.rm1hbasectrl339.frc3.facebook.com:8090yarn-site.xml yarn.resourcemanager.webapp.https.address.rm0hbasectrl338.frc3.facebook.com:8090yarn-site.xml hadoop.http.authentication.typesimplecore-default.xml dfs.namenode.path.based.cache.refresh.interval.ms30000hdfs-default.xml mapreduce.local.clientfactory.class.nameorg.apache.hadoop.mapred.LocalClientFactorymapred-default.xml dfs.datanode.cache.revocation.timeout.ms900000hdfs-default.xml ipc.client.connection.maxidletime10000core-default.xml dfs.namenode.safemode.threshold-pct0.999fhdfs-default.xml hfile.block.cache.size0.4hbase-site.xml fs.s3a.multipart.purge.age86400core-default.xml dfs.namenode.num.checkpoints.retained2hdfs-default.xml hbase.hregion.memstore.mslab.enabledtruehbase-default.xml yarn.timeline-service.bind-host0.0.0.0yarn-site.xml mapreduce.job.ubertask.maxmaps9mapred-default.xml dfs.namenode.stale.datanode.interval30000hdfs-default.xml yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage90.0yarn-default.xml mapreduce.tasktracker.http.address0.0.0.0:50060mapred-default.xml mapreduce.ifile.readahead.bytes4194304mapred-default.xml mapreduce.jobhistory.admin.addresshbasectrl342.frc3.facebook.com:10033mapred-site.xml s3.client-write-packet-size65536core-default.xml hbase.master.port16000hbase-default.xml dfs.block.access.token.lifetime600hdfs-default.xml yarn.app.mapreduce.am.resource.cpu-vcores1mapred-default.xml mapreduce.input.lineinputformat.linespermap1mapred-default.xml hbase.regionserver.checksum.verifytruehbase-default.xml dfs.namenode.num.extra.edits.retained1000000hdfs-default.xml hbase.security.visibility.mutations.checkauthsfalsehbase-default.xml mapreduce.reduce.shuffle.input.buffer.percent0.25mapred-site.xml hadoop.http.staticuser.userdr.whocore-default.xml mapreduce.reduce.maxattempts4mapred-default.xml hadoop.security.group.mapping.ldap.search.filter.user(&(objectClass=user)(sAMAccountName={0}))core-default.xml mapreduce.jobhistory.admin.acl*mapred-default.xml dfs.client.contextdefaulthdfs-default.xml mapreduce.map.maxattempts4mapred-default.xml yarn.resourcemanager.zk-retry-interval-ms1000yarn-default.xml mapreduce.jobhistory.cleaner.interval-ms86400000mapred-default.xml dfs.datanode.drop.cache.behind.readsfalsehdfs-default.xml hbase.server.versionfile.writeattempts3hbase-default.xml dfs.permissions.superusergrouproothdfs-site.xml hbase.zookeeper.useMultiTruehbase-site.xml fs.s3n.block.size67108864core-default.xml hadoop.registry.system.aclssasl:yarn@, sasl:mapred@, sasl:mapred@hdfs@yarn-default.xml dfs.namenode.list.cache.pools.num.responses100hdfs-default.xml yarn.resourcemanager.resource-tracker.address.rm4hbasectrl342.frc3.facebook.com:8031yarn-site.xml yarn.resourcemanager.resource-tracker.address.rm3hbasectrl341.frc3.facebook.com:8031yarn-site.xml yarn.resourcemanager.resource-tracker.address.rm2hbasectrl340.frc3.facebook.com:8031yarn-site.xml yarn.resourcemanager.resource-tracker.address.rm1hbasectrl339.frc3.facebook.com:8031yarn-site.xml hbase.zookeeper.leaderport3888hbase-default.xml yarn.resourcemanager.resource-tracker.address.rm0hbasectrl338.frc3.facebook.com:8031yarn-site.xml dfs.datanode.slow.io.warning.threshold.ms300hdfs-default.xml hbase.master.info.port16010programatically yarn.nodemanager.bind-host0.0.0.0yarn-site.xml dfs.namenode.fs-limits.max-blocks-per-file1048576hdfs-default.xml yarn.nodemanager.vmem-check-enabledtrueyarn-default.xml hadoop.security.authenticationsimplecore-default.xml mapreduce.reduce.cpu.vcores1mapred-default.xml net.topology.node.switch.mapping.implorg.apache.hadoop.net.ScriptBasedMappingcore-default.xml fs.s3.sleepTimeSeconds10core-default.xml yarn.timeline-service.ttl-ms604800000yarn-default.xml yarn.resourcemanager.keytab/etc/krb5.keytabyarn-default.xml yarn.resourcemanager.container.liveness-monitor.interval-ms600000yarn-default.xml mapreduce.jobtracker.heartbeats.in.second100mapred-default.xml yarn.app.mapreduce.am.scheduler.heartbeat.interval-ms1000mapred-default.xml yarn.app.mapreduce.client-am.ipc.max-retries-on-timeouts3mapred-default.xml yarn.nodemanager.linux-container-executor.cgroups.hierarchy/hadoop-yarnyarn-default.xml s3.bytes-per-checksum512core-default.xml hbase.regionserver.dns.nameserverdefaulthbase-default.xml hadoop.ssl.require.client.certfalsecore-default.xml yarn.resourcemanager.ha.rm-idsrm0,rm1,rm2,rm3,rm4yarn-site.xml dfs.journalnode.http-address0.0.0.0:8480hdfs-default.xml mapreduce.output.fileoutputformat.compressfalsemapred-default.xml dfs.ha.automatic-failover.enabledTruehdfs-site.xml hbase.ipc.server.callqueue.read.ratio0.3hbase-site.xml hbase.cluster.distributedTruehbase-site.xml hbase.rootdirhdfs://fbtrace004/FBTRACE004-HBASE/programatically yarn.resourcemanager.webapp.delegation-token-auth-filter.enabledtrueyarn-default.xml mapreduce.shuffle.max.threads0mapred-default.xml dfs.namenode.invalidate.work.pct.per.iteration0.32fhdfs-default.xml s3native.client-write-packet-size65536core-default.xml dfs.client.block.write.replace-datanode-on-failure.policyDEFAULThdfs-default.xml mapreduce.client.submit.file.replication10mapred-default.xml yarn.app.mapreduce.am.job.committer.commit-window10000mapred-default.xml yarn.nodemanager.sleep-delay-before-sigkill.ms250yarn-default.xml yarn.nodemanager.env-whitelistJAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,HADOOP_YARN_HOMEyarn-default.xml dfs.namenode.acls.enabledfalsehdfs-default.xml dfs.namenode.secondary.http-address0.0.0.0:50090hdfs-default.xml mapreduce.map.speculativeFalsemapred-site.xml mapreduce.job.speculative.slowtaskthreshold1.0mapred-default.xml mapreduce.task.tmp.dir./tmpmapred-default.xml yarn.nodemanager.linux-container-executor.cgroups.mountfalseyarn-default.xml hbase.auth.token.max.lifetime604800000hbase-default.xml hbase.regionserver.msginterval3000hbase-default.xml mapreduce.tasktracker.http.threads40mapred-default.xml mapreduce.jobhistory.http.policyHTTP_ONLYmapred-default.xml hbase.ipc.client.fallback-to-simple-auth-allowedfalsehbase-default.xml fs.s3a.paging.maximum5000core-default.xml hbase.rest.threads.max100hbase-default.xml fs.s3.buffer.dir${hadoop.tmp.dir}/s3core-default.xml hbase.snapshot.enabledtruehbase-default.xml hbase.dynamic.jars.dir${hbase.rootdir}/libhbase-default.xml hbase.defaults.for.version1.2.0-SNAPSHOThbase-default.xml io.native.lib.availabletruecore-default.xml mapreduce.jobhistory.done-dir${yarn.app.mapreduce.am.staging-dir}/history/donemapred-default.xml hbase.regions.slop0.2hbase-default.xml hadoop.registry.zk.retry.interval.ms1000yarn-default.xml dfs.namenode.avoid.write.stale.datanodeTruehdfs-site.xml dfs.namenode.checkpoint.txns1000000hdfs-default.xml hadoop.ssl.hostname.verifierDEFAULTcore-default.xml zookeeper.znode.rootserverroot-region-serverhbase-default.xml mapreduce.task.timeout600000mapred-default.xml hbase.client.max.perserver.tasks5hbase-default.xml yarn.nodemanager.disk-health-checker.interval-ms120000yarn-default.xml dfs.namenode.replication.max-streams-hard-limit1hdfs-site.xml dfs.journalnode.https-address0.0.0.0:8481hdfs-default.xml hadoop.security.groups.cache.secs300core-default.xml mapreduce.input.fileinputformat.split.minsize0mapred-default.xml dfs.datanode.sync.behind.writesTruehdfs-site.xml zookeeper.session.timeout90000hbase-default.xml ipc.server.tcpnodelayTruecore-site.xml mapreduce.shuffle.port13562mapred-default.xml hadoop.rpc.protectionauthenticationcore-default.xml dfs.client.https.keystore.resourcessl-client.xmlhdfs-default.xml dfs.namenode.list.encryption.zones.num.responses100hdfs-default.xml yarn.client.failover-proxy-providerorg.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvideryarn-site.xml mapreduce.jobtracker.retiredjobs.cache.size1000mapred-default.xml hbase.balancer.period300000hbase-default.xml ipc.client.tcpnodelayTruecore-site.xml dfs.ha.tail-edits.period60hdfs-default.xml dfs.datanode.drop.cache.behind.writesfalsehdfs-default.xml fs.s3.maxRetries4core-default.xml mapreduce.jobtracker.addresslocalmapred-default.xml hadoop.http.authentication.kerberos.principalHTTP/_HOST@LOCALHOSTcore-default.xml nfs.server.port2049hdfs-default.xml yarn.resourcemanager.webapp.address${yarn.resourcemanager.hostname}:8088yarn-default.xml mapreduce.task.profile.reduces0-2mapred-default.xml yarn.timeline-service.client.max-retries30yarn-default.xml yarn.resourcemanager.am.max-attempts2yarn-default.xml hbase.normalizer.enabledfalsehbase-default.xml hbase.hstore.blockingWaitTime90000hbase-default.xml nfs.dump.dir/tmp/.hdfs-nfshdfs-default.xml hbase.client.pause100hbase-default.xml hbase.client.write.buffer2097152hbase-default.xml dfs.bytes-per-checksum512hdfs-default.xml mapreduce.job.end-notification.max.retry.interval5000mapred-default.xml ipc.client.connect.retry.interval1000core-default.xml fs.s3a.multipart.size104857600core-default.xml yarn.app.mapreduce.am.command-opts-server -Xmx3g -XX:+UseG1GC -XX:MaxGCPauseMillis=150yarn-site.xml yarn.nodemanager.process-kill-wait.ms2000yarn-default.xml hbase.rpc.timeout60000hbase-default.xml hbase.metrics.exposeOperationTimestruehbase-default.xml dfs.namenode.safemode.min.datanodes0hdfs-default.xml hbase.thrift.maxWorkerThreads1000hbase-default.xml dfs.namenode.write.stale.datanode.ratio0.5fhdfs-default.xml hadoop.jetty.logs.serve.aliasestruecore-default.xml mapreduce.reduce.shuffle.fetch.retry.timeout-ms30000mapred-default.xml fs.du.interval600000core-default.xml mapreduce.tasktracker.dns.nameserverdefaultmapred-default.xml hbase.master.catalog.timeout600000hbase-default.xml hbase.regionserver.global.memstore.size.max.range0.7hbase-site.xml hfile.block.cache.size.max.range0.7hbase-site.xml hadoop.security.random.device.file.path/dev/urandomcore-default.xml mapreduce.task.merge.progress.records10000mapred-default.xml hbase.region.replica.replication.enabledfalsehbase-default.xml dfs.webhdfs.enabledtruehdfs-default.xml hadoop.registry.securefalseyarn-default.xml hadoop.ssl.client.confssl-client.xmlcore-default.xml mapreduce.job.counters.max120mapred-default.xml yarn.nodemanager.localizer.fetch.thread-count4yarn-default.xml io.mapfile.bloom.size1048576core-default.xml yarn.nodemanager.localizer.client.thread-count5yarn-default.xml fs.automatic.closetruecore-default.xml mapreduce.task.profilefalsemapred-default.xml dfs.namenode.edit.log.autoroll.multiplier.threshold2.0hdfs-default.xml mapreduce.task.combine.progress.records10000mapred-default.xml mapreduce.shuffle.ssl.file.buffer.size65536mapred-default.xml fs.swift.implorg.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemcore-default.xml yarn.app.mapreduce.am.container.log.backups0mapred-default.xml hbase.hstore.bytes.per.checksum16384hbase-default.xml dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction0.75fhdfs-default.xml hbase.hstore.flusher.count2hbase-default.xml dfs.namenode.backup.address0.0.0.0:50100hdfs-default.xml dfs.client.https.need-authfalsehdfs-default.xml mapreduce.app-submission.cross-platformfalsemapred-default.xml yarn.timeline-service.ttl-enabletrueyarn-default.xml dfs.user.home.dir.prefix/userhdfs-default.xml yarn.nodemanager.container-monitor.procfs-tree.smaps-based-rss.enabledfalseyarn-default.xml yarn.nodemanager.keytab/etc/krb5.keytabyarn-default.xml dfs.namenode.xattrs.enabledtruehdfs-default.xml dfs.client.write.exclude.nodes.cache.expiry.interval.millis600000hdfs-default.xml mapreduce.jobtracker.restart.recoverfalsemapred-default.xml mapreduce.map.skip.proc.count.autoincrtruemapred-default.xml dfs.namenode.datanode.registration.ip-hostname-checktruehdfs-default.xml dfs.image.transfer.chunksize65536hdfs-default.xml hadoop.security.instrumentation.requires.adminfalsecore-default.xml io.compression.codec.bzip2.librarysystem-nativecore-default.xml dfs.namenode.name.dir.restoreTruehdfs-site.xml hbase.client.retries.number350programatically dfs.namenode.resource.checked.volumes.minimum1hdfs-default.xml hadoop.ssl.keystores.factory.classorg.apache.hadoop.security.ssl.FileBasedKeyStoresFactorycore-default.xml dfs.namenode.list.cache.directives.num.responses100hdfs-default.xml hbase.status.multicast.address.port16100hbase-default.xml fs.ftp.host0.0.0.0core-default.xml hbase.hstore.checksum.algorithmCRC32Chbase-default.xml s3.blocksize67108864core-default.xml s3native.stream-buffer-size4096core-default.xml mapreduce.jobtracker.taskschedulerorg.apache.hadoop.mapred.JobQueueTaskSchedulermapred-default.xml dfs.datanode.dns.nameserverdefaulthdfs-default.xml yarn.nodemanager.resource.memory-mb10736yarn-site.xml mapreduce.task.userlog.limit.kb0mapred-default.xml hadoop.security.crypto.codec.classes.aes.ctr.nopaddingorg.apache.hadoop.crypto.OpensslAesCtrCryptoCodec,org.apache.hadoop.crypto.JceAesCtrCryptoCodeccore-default.xml mapreduce.reduce.speculativeFalsemapred-site.xml yarn.resourcemanager.admin.address.rm4hbasectrl342.frc3.facebook.com:8033yarn-site.xml yarn.nodemanager.container-monitor.interval-ms3000yarn-default.xml yarn.resourcemanager.admin.address.rm3hbasectrl341.frc3.facebook.com:8033yarn-site.xml yarn.resourcemanager.admin.address.rm2hbasectrl340.frc3.facebook.com:8033yarn-site.xml yarn.resourcemanager.admin.address.rm1hbasectrl339.frc3.facebook.com:8033yarn-site.xml yarn.resourcemanager.admin.address.rm0hbasectrl338.frc3.facebook.com:8033yarn-site.xml net.topology.script.file.name/opt/hbase/bin/locationforcore-site.xml dfs.replication.max512hdfs-default.xml dfs.replication3hdfs-default.xml dfs.ha.fencing.methodsshell(/bin/true)core-site.xml yarn.client.failover-retries0yarn-default.xml yarn.nodemanager.resource.cpu-vcores8yarn-default.xml mapreduce.jobhistory.recovery.enableTruemapred-site.xml hbase.server.thread.wakefrequency10000hbase-default.xml nfs.exports.allowed.hosts* rwcore-default.xml hbase.lease.recovery.timeout900000hbase-default.xml hbase.coordinated.state.manager.classorg.apache.hadoop.hbase.coordination.ZkCoordinatedStateManagerhbase-default.xml mapreduce.reduce.shuffle.memory.limit.percent0.25mapred-default.xml file.replication3core-site.xml mapreduce.job.reduce.shuffle.consumer.plugin.classorg.apache.hadoop.mapreduce.task.reduce.Shufflemapred-default.xml hfile.format.version3hbase-default.xml mapreduce.job.jvm.numtasks1mapred-default.xml dfs.datanode.fsdatasetcache.max.threads.per.volume4hdfs-default.xml mapreduce.am.max-attempts2mapred-default.xml mapreduce.shuffle.connection-keep-alive.timeout5mapred-default.xml hadoop.fuse.timer.period5hdfs-default.xml mapreduce.job.reduces50mapred-site.xml hbase.thrift.minWorkerThreads16hbase-default.xml hbase.zookeeper.dns.interfacedefaulthbase-default.xml yarn.app.mapreduce.am.job.task.listener.thread-count30mapred-default.xml yarn.resourcemanager.store.classorg.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStoreyarn-site.xml mapreduce.job.speculative.slownodethreshold1.0mapred-default.xml s3native.replication3core-default.xml mapreduce.tasktracker.reduce.tasks.maximum2mapred-default.xml hbase.snapshot.restore.failsafe.namehbase-failsafe-{snapshot.name}-{restore.timestamp}hbase-default.xml fs.permissions.umask-mode022core-default.xml mapreduce.cluster.local.dir${hadoop.tmp.dir}/mapred/localmapred-default.xml mapreduce.client.output.filterFAILEDmapred-default.xml yarn.nodemanager.pmem-check-enabledtrueyarn-default.xml dfs.client.failover.connection.retries.on.timeouts0hdfs-default.xml mapreduce.jobtracker.instrumentationorg.apache.hadoop.mapred.JobTrackerMetricsInstmapred-default.xml ftp.replication3core-default.xml hbase.hstore.blockingStoreFiles25hbase-site.xml hadoop.security.group.mapping.ldap.search.attr.membermembercore-default.xml hbase.regionserver.hlog.reader.implorg.apache.hadoop.hbase.regionserver.wal.ProtobufLogReaderhbase-default.xml dfs.namenode.replication.work.multiplier.per.iteration2hdfs-default.xml yarn.resourcemanager.resource-tracker.address${yarn.resourcemanager.hostname}:8031yarn-default.xml mapreduce.tasktracker.outofband.heartbeatfalsemapred-default.xml hbase.master.info.bindAddress0.0.0.0hbase-default.xml dfs.namenode.edits.dir/hay/dfs/hdfs-site.xml yarn.resourcemanager.scheduler.monitor.enablefalseyarn-default.xml fs.trash.checkpoint.interval0core-default.xml hadoop.registry.zk.retry.times5yarn-default.xml dfs.client.read.shortcircuit.streams.cache.expiry.ms300000hdfs-default.xml yarn.timeline-service.leveldb-timeline-store.start-time-write-cache-size10000yarn-default.xml s3.stream-buffer-size4096core-default.xml fs.s3a.connection.maximum15core-default.xml file.client-write-packet-size65536core-default.xml mapreduce.tasktracker.healthchecker.script.timeout600000mapred-default.xml hbase.status.listener.classorg.apache.hadoop.hbase.client.ClusterStatusListener$MulticastListenerhbase-default.xml dfs.namenode.fs-limits.max-directory-items1048576hdfs-default.xml mapreduce.tasktracker.taskcontrollerorg.apache.hadoop.mapred.DefaultTaskControllermapred-default.xml dfs.namenode.path.based.cache.block.map.allocation.percent0.25hdfs-default.xml fs.s3a.implorg.apache.hadoop.fs.s3a.S3AFileSystemcore-default.xml dfs.namenode.checkpoint.dirfile://${hadoop.tmp.dir}/dfs/namesecondaryhdfs-default.xml hbase.regionserver.dns.interfacedefaulthbase-default.xml yarn.nodemanager.remote-app-log-dir${fs.defaultFS}/yarn-logsyarn-site.xml mapreduce.reduce.shuffle.retry-delay.max.ms60000mapred-default.xml io.map.index.interval128core-default.xml dfs.namenode.replication.interval3hdfs-default.xml dfs.client.block.write.replace-datanode-on-failure.enabletruehdfs-default.xml hbase.rest.port8080hbase-default.xml hbase.regionserver.handler.count30hbase-default.xml hadoop.ssl.server.confssl-server.xmlcore-default.xml dfs.nameservicesfbtrace004hdfs-site.xml hadoop.rpc.socket.factory.class.defaultorg.apache.hadoop.net.StandardSocketFactorycore-default.xml yarn.app.mapreduce.client.max-retries3mapred-default.xml yarn.nodemanager.address${yarn.nodemanager.hostname}:8140yarn-site.xml hbase.ipc.server.callqueue.scan.ratio0.2hbase-site.xml dfs.datanode.max.transfer.threads4096hdfs-default.xml ha.failover-controller.graceful-fence.rpc-timeout.ms5000core-default.xml dfs.datanode.ipc.address0.0.0.0:50020hdfs-default.xml yarn.resourcemanager.delayed.delegation-token.removal-interval-ms30000yarn-default.xml yarn.timeline-service.enabledfalseyarn-default.xml dfs.client.cached.conn.retry3hdfs-default.xml dfs.namenode.backup.http-address0.0.0.0:50105hdfs-default.xml mapreduce.tasktracker.report.address127.0.0.1:0mapred-default.xml hbase.bulkload.retries.number10hbase-default.xml dfs.namenode.checkpoint.period3600hdfs-default.xml mapreduce.task.attempt.idhb_rs_hbase4537.frc3.facebook.com,16020,1440656248801programatically hbase.hregion.max.filesize343597383680hbase-site.xml dfs.datanode.shared.file.descriptor.paths/dev/shm,/tmphdfs-default.xml hbase.master.loadbalancer.classorg.apache.hadoop.hbase.master.balancer.StochasticLoadBalancerhbase-default.xml dfs.http.policyHTTP_ONLYhdfs-default.xml hadoop.security.groups.cache.warn.after.ms5000core-default.xml dfs.namenode.fs-limits.max-xattrs-per-inode32hdfs-default.xml yarn.resourcemanager.zk-aclworld:anyone:rwcdayarn-default.xml mapreduce.job.speculative.speculativecap0.1mapred-default.xml dfs.namenode.support.allow.formatFalsehdfs-site.xml dfs.namenode.checkpoint.max-retries3hdfs-default.xml zookeeper.znode.acl.parentaclhbase-default.xml hbase.status.publisher.classorg.apache.hadoop.hbase.master.ClusterStatusPublisher$MulticastPublisherhbase-default.xml yarn.resourcemanager.fs.state-store.retry-policy-spec2000, 500yarn-default.xml hbase.tmp.dir${java.io.tmpdir}/hbase-${user.name}hbase-default.xml dfs.namenode.decommission.nodes.per.interval5hdfs-default.xml mapreduce.job.committer.setup.cleanup.neededtruemapred-default.xml dfs.datanode.cache.revocation.polling.ms500hdfs-default.xml mapreduce.job.end-notification.retry.attempts0mapred-default.xml yarn.resourcemanager.state-store.max-completed-applications${yarn.resourcemanager.max-completed-applications}yarn-default.xml hbase.http.max.threads10hbase-default.xml mapreduce.map.output.compressTruemapred-site.xml hbase.client.localityCheck.threadPoolSize2hbase-default.xml mapreduce.jobhistory.cleaner.enabletruemapred-default.xml dfs.namenode.replication.max-streams1hdfs-site.xml io.seqfile.local.dir${hadoop.tmp.dir}/io/localcore-default.xml dfs.blockreport.split.threshold1000000hdfs-default.xml mapreduce.reduce.shuffle.read.timeout180000mapred-default.xml mapreduce.job.queuenamedefaultmapred-default.xml ipc.client.connect.max.retries10core-default.xml io.seqfile.lazydecompresstruecore-default.xml yarn.app.mapreduce.am.staging-dir/tmp/hadoop-yarn/stagingmapred-default.xml yarn.nodemanager.linux-container-executor.resources-handler.classorg.apache.hadoop.yarn.server.nodemanager.util.DefaultLCEResourcesHandleryarn-default.xml yarn.timeline-service.leveldb-timeline-store.read-cache-size104857600yarn-default.xml io.file.buffer.size65536core-site.xml yarn.resourcemanager.am-rm-tokens.master-key-rolling-interval-secs86400yarn-default.xml ha.zookeeper.parent-znode/hadoop-hacore-default.xml dfs.namenode.rpc-address.fbtrace004.nn1hbasectrl339.frc3.facebook.com:9000hdfs-site.xml mapreduce.tasktracker.indexcache.mb10mapred-default.xml dfs.namenode.rpc-address.fbtrace004.nn0hbasectrl338.frc3.facebook.com:9000hdfs-site.xml tfile.io.chunk.size1048576core-default.xml yarn.resourcemanager.work-preserving-recovery.scheduling-wait-ms10000yarn-default.xml yarn.timeline-service.keytab/etc/krb5.keytabyarn-default.xml yarn.acl.enablefalseyarn-default.xml hbase.regionserver.regionSplitLimit1000hbase-default.xml hadoop.security.group.mapping.ldap.directory.search.timeout10000core-default.xml mapreduce.job.token.tracking.ids.enabledfalsemapred-default.xml hbase.thrift.maxQueuedRequests1000hbase-default.xml mapreduce.map.output.compress.codecorg.apache.hadoop.io.compress.Lz4Codecmapred-site.xml s3.replication3core-default.xml hadoop.registry.zk.root/registryyarn-default.xml tfile.fs.input.buffer.size262144core-default.xml yarn.timeline-service.http-authentication.typesimpleyarn-default.xml ha.failover-controller.graceful-fence.connection.retries1core-default.xml hbase.master.balancer.stochastic.localityCost0hbase-site.xml net.topology.script.number.args100core-default.xml fs.s3n.multipart.uploads.block.size67108864core-default.xml hfile.block.index.cacheonwritefalsehbase-default.xml yarn.nodemanager.recovery.dir/mnt/d3/FBTRACE004-YARN/recoveryyarn-site.xml hadoop.ssl.enabledfalsecore-default.xml yarn.timeline-service.handler-thread-count10yarn-default.xml hbase.config.read.zookeeper.configfalsehbase-default.xml hbase.column.max.version1hbase-default.xml dfs.client.read.shortcircuit.buffer.size131072programatically dfs.namenode.reject-unresolved-dn-topology-mappingfalsehdfs-default.xml mapreduce.jobhistory.recovery.store.classorg.apache.hadoop.mapreduce.v2.hs.HistoryServerFileSystemStateStoreServicemapred-default.xml yarn.nodemanager.log.retain-seconds10800yarn-default.xml yarn.resourcemanager.admin.address${yarn.resourcemanager.hostname}:8033yarn-default.xml yarn.resourcemanager.recovery.enabledfalseyarn-default.xml dfs.client.slow.io.warning.threshold.ms30000hdfs-default.xml yarn.resourcemanager.ha.automatic-failover.zk-base-path/yarn-leader-electionyarn-default.xml fs.AbstractFileSystem.viewfs.implorg.apache.hadoop.fs.viewfs.ViewFscore-default.xml mapreduce.tasktracker.dns.interfacedefaultmapred-default.xml mapreduce.jobtracker.handler.count10mapred-default.xml dfs.blockreport.initialDelay0hdfs-default.xml fs.AbstractFileSystem.hdfs.implorg.apache.hadoop.fs.Hdfscore-default.xml dfs.namenode.retrycache.expirytime.millis600000hdfs-default.xml dfs.client.failover.sleep.max.millis15000hdfs-default.xml hbase.regionserver.thread.compaction.large5hbase-site.xml hbase.zookeeper.property.clientPort2181hbase-default.xml yarn.resourcemanager.max-completed-applications10000yarn-default.xml yarn.nodemanager.log-dirs/mnt/d0/FBTRACE004-YARN/logs,/mnt/d1/FBTRACE004-YARN/logs,/mnt/d2/FBTRACE004-YARN/logs,/mnt/d3/FBTRACE004-YARN/logs,/mnt/d4/FBTRACE004-YARN/logs,/mnt/d5/FBTRACE004-YARN/logs,/mnt/d6/FBTRACE004-YARN/logs,/mnt/d7/FBTRACE004-YARN/logs,/mnt/d8/FBTRACE004-YARN/logs,/mnt/d9/FBTRACE004-YARN/logs,/mnt/d10/FBTRACE004-YARN/logs,/mnt/d11/FBTRACE004-YARN/logs,/mnt/d12/FBTRACE004-YARN/logs,/mnt/d13/FBTRACE004-YARN/logs,/mnt/d14/FBTRACE004-YARN/logsyarn-site.xml dfs.client.failover.sleep.base.millis500hdfs-default.xml yarn.nodemanager.linux-container-executor.nonsecure-mode.user-pattern^[_.A-Za-z0-9][-@_.A-Za-z0-9]{0,255}?[$]?$yarn-default.xml hbase.rest.readonlyfalsehbase-default.xml dfs.hosts.exclude/hay/excludeshdfs-site.xml dfs.default.chunk.view.size32768hdfs-default.xml dfs.client.read.shortcircuitfalsehdfs-default.xml ftp.blocksize67108864core-default.xml mapreduce.job.acl-modify-job mapred-default.xml zookeeper.znode.parent/hbasehbase-default.xml fs.defaultFShdfs://fbtrace004programatically hbase.rpc.shortoperation.timeout10000hbase-default.xml hadoop.http.filter.initializersorg.apache.hadoop.http.lib.StaticUserWebFiltercore-default.xml fs.s3n.multipart.copy.block.size5368709120core-default.xml mapreduce.map.java.opts-server -Xms2g -Xmx3g -XX:+UseG1GC -XX:MaxGCPauseMillis=500mapred-site.xml yarn.resourcemanager.connect.max-wait.ms900000yarn-default.xml dfs.datanode.synconcloseTruehdfs-site.xml hadoop.security.group.mapping.ldap.sslfalsecore-default.xml dfs.namenode.max.extra.edits.segments.retained10000hdfs-default.xml dfs.namenode.https-address0.0.0.0:50470hdfs-default.xml yarn.nodemanager.aux-servicesmapreduce_shuffleyarn-site.xml yarn.resourcemanager.bind-host0.0.0.0yarn-site.xml yarn.resourcemanager.admin.client.thread-count1yarn-default.xml hadoop.security.kms.client.encrypted.key.cache.size500core-default.xml ipc.client.kill.max10core-default.xml hadoop.security.group.mapping.ldap.search.filter.group(objectClass=group)core-default.xml fs.AbstractFileSystem.file.implorg.apache.hadoop.fs.local.LocalFscore-default.xml hadoop.http.authentication.kerberos.keytab${user.home}/hadoop.keytabcore-default.xml yarn.client.nodemanager-connect.max-wait-ms900000yarn-default.xml mapreduce.job.map.output.collector.classorg.apache.hadoop.mapred.MapTask$MapOutputBuffermapred-default.xml dfs.namenode.path.based.cache.retry.interval.ms30000hdfs-default.xml yarn.log.dir/logsyarn-site.xml hadoop.security.uid.cache.secs14400core-default.xml mapreduce.map.cpu.vcores1mapred-default.xml yarn.log-aggregation.retain-check-interval-seconds-1yarn-default.xml mapreduce.map.log.levelINFOmapred-default.xml mapred.child.java.opts-Xmx200mmapred-default.xml hfile.index.block.max.size131072hbase-default.xml hbase.client.scanner.timeout.period60000hbase-default.xml hadoop.registry.zk.session.timeout.ms60000yarn-default.xml yarn.nodemanager.local-cache.max-files-per-directory8192yarn-default.xml dfs.https.server.keystore.resourcessl-server.xmlhdfs-default.xml mapreduce.jobtracker.taskcache.levels2mapred-default.xml hbase.regionserver.global.memstore.size0.4hbase-site.xml dfs.datanode.handler.count10hdfs-default.xml s3native.blocksize67108864core-default.xml dfs.datanode.fsdataset.volume.choosing.policyorg.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicyhdfs-site.xml mapreduce.client.completion.pollinterval5000mapred-default.xml hbase.hstore.compactionThreshold3hbase-default.xml dfs.stream-buffer-size4096hdfs-default.xml dfs.namenode.delegation.key.update-interval86400000hdfs-default.xml mapreduce.job.maps100mapred-site.xml hbase.master.logcleaner.ttl600000hbase-default.xml mapreduce.job.acl-view-job mapred-default.xml dfs.namenode.enable.retrycachetruehdfs-default.xml yarn.resourcemanager.connect.retry-interval.ms1000yarn-site.xml dfs.namenode.shared.edits.dirqjournal://hbasectrl338.frc3.facebook.com:8485;hbasectrl339.frc3.facebook.com:8485;hbasectrl340.frc3.facebook.com:8485;hbasectrl341.frc3.facebook.com:8485;hbasectrl342.frc3.facebook.com:8485/fbtrace004hdfs-site.xml yarn.timeline-service.leveldb-timeline-store.ttl-interval-ms300000yarn-default.xml fs.s3a.multipart.threshold2147483647core-default.xml dfs.namenode.decommission.interval30hdfs-default.xml hbase.rootdir.perms700hbase-default.xml hbase.hregion.majorcompaction604800000hbase-default.xml mapreduce.shuffle.max.connections0mapred-default.xml yarn.log-aggregation-enableTrueyarn-site.xml dfs.client-write-packet-size65536hdfs-default.xml dfs.client.file-block-storage-locations.timeout.millis1000hdfs-default.xml mapreduce.jobtracker.expire.trackers.interval600000mapred-default.xml dfs.client.block.write.retries3hdfs-default.xml mapreduce.task.io.sort.factor10mapred-default.xml hbase.hregion.memstore.flush.size134217728hbase-default.xml ha.health-monitor.sleep-after-disconnect.ms1000core-default.xml ha.zookeeper.session-timeout.ms5000core-default.xml yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-userstrueyarn-default.xml dfs.support.appendtruehdfs-default.xml hbase.master.balancer.stochastic.maxMovePercent0.01hbase-site.xml mapreduce.input.fileinputformat.list-status.num-threads1mapred-default.xml io.skip.checksum.errorsfalsecore-default.xml hbase.ipc.client.tcpnodelaytruehbase-default.xml hbase.regionserver.optionalcacheflushinterval3600000hbase-default.xml yarn.resourcemanager.scheduler.client.thread-count50yarn-default.xml dfs.namenode.safemode.extension10000hdfs-site.xml hbase.normalizer.period1800000hbase-default.xml mapreduce.jobhistory.move.thread-count3mapred-default.xml yarn.resourcemanager.zk-state-store.parent-path/rmstoreyarn-default.xml ipc.client.idlethreshold4000core-default.xml hbase.regionserver.port16020hbase-site.xml dfs.namenode.accesstime.precision3600000hdfs-default.xml mapreduce.task.profile.params-agentlib:hprof=cpu=samples,heap=sites,force=n,thread=y,verbose=n,file=%smapred-default.xml hbase.hstore.time.to.purge.deletes0hbase-default.xml hbase.regionserver.logroll.errors.tolerated2hbase-default.xml mapreduce.jobhistory.keytab/etc/security/keytab/jhs.service.keytabmapred-default.xml hbase.hstore.compaction.max10hbase-default.xml dfs.datanode.hdfs-blocks-metadata.enabledfalsehdfs-default.xml yarn.scheduler.minimum-allocation-mb5368yarn-site.xml yarn.resourcemanager.container-tokens.master-key-rolling-interval-secs86400yarn-default.xml hbase.master.infoserver.redirecttruehbase-default.xml mapreduce.reduce.shuffle.fetch.retry.interval-ms1000mapred-default.xml hadoop.user.group.static.mapping.overridesdr.who=;core-default.xml hadoop.security.kms.client.encrypted.key.cache.low-watermark0.3fcore-default.xml hbase.hregion.percolumnfamilyflush.size.lower.bound16777216hbase-default.xml fs.s3a.connection.ssl.enabledtruecore-default.xml dfs.datanode.directoryscan.interval21600hdfs-default.xml hbase.zookeeper.property.initLimit10hbase-default.xml yarn.resourcemanager.scheduler.monitor.policiesorg.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicyyarn-default.xml ipc.server.listen.queue.size128core-default.xml rpc.metrics.quantile.enablefalsecore-default.xml yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds-1yarn-default.xml mapreduce.jobtracker.persist.jobstatus.dir/jobtracker/jobsInfomapred-default.xml yarn.client.nodemanager-client-async.thread-pool-max-size500yarn-default.xml hadoop.security.group.mappingorg.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallbackcore-default.xml yarn.resourcemanager.system-metrics-publisher.enabledfalseyarn-default.xml dfs.namenode.name.dir/hay/dfs/hdfs-site.xml hbase.coprocessor.abortonerrortruehbase-default.xml yarn.am.liveness-monitor.expiry-interval-ms600000yarn-default.xml yarn.nm.liveness-monitor.expiry-interval-ms600000yarn-default.xml hbase.hstore.compaction.kv.max10hbase-default.xml hbase.hregion.preclose.flush.size5242880hbase-default.xml ftp.bytes-per-checksum512core-default.xml dfs.namenode.max.objects0hdfs-default.xml ha.zookeeper.quorumhbasectrl338.frc3.facebook.com:2181,hbasectrl339.frc3.facebook.com:2181,hbasectrl340.frc3.facebook.com:2181,hbasectrl341.frc3.facebook.com:2181,hbasectrl342.frc3.facebook.com:2181core-site.xml hbase.master.hfilecleaner.pluginsorg.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleanerhbase-default.xml mapreduce.job.emit-timeline-datafalsemapred-default.xml hbase.regionserver.thread.compaction.small4hbase-site.xml hbase.metrics.showTableNametruehbase-default.xml mapreduce.map.memory.mb5120mapred-site.xml yarn.client.nodemanager-connect.retry-interval-ms10000yarn-default.xml dfs.namenode.edits.journal-plugin.qjournalorg.apache.hadoop.hdfs.qjournal.client.QuorumJournalManagerhdfs-default.xml mapreduce.tasktracker.healthchecker.interval60000mapred-default.xml nfs.wtmax1048576hdfs-default.xml yarn.timeline-service.leveldb-timeline-store.start-time-read-cache-size10000yarn-default.xml hbase.lease.recovery.dfs.timeout64000hbase-default.xml hadoop.registry.zk.connection.timeout.ms15000yarn-default.xml yarn.resourcemanager.address${yarn.resourcemanager.hostname}:8032yarn-default.xml dfs.cachereport.intervalMsec10000hdfs-default.xml mapreduce.task.skip.start.attempts2mapred-default.xml yarn.resourcemanager.zk-timeout-ms10000yarn-default.xml dfs.namenode.checkpoint.edits.dir${dfs.namenode.checkpoint.dir}hdfs-default.xml hbase.server.scanner.max.result.size104857600hbase-default.xml hadoop.hdfs.configuration.version1hdfs-default.xml mapreduce.map.skip.maxrecords0mapred-default.xml yarn.resourcemanager.system-metrics-publisher.dispatcher.pool-size10yarn-default.xml dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold10737418240hdfs-default.xml nfs.allow.insecure.portstruehdfs-default.xml mapreduce.jobtracker.system.dir${hadoop.tmp.dir}/mapred/systemmapred-default.xml yarn.timeline-service.hostnamehbasectrl342.frc3.facebook.comyarn-site.xml hadoop.registry.rm.enabledfalseyarn-default.xml mapreduce.job.reducer.preempt.delay.sec0mapred-default.xml hbase.zookeeper.dns.nameserverdefaulthbase-default.xml hbase.ipc.server.callqueue.handler.factor0.7hbase-site.xml mapreduce.shuffle.ssl.enabledfalsemapred-default.xml yarn.nodemanager.vmem-pmem-ratio2.1yarn-default.xml yarn.nodemanager.container-manager.thread-count20yarn-default.xml dfs.encrypt.data.transferfalsehdfs-default.xml dfs.block.access.key.update.interval600hdfs-default.xml yarn.resourcemanager.cluster-idfbtrace004yarn-site.xml hbase.bulkload.staging.dir${hbase.fs.tmp.dir}hbase-default.xml hadoop.tmp.dir/haycore-site.xml dfs.namenode.audit.loggersdefaulthdfs-default.xml fs.AbstractFileSystem.har.implorg.apache.hadoop.fs.HarFscore-default.xml yarn.nodemanager.localizer.cache.target-size-mb10240yarn-default.xml yarn.http.policyHTTP_ONLYyarn-default.xml hbase.regionserver.logroll.period3600000hbase-default.xml dfs.client.short.circuit.replica.stale.threshold.ms1800000hdfs-default.xml yarn.timeline-service.webapp.https.address${yarn.timeline-service.hostname}:8190yarn-default.xml mapreduce.jobtracker.persist.jobstatus.hours1mapred-default.xml tfile.fs.output.buffer.size262144core-default.xml hbase.hregion.memstore.block.multiplier4hbase-default.xml dfs.namenode.checkpoint.check.period60hdfs-default.xml dfs.datanode.dns.interfacedefaulthdfs-default.xml fs.ftp.host.port21core-default.xml mapreduce.task.io.sort.mb100mapred-default.xml dfs.namenode.inotify.max.events.per.rpc1000hdfs-default.xml hadoop.security.group.mapping.ldap.search.attr.group.namecncore-default.xml dfs.namenode.avoid.read.stale.datanodeTruehdfs-site.xml mapreduce.output.fileoutputformat.compress.typeRECORDmapred-default.xml hbase.storescanner.parallel.seek.enablefalsehbase-default.xml hbase.snapshot.master.timeout.millis6000000hbase-site.xml hbase.regionserver.storefile.refresh.period0hbase-default.xml mapreduce.reduce.skip.proc.count.autoincrtruemapred-default.xml hbase.dfs.client.read.shortcircuit.buffer.size131072hbase-default.xml file.bytes-per-checksum512core-default.xml mapreduce.job.userlog.retain.hours24mapred-default.xml dfs.datanode.http.address0.0.0.0:50075hdfs-default.xml dfs.image.compressfalsehdfs-default.xml mapreduce.reduce.java.opts-server -Xms3g -Xmx4g -XX:+UseG1GC -XX:MaxGCPauseMillis=500mapred-site.xml dfs.journalnode.edits.dir/hay/journalhdfs-site.xml yarn.resourcemanager.zk-addresshbasectrl338.frc3.facebook.com,hbasectrl339.frc3.facebook.com,hbasectrl340.frc3.facebook.com,hbasectrl341.frc3.facebook.com,hbasectrl342.frc3.facebook.comyarn-site.xml ha.health-monitor.check-interval.ms1000core-default.xml dfs.permissions.enabledtruehdfs-default.xml hbase.thrift.htablepool.size.max1000hbase-default.xml yarn.resourcemanager.resource-tracker.client.thread-count50yarn-default.xml dfs.client.domain.socket.data.trafficfalsehdfs-default.xml dfs.image.compression.codecorg.apache.hadoop.io.compress.DefaultCodechdfs-default.xml dfs.datanode.address0.0.0.0:50010hdfs-default.xml dfs.block.access.token.enablefalsehdfs-default.xml mapreduce.reduce.input.buffer.percent0.0mapred-default.xml hbase.client.scanner.caching2147483647hbase-default.xml yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usagefalseyarn-default.xml mapreduce.tasktracker.local.dir.minspacestart0mapred-default.xml dfs.blockreport.intervalMsec21600000hdfs-default.xml hbase.snapshot.restore.take.failsafe.snapshottruehbase-default.xml ha.health-monitor.rpc-timeout.ms45000core-default.xml dfs.client.failover.connection.retries0hdfs-default.xml dfs.namenode.kerberos.internal.spnego.principal${dfs.web.authentication.kerberos.principal}hdfs-default.xml hadoop.policy.filehbase-policy.xmlhbase-default.xml hbase.fs.tmp.dir/user/${user.name}/hbase-staginghbase-default.xml yarn.scheduler.maximum-allocation-mb10736yarn-site.xml mapreduce.task.files.preserve.failedtasksfalsemapred-default.xml hbase.status.publishedfalsehbase-default.xml hbase.snapshot.master.timeoutMillis6000000hbase-site.xml yarn.nodemanager.delete.thread-count4yarn-default.xml dfs.https.enablefalsehdfs-default.xml mapreduce.output.fileoutputformat.compress.codecorg.apache.hadoop.io.compress.DefaultCodecmapred-default.xml map.sort.classorg.apache.hadoop.util.QuickSortmapred-default.xml mapreduce.job.classloaderfalsemapred-default.xml hadoop.registry.zk.retry.ceiling.ms60000yarn-default.xml mapreduce.jobtracker.tasktracker.maxblacklists4mapred-default.xml io.seqfile.compress.blocksize1000000core-default.xml dfs.blocksize268435456hdfs-site.xml hbase.client.scanner.max.result.size2097152hbase-default.xml mapreduce.task.profile.maps0-2mapred-default.xml mapreduce.jobtracker.staging.root.dir${hadoop.tmp.dir}/mapred/stagingmapred-default.xml yarn.nodemanager.localizer.cache.cleanup.interval-ms600000yarn-default.xml mapreduce.jobtracker.http.address0.0.0.0:50030mapred-default.xml hbase.regionserver.info.bindAddress0.0.0.0hbase-default.xml dfs.client.mmap.cache.timeout.ms3600000hdfs-default.xml yarn.timeline-service.generic-application-history.fs-history-store.uri/hay/yarn/timeline/generic-historyyarn-site.xml hadoop.security.java.secure.random.algorithmSHA1PRNGcore-default.xml fs.client.resolve.remote.symlinkstruecore-default.xml hbase.master.logcleaner.pluginsorg.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleanerhbase-default.xml hbase.master.info.port.orig16010programatically hbase.data.umask.enablefalsehbase-default.xml hbase.coprocessor.user.enabledtruehbase-default.xml mapreduce.tasktracker.local.dir.minspacekill0mapred-default.xml nfs.mountd.port4242hdfs-default.xml yarn.nodemanager.disk-health-checker.min-healthy-disks0.25yarn-default.xml mapreduce.tasktracker.taskmemorymanager.monitoringinterval5000mapred-default.xml hbase.local.dir${hbase.tmp.dir}/local/hbase-default.xml dfs.namenode.resource.du.reserved104857600hdfs-default.xml mapreduce.job.end-notification.retry.interval1000mapred-default.xml mapreduce.jobhistory.loadedjobs.cache.size5mapred-default.xml dfs.client.datanode-restart.timeout30hdfs-default.xml yarn.nodemanager.local-dirs/mnt/d0/FBTRACE004-YARN/nm-local-dir,/mnt/d1/FBTRACE004-YARN/nm-local-dir,/mnt/d2/FBTRACE004-YARN/nm-local-dir,/mnt/d3/FBTRACE004-YARN/nm-local-dir,/mnt/d4/FBTRACE004-YARN/nm-local-dir,/mnt/d5/FBTRACE004-YARN/nm-local-dir,/mnt/d6/FBTRACE004-YARN/nm-local-dir,/mnt/d7/FBTRACE004-YARN/nm-local-dir,/mnt/d8/FBTRACE004-YARN/nm-local-dir,/mnt/d9/FBTRACE004-YARN/nm-local-dir,/mnt/d10/FBTRACE004-YARN/nm-local-dir,/mnt/d11/FBTRACE004-YARN/nm-local-dir,/mnt/d12/FBTRACE004-YARN/nm-local-dir,/mnt/d13/FBTRACE004-YARN/nm-local-dir,/mnt/d14/FBTRACE004-YARN/nm-local-diryarn-site.xml hbase.table.lock.enabletruehbase-default.xml dfs.datanode.block.id.layout.upgrade.threads12hdfs-default.xml hbase.storescanner.parallel.seek.threads10hbase-default.xml hadoop.registry.jaas.contextClientyarn-default.xml yarn.timeline-service.webapp.address${yarn.timeline-service.hostname}:8188yarn-default.xml hbase.online.schema.update.enabletruehbase-default.xml mapreduce.jobhistory.addresshbasectrl342.frc3.facebook.com:10020mapred-site.xml mapreduce.jobtracker.persist.jobstatus.activetruemapred-default.xml file.blocksize67108864core-default.xml dfs.datanode.readahead.bytes4193404hdfs-default.xml hbase.zookeeper.property.dataDir/hay/zookeeperhbase-site.xml dfs.namenode.http-address0.0.0.0:50070hdfs-default.xml hadoop.work.around.non.threadsafe.getpwuidfalsecore-default.xml yarn.resourcemanager.configuration.provider-classorg.apache.hadoop.yarn.LocalConfigurationProvideryarn-default.xml yarn.nodemanager.recovery.enabledTrueyarn-site.xml yarn.resourcemanager.hostname0.0.0.0yarn-default.xml yarn.resourcemanager.webapp.address.rm4hbasectrl342.frc3.facebook.com:8088yarn-site.xml yarn.resourcemanager.webapp.address.rm3hbasectrl341.frc3.facebook.com:8088yarn-site.xml hbase.regionserver.handler.abort.on.error.percent0.5hbase-default.xml yarn.resourcemanager.webapp.address.rm2hbasectrl340.frc3.facebook.com:8088yarn-site.xml yarn.resourcemanager.webapp.address.rm1hbasectrl339.frc3.facebook.com:8088yarn-site.xml yarn.resourcemanager.webapp.address.rm0hbasectrl338.frc3.facebook.com:8088yarn-site.xml fs.s3n.multipart.uploads.enabledfalsecore-default.xml hbase.security.exec.permission.checksfalsehbase-default.xml hbase.master.normalizer.classorg.apache.hadoop.hbase.master.normalizer.SimpleRegionNormalizerhbase-default.xml dfs.namenode.fs-limits.max-component-length255hdfs-default.xml hbase.regionserver.info.port.autofalsehbase-default.xml ha.failover-controller.cli-check.rpc-timeout.ms20000core-default.xml hbase.auth.key.update.interval86400000hbase-default.xml ftp.client-write-packet-size65536core-default.xml mapreduce.reduce.shuffle.parallelcopies5mapred-default.xml mapreduce.jobhistory.principaljhs/_HOST@REALM.TLDmapred-default.xml hadoop.http.authentication.simple.anonymous.allowedtruecore-default.xml yarn.log-aggregation.retain-seconds604800yarn-site.xml hbase.regionserver.thrift.framedfalsehbase-default.xml hbase.zookeeper.property.maxClientCnxns300hbase-default.xml mapreduce.client.genericoptionsparser.usedtrueprogramatically yarn.timeline-service.http-authentication.simple.anonymous.allowedtrueyarn-default.xml dfs.ha.namenodes.fbtrace004nn0,nn1hdfs-site.xml dfs.namenode.secondary.https-address0.0.0.0:50091hdfs-default.xml mapreduce.job.ubertask.maxreduces1mapred-default.xml hbase.assignment.usezkFalsehbase-site.xml yarn.nodemanager.health-checker.interval-ms600000yarn-default.xml dfs.namenode.fs-limits.max-xattr-size16384hdfs-default.xml fs.s3a.multipart.purgefalsecore-default.xml hadoop.security.kms.client.encrypted.key.cache.num.refill.threads2core-default.xml hbase.server.compactchecker.interval.multiplier1000hbase-default.xml yarn.timeline-service.store-classorg.apache.hadoop.yarn.server.timeline.LeveldbTimelineStoreyarn-default.xml mapreduce.shuffle.transfer.buffer.size131072mapred-default.xml yarn.resourcemanager.zk-num-retries1000yarn-default.xml mapreduce.jobtracker.jobhistory.task.numberprogresssplits12mapred-default.xml yarn.nodemanager.webapp.address${yarn.nodemanager.hostname}:8042yarn-default.xml yarn.app.mapreduce.client-am.ipc.max-retries3mapred-default.xml ha.failover-controller.new-active.rpc-timeout.ms60000core-default.xml mapreduce.jobhistory.client.thread-count10mapred-default.xml fs.trash.interval0core-default.xml hbase.client.max.perregion.tasks1hbase-default.xml mapreduce.reduce.skip.maxgroups0mapred-default.xml hfile.block.cache.size.min.range0.1hbase-site.xml mapreduce.reduce.memory.mb5120mapred-site.xml yarn.nodemanager.health-checker.script.timeout-ms1200000yarn-default.xml dfs.datanode.du.reserved0hdfs-default.xml dfs.namenode.resource.check.interval5000hdfs-default.xml mapreduce.client.progressmonitor.pollinterval1000mapred-default.xml yarn.nodemanager.hostname0.0.0.0yarn-default.xml yarn.resourcemanager.ha.enabledTrueyarn-site.xml dfs.ha.log-roll.period120hdfs-default.xml yarn.scheduler.minimum-allocation-vcores1yarn-default.xml dfs.client.block.write.replace-datanode-on-failure.best-effortfalsehdfs-default.xml yarn.app.mapreduce.am.container.log.limit.kb0mapred-default.xml hadoop.http.authentication.signature.secret.file${user.home}/hadoop-http-auth-signature-secretcore-default.xml mapreduce.jobhistory.move.interval-ms180000mapred-default.xml yarn.nodemanager.container-executor.classorg.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutoryarn-default.xml hadoop.security.authorizationfalsecore-default.xml dfs.storage.policy.enabledtruehdfs-default.xml dfs.datanode.https.address0.0.0.0:50475hdfs-default.xml yarn.nodemanager.localizer.address${yarn.nodemanager.hostname}:8040yarn-default.xml mapreduce.jobhistory.recovery.store.fs.uri${fs.defaultFS}/mapred/history/recoverystoremapred-site.xml dfs.namenode.replication.min1hdfs-default.xml mapreduce.shuffle.connection-keep-alive.enablefalsemapred-default.xml hadoop.common.configuration.version0.23.0core-default.xml yarn.app.mapreduce.task.container.log.backups0mapred-default.xml hadoop.security.groups.negative-cache.secs30core-default.xml yarn.resourcemanager.hostname.rm4hbasectrl342.frc3.facebook.comyarn-site.xml yarn.resourcemanager.hostname.rm3hbasectrl341.frc3.facebook.comyarn-site.xml mapreduce.ifile.readaheadtruemapred-default.xml yarn.nodemanager.resource.percentage-physical-cpu-limit100yarn-default.xml yarn.resourcemanager.hostname.rm2hbasectrl340.frc3.facebook.comyarn-site.xml yarn.resourcemanager.hostname.rm1hbasectrl339.frc3.facebook.comyarn-site.xml mapreduce.job.max.split.locations10mapred-default.xml dfs.datanode.max.locked.memory0hdfs-default.xml yarn.resourcemanager.hostname.rm0hbasectrl338.frc3.facebook.comyarn-site.xml hadoop.registry.zk.quorumlocalhost:2181yarn-default.xml mapreduce.jobhistory.joblist.cache.size20000mapred-default.xml mapreduce.job.end-notification.max.attempts5mapred-default.xml dfs.image.transfer.timeout60000hdfs-default.xml dfs.client.read.shortcircuit.skip.checksumfalsehdfs-default.xml nfs.rtmax1048576hdfs-default.xml dfs.namenode.edit.log.autoroll.check.interval.ms300000hdfs-default.xml mapreduce.reduce.shuffle.connect.timeout180000mapred-default.xml dfs.datanode.failed.volumes.tolerated3hdfs-site.xml mapreduce.jobhistory.webapp.addresshbasectrl342.frc3.facebook.com:19888mapred-site.xml fs.s3a.connection.timeout5000core-default.xml dfs.client.mmap.retry.timeout.ms300000hdfs-default.xml dfs.datanode.data.dir.perm700hdfs-default.xml hadoop.http.authentication.token.validity36000core-default.xml ipc.client.connect.max.retries.on.timeouts45core-default.xml yarn.nodemanager.docker-container-executor.exec-name/usr/bin/dockeryarn-default.xml yarn.app.mapreduce.am.job.committer.cancel-timeout60000mapred-default.xml dfs.ha.fencing.ssh.connect-timeout30000core-default.xml hbase.data.umask000hbase-default.xml mapreduce.reduce.log.levelINFOmapred-default.xml mapreduce.reduce.shuffle.merge.percent0.66mapred-default.xml ipc.client.fallback-to-simple-auth-allowedfalsecore-default.xml io.serializationsorg.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerializationcore-default.xml hbase.regionserver.hlog.writer.implorg.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriterhbase-default.xml fs.s3.block.size67108864core-default.xml yarn.nodemanager.linux-container-executor.nonsecure-mode.local-usernobodyyarn-default.xml hadoop.kerberos.kinit.commandkinitcore-default.xml hadoop.security.kms.client.encrypted.key.cache.expiry43200000core-default.xml yarn.resourcemanager.fs.state-store.uri${hadoop.tmp.dir}/yarn/system/rmstoreyarn-default.xml hbase.regionserver.region.split.policyorg.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicyhbase-site.xml hbase.cells.scanned.per.heartbeat.check10000hbase-default.xml yarn.admin.acl*yarn-default.xml dfs.namenode.delegation.token.max-lifetime604800000hdfs-default.xml mapreduce.reduce.merge.inmem.threshold1000mapred-default.xml net.topology.implorg.apache.hadoop.net.NetworkTopologycore-default.xml yarn.resourcemanager.ha.automatic-failover.enabledtrueyarn-default.xml dfs.datanode.use.datanode.hostnamefalsehdfs-default.xml dfs.heartbeat.interval3hdfs-default.xml yarn.resourcemanager.scheduler.classorg.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduleryarn-default.xml io.map.index.skip0core-default.xml dfs.namenode.handler.count24hdfs-site.xml yarn.resourcemanager.webapp.https.address${yarn.resourcemanager.hostname}:8090yarn-default.xml yarn.nodemanager.admin-envMALLOC_ARENA_MAX=$MALLOC_ARENA_MAXyarn-default.xml hbase.client.max.total.tasks100hbase-default.xml yarn.resourcemanager.address.rm4hbasectrl342.frc3.facebook.com:8032yarn-site.xml yarn.resourcemanager.address.rm3hbasectrl341.frc3.facebook.com:8032yarn-site.xml hadoop.security.crypto.cipher.suiteAES/CTR/NoPaddingcore-default.xml yarn.resourcemanager.address.rm2hbasectrl340.frc3.facebook.com:8032yarn-site.xml mapreduce.task.profile.map.params${mapreduce.task.profile.params}mapred-default.xml mapreduce.jobtracker.jobhistory.block.size3145728mapred-default.xml yarn.resourcemanager.address.rm1hbasectrl339.frc3.facebook.com:8032yarn-site.xml yarn.resourcemanager.address.rm0hbasectrl338.frc3.facebook.com:8032yarn-site.xml hbase.zookeeper.peerport2888hbase-default.xml hadoop.security.crypto.buffer.size8192core-default.xml hbase.table.max.rowsize1073741824hbase-default.xml yarn.nodemanager.aux-services.mapreduce_shuffle.classorg.apache.hadoop.mapred.ShuffleHandleryarn-default.xml hbase.regionserver.ipc.address0.0.0.0hbase-site.xml hbase.master.executor.serverops.threads18hbase-site.xml mapreduce.cluster.acls.enabledfalsemapred-default.xml hbase.regionserver.info.port16030programatically hbase.coprocessor.enabledtruehbase-default.xml hbase.hregion.majorcompaction.jitter0.50hbase-default.xml hbase.snapshot.region.timeout6000000hbase-site.xml fs.har.impl.disable.cachetruecore-default.xml mapreduce.tasktracker.map.tasks.maximum2mapred-default.xml ipc.client.connect.timeout20000core-default.xml yarn.timeline-service.generic-application-history.enabledTrueyarn-site.xml dfs.client.failover.proxy.provider.fbtrace004org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProviderhdfs-site.xml yarn.nodemanager.remote-app-log-dir-suffixlogsyarn-default.xml fs.df.interval60000core-default.xml hbase.regionserver.thrift.framed.max_frame_size_in_mb2hbase-default.xml hbase.http.filter.initializersorg.apache.hadoop.hbase.http.lib.StaticUserWebFilterhbase-default.xml hadoop.util.hash.typemurmurcore-default.xml mapreduce.jobhistory.minicluster.fixed.portsfalsemapred-default.xml mapreduce.jobtracker.jobhistory.lru.cache.size5mapred-default.xml dfs.client.failover.max.attempts15hdfs-default.xml dfs.client.use.datanode.hostnamefalsehdfs-default.xml ha.zookeeper.aclworld:anyone:rwcdacore-default.xml hbase.replication.rpc.codecorg.apache.hadoop.hbase.codec.KeyValueCodecWithTagshbase-default.xml mapreduce.jobtracker.maxtasks.perjob-1mapred-default.xml mapreduce.map.sort.spill.percent0.80mapred-default.xml file.stream-buffer-size4096core-default.xml hbase.meta.replicas.usefalseprogramatically yarn.resourcemanager.ha.automatic-failover.embeddedtrueyarn-default.xml hbase.regionserver.catalog.timeout600000hbase-default.xml hbase.security.authenticationsimplehbase-default.xml yarn.resourcemanager.nodemanager.minimum.versionNONEyarn-default.xml hadoop.fuse.connection.timeout300hdfs-default.xml hbase.client.keyvalue.maxsize10485760hbase-default.xml hbase.regionserver.thrift.compactfalsehbase-default.xml mapreduce.tasktracker.instrumentationorg.apache.hadoop.mapred.TaskTrackerMetricsInstmapred-default.xml io.seqfile.sorter.recordlimit1000000core-default.xml yarn.app.mapreduce.am.resource.mb5368yarn-site.xml mapreduce.framework.nameyarnmapred-site.xml mapreduce.job.reduce.slowstart.completedmaps0.05mapred-default.xml yarn.resourcemanager.client.thread-count50yarn-default.xml mapreduce.cluster.temp.dir${hadoop.tmp.dir}/mapred/tempmapred-default.xml dfs.client.mmap.enabledtruehdfs-default.xml mapreduce.jobhistory.intermediate-done-dir${yarn.app.mapreduce.am.staging-dir}/history/done_intermediatemapred-default.xml hbase.defaults.for.version.skipfalsehbase-default.xml fs.s3a.attempts.maximum10core-default.xml hbase.rest.support.proxyuserfalsehbase-default.xml Logs =========================================================== RS Queue: =========================================================== Compaction/Split Queue summary: compaction_queue=(0:0), split_queue=0, merge_queue=0 Compaction/Split Queue dump: LargeCompation Queue: SmallCompation Queue: Split Queue: Region Merge Queue: Flush Queue summary: flush_queue=13 Flush Queue Queue dump: Flush Queue: [flush region mytable,52e64cf2:gAu75CpwU2IH,1437586870140.8529b87cf90caa664f8c8f76c29f66d7.] [flush region mytable,38cf001b:gCNaR20NqRAx,1437586774027.8364b228aae1db4fd387fb43a84c0887.] [flush region mytable,8df255a2:gAWvGhVrJfHH,1437587092723.1cd01856ba48fda59932ab30828d0249.] [flush region mytable,c4535489:gFM60+4imDRV,1438958101332.941ab2c011d71e4eb97e0cac1f39a146.] [flush region mytable,30345c16:gEGY/DmcjWU5,1437586743341.536c705f415e867ce5bac86f9514d354.] [flush region mytable,69e9f9f0:gG45UUQO3OJh,1440095987816.d7da859cefb4c000ab8fb464aee4fba4.] [flush region mytable,eaa9c6c2:gHExFMSl9zE+,1439309668812.e4fd58c273beae6a10d95d393a255716.] [flush region mytable,43226c56:gELoKpZaukZu,1437586808965.471189bf69582d8d028c205abe907099.] [flush region mytable,64f88d,1437586940189.81faa5084559083c407043687da4629d.] [flush region mytable,0e9c40df:gHRMnoeRoHfi,1437586617090.cfd4f0e2a31879aaf47392fffb22454d.] [flush region mytable,afc36305:gGgjvtvVY9ua,1437587156051.192b4cbd2559379ebfdd632edfed85b4.] [flush region mytable,3f4bc8,1437586791858.8277dad8e0af2c9a4a85454cc61fc3f7.] [flush region mytable,9767b73d:gGMi5MBknVQn,1437587109978.6ff8e150920337728aae90a5d6506505.]