2020-06-19 23:24:56,293 INFO [RS_OPEN_PRIORITY_REGION-regionserver/hadoop-hbnn0005:16020-0] coprocessor.CoprocessorHost: System coprocessor org.apache.hadoop.hbase.security.access.AccessController loaded, priority=536870911. 2020-06-19 23:24:56,294 WARN [RS_OPEN_PRIORITY_REGION-regionserver/hadoop-hbnn0005:16020-0] access.SecureBulkLoadEndpoint: SecureBulkLoadEndpoint is deprecated. It will be removed in future releases. 2020-06-19 23:24:56,295 WARN [RS_OPEN_PRIORITY_REGION-regionserver/hadoop-hbnn0005:16020-0] access.SecureBulkLoadEndpoint: Secure bulk load has been integrated into HBase core. 2020-06-19 23:24:56,295 INFO [RS_OPEN_PRIORITY_REGION-regionserver/hadoop-hbnn0005:16020-0] coprocessor.CoprocessorHost: System coprocessor org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint loaded, priority=536870912. 2020-06-19 23:24:56,295 INFO [RS_OPEN_PRIORITY_REGION-regionserver/hadoop-hbnn0005:16020-0] coprocessor.CoprocessorHost: System coprocessor org.apache.hadoop.hbase.security.token.TokenProvider loaded, priority=536870913. 2020-06-19 23:24:56,315 INFO [RpcServer.priority.FPBQ.Fifo.handler=18,queue=0,port=16020] regionserver.RSRpcServices: Open hbase:quota,,1592548149597.bb192d28c3c02a089e5796d829e9f3f2. 2020-06-19 23:24:56,317 INFO [RS_OPEN_PRIORITY_REGION-regionserver/hadoop-hbnn0005:16020-1] coprocessor.CoprocessorHost: System coprocessor org.apache.hadoop.hbase.security.access.AccessController loaded, priority=536870911. 2020-06-19 23:24:56,317 WARN [RS_OPEN_PRIORITY_REGION-regionserver/hadoop-hbnn0005:16020-1] access.SecureBulkLoadEndpoint: SecureBulkLoadEndpoint is deprecated. It will be removed in future releases. 2020-06-19 23:24:56,317 WARN [RS_OPEN_PRIORITY_REGION-regionserver/hadoop-hbnn0005:16020-1] access.SecureBulkLoadEndpoint: Secure bulk load has been integrated into HBase core. 2020-06-19 23:24:56,317 INFO [RS_OPEN_PRIORITY_REGION-regionserver/hadoop-hbnn0005:16020-1] coprocessor.CoprocessorHost: System coprocessor org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint loaded, priority=536870912. 2020-06-19 23:24:56,317 INFO [RS_OPEN_PRIORITY_REGION-regionserver/hadoop-hbnn0005:16020-1] coprocessor.CoprocessorHost: System coprocessor org.apache.hadoop.hbase.security.token.TokenProvider loaded, priority=536870913. 2020-06-19 23:24:56,337 INFO [StoreOpener-580526f82caa92a09df9e02dac21ad10-1] hfile.CacheConfig: Created cacheConfig for l: blockCache=LruBlockCache{blockCount=3, currentSize=927.80 KB, freeSize=1.19 GB, maxSize=1.19 GB, heapSize=927.80 KB, minSize=1.13 GB, minFactor=0.95, multiSize=579.96 MB, multiFactor=0.5, singleSize=289.98 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2020-06-19 23:24:56,337 INFO [RS_OPEN_PRIORITY_REGION-regionserver/hadoop-hbnn0005:16020-2] coprocessor.CoprocessorHost: System coprocessor org.apache.hadoop.hbase.security.access.AccessController loaded, priority=536870911. 2020-06-19 23:24:56,338 WARN [RS_OPEN_PRIORITY_REGION-regionserver/hadoop-hbnn0005:16020-2] access.SecureBulkLoadEndpoint: SecureBulkLoadEndpoint is deprecated. It will be removed in future releases. 2020-06-19 23:24:56,338 WARN [RS_OPEN_PRIORITY_REGION-regionserver/hadoop-hbnn0005:16020-2] access.SecureBulkLoadEndpoint: Secure bulk load has been integrated into HBase core. 2020-06-19 23:24:56,338 INFO [RS_OPEN_PRIORITY_REGION-regionserver/hadoop-hbnn0005:16020-2] coprocessor.CoprocessorHost: System coprocessor org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint loaded, priority=536870912. 2020-06-19 23:24:56,338 INFO [RS_OPEN_PRIORITY_REGION-regionserver/hadoop-hbnn0005:16020-2] coprocessor.CoprocessorHost: System coprocessor org.apache.hadoop.hbase.security.token.TokenProvider loaded, priority=536870913. 2020-06-19 23:24:56,338 INFO [StoreOpener-580526f82caa92a09df9e02dac21ad10-1] compactions.CompactionConfiguration: size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2020-06-19 23:24:56,367 INFO [StoreOpener-f5c7e71fb5e5cab3b27e52600996f7fd-1] hfile.CacheConfig: Created cacheConfig for info: blockCache=LruBlockCache{blockCount=3, currentSize=927.80 KB, freeSize=1.19 GB, maxSize=1.19 GB, heapSize=927.80 KB, minSize=1.13 GB, minFactor=0.95, multiSize=579.96 MB, multiFactor=0.5, singleSize=289.98 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2020-06-19 23:24:56,368 INFO [StoreOpener-f5c7e71fb5e5cab3b27e52600996f7fd-1] compactions.CompactionConfiguration: size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2020-06-19 23:24:56,384 INFO [StoreOpener-bb192d28c3c02a089e5796d829e9f3f2-1] hfile.CacheConfig: Created cacheConfig for q: blockCache=LruBlockCache{blockCount=3, currentSize=927.80 KB, freeSize=1.19 GB, maxSize=1.19 GB, heapSize=927.80 KB, minSize=1.13 GB, minFactor=0.95, multiSize=579.96 MB, multiFactor=0.5, singleSize=289.98 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2020-06-19 23:24:56,386 INFO [StoreOpener-bb192d28c3c02a089e5796d829e9f3f2-1] compactions.CompactionConfiguration: size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2020-06-19 23:24:56,390 INFO [StoreOpener-bb192d28c3c02a089e5796d829e9f3f2-1] regionserver.HStore: Store=q, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2020-06-19 23:24:56,405 INFO [StoreOpener-bb192d28c3c02a089e5796d829e9f3f2-1] hfile.CacheConfig: Created cacheConfig for u: blockCache=LruBlockCache{blockCount=3, currentSize=927.80 KB, freeSize=1.19 GB, maxSize=1.19 GB, heapSize=927.80 KB, minSize=1.13 GB, minFactor=0.95, multiSize=579.96 MB, multiFactor=0.5, singleSize=289.98 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2020-06-19 23:24:56,405 INFO [StoreOpener-bb192d28c3c02a089e5796d829e9f3f2-1] compactions.CompactionConfiguration: size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2020-06-19 23:24:56,408 INFO [StoreOpener-580526f82caa92a09df9e02dac21ad10-1] regionserver.HStore: Store=l, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2020-06-19 23:24:56,410 INFO [StoreOpener-bb192d28c3c02a089e5796d829e9f3f2-1] regionserver.HStore: Store=u, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2020-06-19 23:24:56,448 INFO [StoreOpener-f5c7e71fb5e5cab3b27e52600996f7fd-1] regionserver.HStore: Store=info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2020-06-19 23:24:56,469 INFO [RS_OPEN_PRIORITY_REGION-regionserver/hadoop-hbnn0005:16020-2] regionserver.HRegion: Opened bb192d28c3c02a089e5796d829e9f3f2; next sequenceid=26 2020-06-19 23:24:56,474 INFO [RS_OPEN_PRIORITY_REGION-regionserver/hadoop-hbnn0005:16020-0] regionserver.HRegion: Opened 580526f82caa92a09df9e02dac21ad10; next sequenceid=33 2020-06-19 23:24:56,480 INFO [PostOpenDeployTasks:bb192d28c3c02a089e5796d829e9f3f2] regionserver.HRegionServer: Post open deploy tasks for hbase:quota,,1592548149597.bb192d28c3c02a089e5796d829e9f3f2. 2020-06-19 23:24:56,518 INFO [PostOpenDeployTasks:580526f82caa92a09df9e02dac21ad10] regionserver.HRegionServer: Post open deploy tasks for hbase:acl,,1592548149755.580526f82caa92a09df9e02dac21ad10. 2020-06-19 23:24:56,672 INFO [RS_OPEN_PRIORITY_REGION-regionserver/hadoop-hbnn0005:16020-1] regionserver.HRegion: Opened f5c7e71fb5e5cab3b27e52600996f7fd; next sequenceid=162 2020-06-19 23:24:56,675 INFO [PostOpenDeployTasks:f5c7e71fb5e5cab3b27e52600996f7fd] regionserver.HRegionServer: Post open deploy tasks for hbase:namespace,,1592548148073.f5c7e71fb5e5cab3b27e52600996f7fd. 2020-06-19 23:25:38,128 INFO [ReadOnlyZKClient-hadoop-hbnn0001.eniot.io:2181,hadoop-hbnn0002.eniot.io:2181,hadoop-hbnn0003.eniot.io:2181@0x7ae224fd] zookeeper.ZooKeeper: Session: 0x30003681d35007d closed 2020-06-19 23:25:38,129 INFO [ReadOnlyZKClient-hadoop-hbnn0001.eniot.io:2181,hadoop-hbnn0002.eniot.io:2181,hadoop-hbnn0003.eniot.io:2181@0x7ae224fd-EventThread] zookeeper.ClientCnxn: EventThread shut down 2020-06-19 23:25:54,346 INFO [ReadOnlyZKClient-hadoop-hbnn0001.eniot.io:2181,hadoop-hbnn0002.eniot.io:2181,hadoop-hbnn0003.eniot.io:2181@0x7ae224fd] zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-hbnn0001.eniot.io:2181,hadoop-hbnn0002.eniot.io:2181,hadoop-hbnn0003.eniot.io:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$38/683281457@267de756 2020-06-19 23:25:54,349 INFO [ReadOnlyZKClient-hadoop-hbnn0001.eniot.io:2181,hadoop-hbnn0002.eniot.io:2181,hadoop-hbnn0003.eniot.io:2181@0x7ae224fd-SendThread(hadoop-hbnn0001.eniot.io:2181)] zookeeper.ClientCnxn: Opening socket connection to server hadoop-hbnn0001.eniot.io/172.20.101.35:2181. Will not attempt to authenticate using SASL (unknown error) 2020-06-19 23:25:54,350 INFO [ReadOnlyZKClient-hadoop-hbnn0001.eniot.io:2181,hadoop-hbnn0002.eniot.io:2181,hadoop-hbnn0003.eniot.io:2181@0x7ae224fd-SendThread(hadoop-hbnn0001.eniot.io:2181)] zookeeper.ClientCnxn: Socket connection established, initiating session, client: /172.20.101.36:33694, server: hadoop-hbnn0001.eniot.io/172.20.101.35:2181 2020-06-19 23:25:54,356 INFO [ReadOnlyZKClient-hadoop-hbnn0001.eniot.io:2181,hadoop-hbnn0002.eniot.io:2181,hadoop-hbnn0003.eniot.io:2181@0x7ae224fd-SendThread(hadoop-hbnn0001.eniot.io:2181)] zookeeper.ClientCnxn: Session establishment complete on server hadoop-hbnn0001.eniot.io/172.20.101.35:2181, sessionid = 0x100035d07170079, negotiated timeout = 40000 2020-06-19 23:26:54,370 INFO [ReadOnlyZKClient-hadoop-hbnn0001.eniot.io:2181,hadoop-hbnn0002.eniot.io:2181,hadoop-hbnn0003.eniot.io:2181@0x7ae224fd] zookeeper.ZooKeeper: Session: 0x100035d07170079 closed 2020-06-19 23:26:54,370 INFO [ReadOnlyZKClient-hadoop-hbnn0001.eniot.io:2181,hadoop-hbnn0002.eniot.io:2181,hadoop-hbnn0003.eniot.io:2181@0x7ae224fd-EventThread] zookeeper.ClientCnxn: EventThread shut down 2020-06-19 23:29:36,601 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=932.66 KB, freeSize=1.19 GB, max=1.19 GB, blockCount=7, accesses=39, hits=32, hitRatio=82.05%, , cachingAccesses=39, cachingHits=32, cachingHitsRatio=82.05%, evictions=29, evicted=0, evictedPerRun=0.0 2020-06-19 23:29:59,324 INFO [regionserver/hadoop-hbnn0005:16020.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 130387 ms 2020-06-19 23:31:19,463 INFO [regionserver/hadoop-hbnn0005:16020.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of hbase:namespace,,1592548148073.f5c7e71fb5e5cab3b27e52600996f7fd. because info has an old edit so flush to free WALs after random delay 230337 ms 2020-06-19 23:32:09,715 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 3/3 column families, dataSize=2.70 KB heapSize=5.08 KB 2020-06-19 23:32:48,282 WARN [RpcServer.priority.FPBQ.Fifo.handler=18,queue=0,port=16020] ipc.RpcServer: Can not complete this request in time, drop it: callId: 30 service: ClientService methodName: Mutate size: 146 connection: 172.20.101.35:58756 deadline: 1592580528240 param: region= hbase:namespace,,1592548148073.f5c7e71fb5e5cab3b27e52600996f7fd., row=abcd2 connection: 172.20.101.35:58756 2020-06-19 23:33:48,404 WARN [RpcServer.priority.FPBQ.Fifo.handler=19,queue=1,port=16020] ipc.RpcServer: Can not complete this request in time, drop it: callId: 34 service: ClientService methodName: Mutate size: 146 connection: 172.20.101.35:58756 deadline: 1592580588364 param: region= hbase:namespace,,1592548148073.f5c7e71fb5e5cab3b27e52600996f7fd., row=abcd2 connection: 172.20.101.35:58756 2020-06-19 23:34:36,601 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=932.66 KB, freeSize=1.19 GB, max=1.19 GB, blockCount=7, accesses=63, hits=56, hitRatio=88.89%, , cachingAccesses=63, cachingHits=56, cachingHitsRatio=88.89%, evictions=59, evicted=0, evictedPerRun=0.0 2020-06-19 23:34:48,634 WARN [RpcServer.priority.FPBQ.Fifo.handler=17,queue=1,port=16020] ipc.RpcServer: Can not complete this request in time, drop it: callId: 40 service: ClientService methodName: Mutate size: 146 connection: 172.20.101.35:58756 deadline: 1592580648594 param: region= hbase:namespace,,1592548148073.f5c7e71fb5e5cab3b27e52600996f7fd., row=abcd2 connection: 172.20.101.35:58756 2020-06-19 23:35:09,801 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, dataSize=37 B heapSize=320 B 2020-06-19 23:35:19,716 INFO [regionserver/hadoop-hbnn0005:16020.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of hbase:namespace,,1592548148073.f5c7e71fb5e5cab3b27e52600996f7fd. because info has an old edit so flush to free WALs after random delay 99170 ms 2020-06-19 23:35:48,960 WARN [RpcServer.priority.FPBQ.Fifo.handler=16,queue=0,port=16020] ipc.RpcServer: Can not complete this request in time, drop it: callId: 44 service: ClientService methodName: Mutate size: 146 connection: 172.20.101.35:58756 deadline: 1592580708920 param: region= hbase:namespace,,1592548148073.f5c7e71fb5e5cab3b27e52600996f7fd., row=abcd2 connection: 172.20.101.35:58756 2020-06-19 23:36:49,480 WARN [RpcServer.priority.FPBQ.Fifo.handler=15,queue=1,port=16020] ipc.RpcServer: Can not complete this request in time, drop it: callId: 50 service: ClientService methodName: Mutate size: 146 connection: 172.20.101.35:58756 deadline: 1592580769441 param: region= hbase:namespace,,1592548148073.f5c7e71fb5e5cab3b27e52600996f7fd., row=abcd2 connection: 172.20.101.35:58756 2020-06-19 23:37:09,770 ERROR [MemStoreFlusher.0] regionserver.MemStoreFlusher: Cache flush failed for region hbase:meta,,1 org.apache.hadoop.hbase.exceptions.TimeoutIOException: Failed to get sync result after 300000 ms for txid=16, WAL system stuck? at org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:145) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:718) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:586) at org.apache.hadoop.hbase.regionserver.HRegion.doSyncOfUnflushedWALChanges(HRegion.java:2674) at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2612) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2470) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2444) at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2334) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:612) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:581) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$1000(MemStoreFlusher.java:68) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:361) at java.lang.Thread.run(Thread.java:748) 2020-06-19 23:37:09,953 INFO [regionserver/hadoop-hbnn0005:16020.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of hbase:namespace,,1592548148073.f5c7e71fb5e5cab3b27e52600996f7fd. because info has an old edit so flush to free WALs after random delay 169470 ms 2020-06-19 23:37:50,503 WARN [RpcServer.priority.FPBQ.Fifo.handler=18,queue=0,port=16020] ipc.RpcServer: Can not complete this request in time, drop it: callId: 54 service: ClientService methodName: Mutate size: 146 connection: 172.20.101.35:58756 deadline: 1592580830464 param: region= hbase:namespace,,1592548148073.f5c7e71fb5e5cab3b27e52600996f7fd., row=abcd2 connection: 172.20.101.35:58756 2020-06-19 23:38:52,528 WARN [RpcServer.priority.FPBQ.Fifo.handler=19,queue=1,port=16020] ipc.RpcServer: Can not complete this request in time, drop it: callId: 58 service: ClientService methodName: Mutate size: 146 connection: 172.20.101.35:58756 deadline: 1592580892488 param: region= hbase:namespace,,1592548148073.f5c7e71fb5e5cab3b27e52600996f7fd., row=abcd2 connection: 172.20.101.35:58756 2020-06-19 23:39:36,601 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=932.66 KB, freeSize=1.19 GB, max=1.19 GB, blockCount=7, accesses=84, hits=77, hitRatio=91.67%, , cachingAccesses=84, cachingHits=77, cachingHitsRatio=91.67%, evictions=89, evicted=0, evictedPerRun=0.0 2020-06-19 23:39:56,577 WARN [RpcServer.priority.FPBQ.Fifo.handler=17,queue=1,port=16020] ipc.RpcServer: Can not complete this request in time, drop it: callId: 64 service: ClientService methodName: Mutate size: 146 connection: 172.20.101.35:58756 deadline: 1592580956539 param: region= hbase:namespace,,1592548148073.f5c7e71fb5e5cab3b27e52600996f7fd., row=abcd2 connection: 172.20.101.35:58756 2020-06-19 23:44:36,601 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=932.66 KB, freeSize=1.19 GB, max=1.19 GB, blockCount=7, accesses=103, hits=96, hitRatio=93.20%, , cachingAccesses=103, cachingHits=96, cachingHitsRatio=93.20%, evictions=119, evicted=0, evictedPerRun=0.0 2020-06-19 23:44:56,617 ERROR [MemStoreFlusher.1] regionserver.MemStoreFlusher: Cache flush failed for region hbase:namespace,,1592548148073.f5c7e71fb5e5cab3b27e52600996f7fd. org.apache.hadoop.hbase.exceptions.TimeoutIOException: Failed to get sync result after 300000 ms for txid=26, WAL system stuck? at org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:145) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:718) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:586) at org.apache.hadoop.hbase.regionserver.HRegion.doSyncOfUnflushedWALChanges(HRegion.java:2674) at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2612) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2470) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2444) at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2334) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:612) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:581) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$1000(MemStoreFlusher.java:68) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:361) at java.lang.Thread.run(Thread.java:748) 2020-06-19 23:44:56,617 WARN [RpcServer.priority.FPBQ.Fifo.handler=16,queue=0,port=16020] ipc.RpcServer: Can not complete this request in time, drop it: callId: 82 service: ClientService methodName: Mutate size: 146 connection: 172.20.101.35:58756 deadline: 1592581236780 param: region= hbase:namespace,,1592548148073.f5c7e71fb5e5cab3b27e52600996f7fd., row=abcd2 connection: 172.20.101.35:58756 2020-06-19 23:45:56,972 WARN [RpcServer.priority.FPBQ.Fifo.handler=17,queue=1,port=16020] ipc.RpcServer: Can not complete this request in time, drop it: callId: 88 service: ClientService methodName: Mutate size: 146 connection: 172.20.101.35:58756 deadline: 1592581316932 param: region= hbase:namespace,,1592548148073.f5c7e71fb5e5cab3b27e52600996f7fd., row=abcd2 connection: 172.20.101.35:58756 2020-06-19 23:47:17,028 WARN [RpcServer.priority.FPBQ.Fifo.handler=19,queue=1,port=16020] ipc.RpcServer: Can not complete this request in time, drop it: callId: 94 service: ClientService methodName: Mutate size: 146 connection: 172.20.101.35:58756 deadline: 1592581396984 param: region= hbase:namespace,,1592548148073.f5c7e71fb5e5cab3b27e52600996f7fd., row=abcd2 connection: 172.20.101.35:58756 2020-06-19 23:48:37,113 WARN [RpcServer.priority.FPBQ.Fifo.handler=18,queue=0,port=16020] ipc.RpcServer: Can not complete this request in time, drop it: callId: 99 service: ClientService methodName: Mutate size: 146 connection: 172.20.101.35:58756 deadline: 1592581477077 param: region= hbase:namespace,,1592548148073.f5c7e71fb5e5cab3b27e52600996f7fd., row=abcd2 connection: 172.20.101.35:58756 2020-06-19 23:49:36,601 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=932.66 KB, freeSize=1.19 GB, max=1.19 GB, blockCount=7, accesses=126, hits=119, hitRatio=94.44%, , cachingAccesses=126, cachingHits=119, cachingHitsRatio=94.44%, evictions=149, evicted=0, evictedPerRun=0.0 2020-06-19 23:49:57,201 WARN [RpcServer.priority.FPBQ.Fifo.handler=16,queue=0,port=16020] ipc.RpcServer: Can not complete this request in time, drop it: callId: 105 service: ClientService methodName: Mutate size: 146 connection: 172.20.101.35:58756 deadline: 1592581557160 param: region= hbase:namespace,,1592548148073.f5c7e71fb5e5cab3b27e52600996f7fd., row=abcd2 connection: 172.20.101.35:58756 2020-06-19 23:51:17,363 WARN [RpcServer.priority.FPBQ.Fifo.handler=17,queue=1,port=16020] ipc.RpcServer: Can not complete this request in time, drop it: callId: 111 service: ClientService methodName: Mutate size: 146 connection: 172.20.101.35:58756 deadline: 1592581637319 param: region= hbase:namespace,,1592548148073.f5c7e71fb5e5cab3b27e52600996f7fd., row=abcd2 connection: 172.20.101.35:58756 2020-06-19 23:52:37,520 WARN [RpcServer.priority.FPBQ.Fifo.handler=19,queue=1,port=16020] ipc.RpcServer: Can not complete this request in time, drop it: callId: 116 service: ClientService methodName: Mutate size: 145 connection: 172.20.101.35:58756 deadline: 1592581668239 param: region= hbase:namespace,,1592548148073.f5c7e71fb5e5cab3b27e52600996f7fd., row=abcd2 connection: 172.20.101.35:58756 2020-06-19 23:52:48,375 WARN [RpcServer.priority.FPBQ.Fifo.handler=15,queue=1,port=16020] ipc.RpcServer: Can not complete this request in time, drop it: callId: 118 service: ClientService methodName: Mutate size: 146 connection: 172.20.101.35:58756 deadline: 1592581728335 param: region= hbase:namespace,,1592548148073.f5c7e71fb5e5cab3b27e52600996f7fd., row=abcd2 connection: 172.20.101.35:58756 2020-06-19 23:53:48,499 WARN [RpcServer.priority.FPBQ.Fifo.handler=18,queue=0,port=16020] ipc.RpcServer: Can not complete this request in time, drop it: callId: 122 service: ClientService methodName: Mutate size: 146 connection: 172.20.101.35:58756 deadline: 1592581788459 param: region= hbase:namespace,,1592548148073.f5c7e71fb5e5cab3b27e52600996f7fd., row=abcd2 connection: 172.20.101.35:58756