2018-02-04 11:00:17,235 DEBUG [main] hbase.HBaseTestingUtility(354): Setting hbase.rootdir to /home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195 2018-02-04 11:00:17,243 INFO [Time-limited test] hbase.HBaseTestingUtility(960): Starting up minicluster with 1 master(s) and 3 regionserver(s) and 3 datanode(s) 2018-02-04 11:00:17,243 INFO [Time-limited test] hbase.HBaseZKTestingUtility(86): Created new mini-cluster data directory: /home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8, deleteOnExit=true 2018-02-04 11:00:17,243 INFO [Time-limited test] hbase.HBaseTestingUtility(975): STARTING DFS 2018-02-04 11:00:17,243 INFO [Time-limited test] hbase.HBaseTestingUtility(752): Setting test.cache.data to /home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cache_data in system properties and HBase conf 2018-02-04 11:00:17,244 INFO [Time-limited test] hbase.HBaseTestingUtility(752): Setting hadoop.tmp.dir to /home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/hadoop_tmp in system properties and HBase conf 2018-02-04 11:00:17,244 INFO [Time-limited test] hbase.HBaseTestingUtility(752): Setting hadoop.log.dir to /home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/hadoop_logs in system properties and HBase conf 2018-02-04 11:00:17,244 INFO [Time-limited test] hbase.HBaseTestingUtility(752): Setting mapreduce.cluster.local.dir to /home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/mapred_local in system properties and HBase conf 2018-02-04 11:00:17,245 INFO [Time-limited test] hbase.HBaseTestingUtility(752): Setting mapreduce.cluster.temp.dir to /home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/mapred_temp in system properties and HBase conf 2018-02-04 11:00:17,245 INFO [Time-limited test] hbase.HBaseTestingUtility(743): read short circuit is OFF 2018-02-04 11:00:17,384 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2018-02-04 11:00:18,005 DEBUG [Time-limited test] fs.HFileSystem(317): The file system is not a DistributedFileSystem. Skipping on block location reordering Formatting using clusterid: testClusterID 2018-02-04 11:00:18,667 WARN [Time-limited test] blockmanagement.DatanodeManager(302): The given interval for marking stale datanode = 30000, which is larger than heartbeat expire interval 20000. 2018-02-04 11:00:19,447 WARN [Time-limited test] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2018-02-04 11:00:19,687 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2018-02-04 11:00:19,814 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2018-02-04 11:00:19,882 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/zhangduo/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.7.4/hadoop-hdfs-2.7.4-tests.jar!/webapps/hdfs to /tmp/Jetty_localhost_43361_hdfs____.gk1ypz/webapp 2018-02-04 11:00:21,284 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:43361 2018-02-04 11:00:21,292 WARN [Time-limited test] blockmanagement.DatanodeManager(302): The given interval for marking stale datanode = 30000, which is larger than heartbeat expire interval 20000. 2018-02-04 11:00:21,821 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2018-02-04 11:00:21,827 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/zhangduo/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.7.4/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to /tmp/Jetty_localhost_40851_datanode____nvlcxr/webapp 2018-02-04 11:00:22,713 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:40851 2018-02-04 11:00:23,040 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2018-02-04 11:00:23,049 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/zhangduo/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.7.4/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to /tmp/Jetty_localhost_34039_datanode____uajfng/webapp 2018-02-04 11:00:23,411 ERROR [DataNode: [[[DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data1/, [DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data2/]] heartbeating to localhost/127.0.0.1:44491] datanode.DirectoryScanner(430): dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000 2018-02-04 11:00:23,639 INFO [Block report processor] blockmanagement.BlockManager(1933): BLOCK* processReport 0x1867312adc46: from storage DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1 node DatanodeRegistration(127.0.0.1:45015, datanodeUuid=1f27f1f7-7f7e-4fcc-97ef-d20669d8600a, infoPort=36289, infoSecurePort=0, ipcPort=45595, storageInfo=lv=-56;cid=testClusterID;nsid=1720375048;c=0), blocks: 0, hasStaleStorage: true, processing time: 2 msecs 2018-02-04 11:00:23,639 INFO [Block report processor] blockmanagement.BlockManager(1933): BLOCK* processReport 0x1867312adc46: from storage DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619 node DatanodeRegistration(127.0.0.1:45015, datanodeUuid=1f27f1f7-7f7e-4fcc-97ef-d20669d8600a, infoPort=36289, infoSecurePort=0, ipcPort=45595, storageInfo=lv=-56;cid=testClusterID;nsid=1720375048;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs 2018-02-04 11:00:23,945 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:34039 2018-02-04 11:00:24,018 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2018-02-04 11:00:24,025 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/zhangduo/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.7.4/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to /tmp/Jetty_localhost_41633_datanode____.reuty8/webapp 2018-02-04 11:00:24,139 ERROR [DataNode: [[[DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data3/, [DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data4/]] heartbeating to localhost/127.0.0.1:44491] datanode.DirectoryScanner(430): dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000 2018-02-04 11:00:24,151 INFO [Block report processor] blockmanagement.BlockManager(1933): BLOCK* processReport 0x186750536e75: from storage DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33 node DatanodeRegistration(127.0.0.1:33785, datanodeUuid=ec0e09c5-3c3f-4fbb-b904-e820adf8f4b9, infoPort=34547, infoSecurePort=0, ipcPort=33493, storageInfo=lv=-56;cid=testClusterID;nsid=1720375048;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs 2018-02-04 11:00:24,151 INFO [Block report processor] blockmanagement.BlockManager(1933): BLOCK* processReport 0x186750536e75: from storage DS-dd1fcbd6-0c82-4629-949f-0463aa9af693 node DatanodeRegistration(127.0.0.1:33785, datanodeUuid=ec0e09c5-3c3f-4fbb-b904-e820adf8f4b9, infoPort=34547, infoSecurePort=0, ipcPort=33493, storageInfo=lv=-56;cid=testClusterID;nsid=1720375048;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs 2018-02-04 11:00:24,934 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:41633 2018-02-04 11:00:25,080 ERROR [DataNode: [[[DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data5/, [DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data6/]] heartbeating to localhost/127.0.0.1:44491] datanode.DirectoryScanner(430): dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000 2018-02-04 11:00:25,094 INFO [Block report processor] blockmanagement.BlockManager(1933): BLOCK* processReport 0x18678874c9c7: from storage DS-44dd590d-1f51-400a-b8c2-20d260aaa289 node DatanodeRegistration(127.0.0.1:46483, datanodeUuid=321c6636-03c4-4e6f-90e5-eb2f341bbda2, infoPort=35539, infoSecurePort=0, ipcPort=43825, storageInfo=lv=-56;cid=testClusterID;nsid=1720375048;c=0), blocks: 0, hasStaleStorage: true, processing time: 1 msecs 2018-02-04 11:00:25,094 INFO [Block report processor] blockmanagement.BlockManager(1933): BLOCK* processReport 0x18678874c9c7: from storage DS-1fd5fc50-1375-4985-9716-c32596d90f81 node DatanodeRegistration(127.0.0.1:46483, datanodeUuid=321c6636-03c4-4e6f-90e5-eb2f341bbda2, infoPort=35539, infoSecurePort=0, ipcPort=43825, storageInfo=lv=-56;cid=testClusterID;nsid=1720375048;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs 2018-02-04 11:00:25,383 DEBUG [Time-limited test] hbase.HBaseTestingUtility(678): Setting hbase.rootdir to /home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195 2018-02-04 11:00:25,419 ERROR [Time-limited test] server.ZooKeeperServer(472): ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes 2018-02-04 11:00:25,428 INFO [Time-limited test] zookeeper.MiniZooKeeperCluster(281): Started MiniZooKeeperCluster and ran successful 'stat' on client port=56841 2018-02-04 11:00:25,436 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-02-04 11:00:25,438 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-02-04 11:00:25,794 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741825_1001{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW]]} size 7 2018-02-04 11:00:25,795 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741825_1001 size 7 2018-02-04 11:00:25,795 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741825_1001 size 7 2018-02-04 11:00:26,207 INFO [Time-limited test] util.FSUtils(513): Created version file at hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3 with version=8 2018-02-04 11:00:26,207 INFO [Time-limited test] hbase.HBaseTestingUtility(1219): Setting hbase.fs.tmp.dir to hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/hbase-staging 2018-02-04 11:00:26,411 INFO [Time-limited test] metrics.MetricRegistriesLoader(66): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2018-02-04 11:00:26,589 INFO [Time-limited test] client.ConnectionUtils(122): master/ubuntu:0 server-side Connection retries=45 2018-02-04 11:00:26,610 INFO [Time-limited test] ipc.RpcExecutor(148): RpcExecutor default.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=5 2018-02-04 11:00:26,611 INFO [Time-limited test] ipc.RpcExecutor(148): RpcExecutor priority.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=6 2018-02-04 11:00:26,611 INFO [Time-limited test] ipc.RpcExecutor(148): RpcExecutor replication.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=3 2018-02-04 11:00:26,701 INFO [Time-limited test] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.ClientService, hbase.pb.AdminService 2018-02-04 11:00:26,875 DEBUG [Time-limited test] util.ClassSize(229): Using Unsafe to estimate memory layout 2018-02-04 11:00:26,944 INFO [Time-limited test] ipc.NettyRpcServer(109): NettyRpcServer bind to address=/127.0.1.1:39063 2018-02-04 11:00:26,954 INFO [Time-limited test] hfile.CacheConfig(553): Allocating On heap LruBlockCache size=1.04 GB, blockSize=64 KB 2018-02-04 11:00:26,960 INFO [Time-limited test] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=819720, freeSize=1116962296, maxSize=1117782016, heapSize=819720, minSize=1061892928, minFactor=0.95, multiSize=530946464, multiFactor=0.5, singleSize=265473232, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-02-04 11:00:26,961 INFO [Time-limited test] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=819720, freeSize=1116962296, maxSize=1117782016, heapSize=819720, minSize=1061892928, minFactor=0.95, multiSize=530946464, multiFactor=0.5, singleSize=265473232, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-02-04 11:00:26,966 DEBUG [Time-limited test] mob.MobFileCache(123): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2018-02-04 11:00:26,969 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-02-04 11:00:26,972 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-02-04 11:00:27,022 INFO [Time-limited test] zookeeper.RecoverableZooKeeper(106): Process identifier=master:39063 connecting to ZooKeeper ensemble=localhost:56841 2018-02-04 11:00:27,071 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(443): master:390630x0, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-02-04 11:00:27,071 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(506): master:39063-0x1615ec1f6c10000 connected 2018-02-04 11:00:27,104 DEBUG [Time-limited test] zookeeper.ZKUtil(357): master:39063-0x1615ec1f6c10000, quorum=localhost:56841, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-02-04 11:00:27,105 DEBUG [Time-limited test] zookeeper.ZKUtil(357): master:39063-0x1615ec1f6c10000, quorum=localhost:56841, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-02-04 11:00:27,112 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started 5 default.FPBQ.Fifo handlers, queues=1, port=39063 2018-02-04 11:00:27,114 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started 6 priority.FPBQ.Fifo handlers, queues=1, port=39063 2018-02-04 11:00:27,114 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started 3 replication.FPBQ.Fifo handlers, queues=1, port=39063 2018-02-04 11:00:27,247 INFO [Time-limited test] master.HMaster(485): hbase.rootdir=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3, hbase.cluster.distributed=false 2018-02-04 11:00:27,337 INFO [Time-limited test] client.ConnectionUtils(122): regionserver/ubuntu:0 server-side Connection retries=45 2018-02-04 11:00:27,338 INFO [Time-limited test] ipc.RpcExecutor(148): RpcExecutor default.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=5 2018-02-04 11:00:27,338 INFO [Time-limited test] ipc.RpcExecutor(148): RpcExecutor priority.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=6 2018-02-04 11:00:27,339 INFO [Time-limited test] ipc.RpcExecutor(148): RpcExecutor replication.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=3 2018-02-04 11:00:27,342 INFO [Time-limited test] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2018-02-04 11:00:27,343 INFO [Time-limited test] io.ByteBufferPool(83): Created ByteBufferPool with bufferSize : 65536 and maxPoolSize : 320 2018-02-04 11:00:27,346 INFO [Time-limited test] ipc.NettyRpcServer(109): NettyRpcServer bind to address=/127.0.1.1:39267 2018-02-04 11:00:27,347 INFO [Time-limited test] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=819720, freeSize=1116962296, maxSize=1117782016, heapSize=819720, minSize=1061892928, minFactor=0.95, multiSize=530946464, multiFactor=0.5, singleSize=265473232, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-02-04 11:00:27,347 INFO [Time-limited test] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=819720, freeSize=1116962296, maxSize=1117782016, heapSize=819720, minSize=1061892928, minFactor=0.95, multiSize=530946464, multiFactor=0.5, singleSize=265473232, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-02-04 11:00:27,348 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-02-04 11:00:27,351 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-02-04 11:00:27,353 INFO [Time-limited test] zookeeper.RecoverableZooKeeper(106): Process identifier=regionserver:39267 connecting to ZooKeeper ensemble=localhost:56841 2018-02-04 11:00:27,357 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(443): regionserver:392670x0, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-02-04 11:00:27,358 DEBUG [Time-limited test] zookeeper.ZKUtil(357): regionserver:392670x0, quorum=localhost:56841, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-02-04 11:00:27,360 DEBUG [Time-limited test] zookeeper.ZKUtil(357): regionserver:392670x0, quorum=localhost:56841, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-02-04 11:00:27,361 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(506): regionserver:39267-0x1615ec1f6c10001 connected 2018-02-04 11:00:27,367 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started 5 default.FPBQ.Fifo handlers, queues=1, port=39267 2018-02-04 11:00:27,368 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started 6 priority.FPBQ.Fifo handlers, queues=1, port=39267 2018-02-04 11:00:27,370 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started 3 replication.FPBQ.Fifo handlers, queues=1, port=39267 2018-02-04 11:00:27,399 INFO [Time-limited test] client.ConnectionUtils(122): regionserver/ubuntu:0 server-side Connection retries=45 2018-02-04 11:00:27,399 INFO [Time-limited test] ipc.RpcExecutor(148): RpcExecutor default.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=5 2018-02-04 11:00:27,399 INFO [Time-limited test] ipc.RpcExecutor(148): RpcExecutor priority.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=6 2018-02-04 11:00:27,399 INFO [Time-limited test] ipc.RpcExecutor(148): RpcExecutor replication.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=3 2018-02-04 11:00:27,400 INFO [Time-limited test] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2018-02-04 11:00:27,400 INFO [Time-limited test] io.ByteBufferPool(83): Created ByteBufferPool with bufferSize : 65536 and maxPoolSize : 320 2018-02-04 11:00:27,402 INFO [Time-limited test] ipc.NettyRpcServer(109): NettyRpcServer bind to address=/127.0.1.1:46739 2018-02-04 11:00:27,402 INFO [Time-limited test] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=819720, freeSize=1116962296, maxSize=1117782016, heapSize=819720, minSize=1061892928, minFactor=0.95, multiSize=530946464, multiFactor=0.5, singleSize=265473232, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-02-04 11:00:27,403 INFO [Time-limited test] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=819720, freeSize=1116962296, maxSize=1117782016, heapSize=819720, minSize=1061892928, minFactor=0.95, multiSize=530946464, multiFactor=0.5, singleSize=265473232, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-02-04 11:00:27,404 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-02-04 11:00:27,407 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-02-04 11:00:27,408 INFO [Time-limited test] zookeeper.RecoverableZooKeeper(106): Process identifier=regionserver:46739 connecting to ZooKeeper ensemble=localhost:56841 2018-02-04 11:00:27,420 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(443): regionserver:467390x0, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-02-04 11:00:27,421 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(506): regionserver:46739-0x1615ec1f6c10002 connected 2018-02-04 11:00:27,421 DEBUG [Time-limited test] zookeeper.ZKUtil(357): regionserver:46739-0x1615ec1f6c10002, quorum=localhost:56841, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-02-04 11:00:27,422 DEBUG [Time-limited test] zookeeper.ZKUtil(357): regionserver:46739-0x1615ec1f6c10002, quorum=localhost:56841, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-02-04 11:00:27,430 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started 5 default.FPBQ.Fifo handlers, queues=1, port=46739 2018-02-04 11:00:27,439 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started 6 priority.FPBQ.Fifo handlers, queues=1, port=46739 2018-02-04 11:00:27,443 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started 3 replication.FPBQ.Fifo handlers, queues=1, port=46739 2018-02-04 11:00:27,466 INFO [Time-limited test] client.ConnectionUtils(122): regionserver/ubuntu:0 server-side Connection retries=45 2018-02-04 11:00:27,467 INFO [Time-limited test] ipc.RpcExecutor(148): RpcExecutor default.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=5 2018-02-04 11:00:27,467 INFO [Time-limited test] ipc.RpcExecutor(148): RpcExecutor priority.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=6 2018-02-04 11:00:27,467 INFO [Time-limited test] ipc.RpcExecutor(148): RpcExecutor replication.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=3 2018-02-04 11:00:27,468 INFO [Time-limited test] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2018-02-04 11:00:27,468 INFO [Time-limited test] io.ByteBufferPool(83): Created ByteBufferPool with bufferSize : 65536 and maxPoolSize : 320 2018-02-04 11:00:27,471 INFO [Time-limited test] ipc.NettyRpcServer(109): NettyRpcServer bind to address=/127.0.1.1:36383 2018-02-04 11:00:27,472 INFO [Time-limited test] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=819720, freeSize=1116962296, maxSize=1117782016, heapSize=819720, minSize=1061892928, minFactor=0.95, multiSize=530946464, multiFactor=0.5, singleSize=265473232, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-02-04 11:00:27,472 INFO [Time-limited test] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=819720, freeSize=1116962296, maxSize=1117782016, heapSize=819720, minSize=1061892928, minFactor=0.95, multiSize=530946464, multiFactor=0.5, singleSize=265473232, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-02-04 11:00:27,473 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-02-04 11:00:27,475 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-02-04 11:00:27,477 INFO [Time-limited test] zookeeper.RecoverableZooKeeper(106): Process identifier=regionserver:36383 connecting to ZooKeeper ensemble=localhost:56841 2018-02-04 11:00:27,485 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(443): regionserver:363830x0, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-02-04 11:00:27,486 DEBUG [Time-limited test] zookeeper.ZKUtil(357): regionserver:363830x0, quorum=localhost:56841, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-02-04 11:00:27,487 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(506): regionserver:36383-0x1615ec1f6c10003 connected 2018-02-04 11:00:27,487 DEBUG [Time-limited test] zookeeper.ZKUtil(357): regionserver:36383-0x1615ec1f6c10003, quorum=localhost:56841, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-02-04 11:00:27,498 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started 5 default.FPBQ.Fifo handlers, queues=1, port=36383 2018-02-04 11:00:27,504 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started 6 priority.FPBQ.Fifo handlers, queues=1, port=36383 2018-02-04 11:00:27,507 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started 3 replication.FPBQ.Fifo handlers, queues=1, port=36383 2018-02-04 11:00:27,509 INFO [M:0;ubuntu:39063] master.HMaster(2005): Adding backup master ZNode /hbase/backup-masters/ubuntu,39063,1517713226282 2018-02-04 11:00:27,522 DEBUG [M:0;ubuntu:39063] zookeeper.ZKUtil(355): master:39063-0x1615ec1f6c10000, quorum=localhost:56841, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/ubuntu,39063,1517713226282 2018-02-04 11:00:27,549 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(443): regionserver:36383-0x1615ec1f6c10003, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-02-04 11:00:27,549 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(443): master:39063-0x1615ec1f6c10000, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-02-04 11:00:27,549 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(443): regionserver:39267-0x1615ec1f6c10001, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-02-04 11:00:27,549 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(443): regionserver:46739-0x1615ec1f6c10002, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-02-04 11:00:27,550 DEBUG [M:0;ubuntu:39063] zookeeper.ZKUtil(355): master:39063-0x1615ec1f6c10000, quorum=localhost:56841, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-02-04 11:00:27,551 INFO [M:0;ubuntu:39063] master.ActiveMasterManager(172): Deleting ZNode for /hbase/backup-masters/ubuntu,39063,1517713226282 from backup master directory 2018-02-04 11:00:27,556 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:39063-0x1615ec1f6c10000, quorum=localhost:56841, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-02-04 11:00:27,556 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(443): master:39063-0x1615ec1f6c10000, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/ubuntu,39063,1517713226282 2018-02-04 11:00:27,556 WARN [M:0;ubuntu:39063] hbase.ZNodeClearer(62): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-02-04 11:00:27,556 INFO [M:0;ubuntu:39063] master.ActiveMasterManager(181): Registered as active master=ubuntu,39063,1517713226282 2018-02-04 11:00:27,559 INFO [M:0;ubuntu:39063] regionserver.ChunkCreator(408): Allocating MemStoreChunkPool with chunk size=2 MB, max count=533, initial count=0 2018-02-04 11:00:27,664 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|FINALIZED]]} size 0 2018-02-04 11:00:27,664 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|FINALIZED], ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|FINALIZED]]} size 0 2018-02-04 11:00:27,669 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|FINALIZED], ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|FINALIZED]]} size 0 2018-02-04 11:00:27,672 DEBUG [M:0;ubuntu:39063] util.FSUtils(665): Created cluster ID file at hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/hbase.id with ID: 990c040f-15f4-4bef-a30c-08f39bc273d8 2018-02-04 11:00:27,698 INFO [M:0;ubuntu:39063] master.MasterFileSystem(385): BOOTSTRAP: creating hbase:meta region 2018-02-04 11:00:27,703 INFO [M:0;ubuntu:39063] regionserver.HRegion(6792): creating HRegion hbase:meta HTD == 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}, {NAME => 'info', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'false', BLOCKSIZE => '8192'}, {NAME => 'rep_barrier', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'}, {NAME => 'rep_meta', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'}, {NAME => 'rep_position', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'}, {NAME => 'table', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} RootDir = hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3 Table name == hbase:meta 2018-02-04 11:00:27,736 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741827_1003{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW]]} size 0 2018-02-04 11:00:27,739 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741827_1003{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW]]} size 0 2018-02-04 11:00:27,739 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741827_1003{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619:NORMAL:127.0.0.1:45015|FINALIZED]]} size 0 2018-02-04 11:00:27,745 DEBUG [M:0;ubuntu:39063] regionserver.HRegion(809): Instantiated hbase:meta,,1.1588230740 2018-02-04 11:00:27,783 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/hbase/meta/1588230740/info 2018-02-04 11:00:27,803 INFO [StoreOpener-1588230740-1] regionserver.HStore(294): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-02-04 11:00:27,810 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=0, currentSize=819720, freeSize=1116962296, maxSize=1117782016, heapSize=819720, minSize=1061892928, minFactor=0.95, multiSize=530946464, multiFactor=0.5, singleSize=265473232, singleFactor=0.25}, cacheDataOnRead=false, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-02-04 11:00:27,819 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-02-04 11:00:27,847 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/hbase/meta/1588230740/rep_barrier 2018-02-04 11:00:27,847 INFO [StoreOpener-1588230740-1] regionserver.HStore(294): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-02-04 11:00:27,848 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_barrier: blockCache=LruBlockCache{blockCount=0, currentSize=819720, freeSize=1116962296, maxSize=1117782016, heapSize=819720, minSize=1061892928, minFactor=0.95, multiSize=530946464, multiFactor=0.5, singleSize=265473232, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-02-04 11:00:27,849 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-02-04 11:00:27,865 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/hbase/meta/1588230740/rep_meta 2018-02-04 11:00:27,865 INFO [StoreOpener-1588230740-1] regionserver.HStore(294): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-02-04 11:00:27,865 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_meta: blockCache=LruBlockCache{blockCount=0, currentSize=819720, freeSize=1116962296, maxSize=1117782016, heapSize=819720, minSize=1061892928, minFactor=0.95, multiSize=530946464, multiFactor=0.5, singleSize=265473232, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-02-04 11:00:27,865 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-02-04 11:00:27,871 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/hbase/meta/1588230740/rep_position 2018-02-04 11:00:27,872 INFO [StoreOpener-1588230740-1] regionserver.HStore(294): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-02-04 11:00:27,872 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_position: blockCache=LruBlockCache{blockCount=0, currentSize=819720, freeSize=1116962296, maxSize=1117782016, heapSize=819720, minSize=1061892928, minFactor=0.95, multiSize=530946464, multiFactor=0.5, singleSize=265473232, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-02-04 11:00:27,873 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-02-04 11:00:27,883 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/hbase/meta/1588230740/table 2018-02-04 11:00:27,884 INFO [StoreOpener-1588230740-1] regionserver.HStore(294): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-02-04 11:00:27,886 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for table: blockCache=LruBlockCache{blockCount=0, currentSize=819720, freeSize=1116962296, maxSize=1117782016, heapSize=819720, minSize=1061892928, minFactor=0.95, multiSize=530946464, multiFactor=0.5, singleSize=265473232, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-02-04 11:00:27,888 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-02-04 11:00:27,901 DEBUG [M:0;ubuntu:39063] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/hbase/meta/1588230740 2018-02-04 11:00:27,915 DEBUG [M:0;ubuntu:39063] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushSize/# of families (25.6M)) instead. 2018-02-04 11:00:27,921 DEBUG [M:0;ubuntu:39063] wal.WALSplitter(675): Wrote file=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/hbase/meta/1588230740/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-02-04 11:00:27,921 INFO [M:0;ubuntu:39063] regionserver.HRegion(962): Onlined 1588230740; next sequenceid=2 2018-02-04 11:00:27,921 DEBUG [M:0;ubuntu:39063] regionserver.HRegion(1483): Closing hbase:meta,,1.1588230740: disabling compactions & flushes 2018-02-04 11:00:27,921 DEBUG [M:0;ubuntu:39063] regionserver.HRegion(1522): Updates disabled for region hbase:meta,,1.1588230740 2018-02-04 11:00:27,922 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(932): Closed info 2018-02-04 11:00:27,923 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(932): Closed rep_barrier 2018-02-04 11:00:27,924 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(932): Closed rep_meta 2018-02-04 11:00:27,924 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(932): Closed rep_position 2018-02-04 11:00:27,924 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(932): Closed table 2018-02-04 11:00:27,924 INFO [M:0;ubuntu:39063] regionserver.HRegion(1635): Closed hbase:meta,,1.1588230740 2018-02-04 11:00:27,975 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741828_1004{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|RBW]]} size 0 2018-02-04 11:00:27,977 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741828_1004{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|FINALIZED]]} size 0 2018-02-04 11:00:27,978 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741828_1004{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|FINALIZED]]} size 0 2018-02-04 11:00:27,983 DEBUG [M:0;ubuntu:39063] util.FSTableDescriptors(716): Wrote into hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2018-02-04 11:00:28,019 INFO [M:0;ubuntu:39063] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-02-04 11:00:28,030 INFO [M:0;ubuntu:39063] coordination.ZKSplitLogManagerCoordination(492): Found 0 orphan tasks and 0 rescan nodes 2018-02-04 11:00:28,057 INFO [M:0;ubuntu:39063] zookeeper.ReadOnlyZKClient(133): Connect 0x7d1e2e0a to localhost:56841 with session timeout=1000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2018-02-04 11:00:28,089 DEBUG [M:0;ubuntu:39063] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@23b6b302, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=1, fallbackAllowed=true, bind address=null 2018-02-04 11:00:28,130 INFO [M:0;ubuntu:39063] balancer.BaseLoadBalancer(1037): slop=0.001, tablesOnMaster=false, systemTablesOnMaster=false 2018-02-04 11:00:28,135 INFO [M:0;ubuntu:39063] balancer.StochasticLoadBalancer(214): Loaded config; maxSteps=1000000, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, etc. 2018-02-04 11:00:28,137 DEBUG [M:0;ubuntu:39063] zookeeper.ZKUtil(357): master:39063-0x1615ec1f6c10000, quorum=localhost:56841, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2018-02-04 11:00:28,139 DEBUG [M:0;ubuntu:39063] zookeeper.ZKUtil(357): master:39063-0x1615ec1f6c10000, quorum=localhost:56841, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2018-02-04 11:00:28,145 DEBUG [M:0;ubuntu:39063] zookeeper.ZKUtil(357): master:39063-0x1615ec1f6c10000, quorum=localhost:56841, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2018-02-04 11:00:28,145 DEBUG [M:0;ubuntu:39063] zookeeper.ZKUtil(357): master:39063-0x1615ec1f6c10000, quorum=localhost:56841, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2018-02-04 11:00:28,192 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(443): regionserver:46739-0x1615ec1f6c10002, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-02-04 11:00:28,194 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(443): regionserver:36383-0x1615ec1f6c10003, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-02-04 11:00:28,194 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(443): master:39063-0x1615ec1f6c10000, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-02-04 11:00:28,195 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(443): regionserver:39267-0x1615ec1f6c10001, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-02-04 11:00:28,199 INFO [M:0;ubuntu:39063] master.HMaster(757): Active/primary master=ubuntu,39063,1517713226282, sessionid=0x1615ec1f6c10000, setting cluster-up flag (Was=false) 2018-02-04 11:00:28,226 DEBUG [M:0;ubuntu:39063] procedure.ZKProcedureUtil(272): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2018-02-04 11:00:28,229 DEBUG [M:0;ubuntu:39063] procedure.ZKProcedureCoordinator(250): Starting controller for procedure member=ubuntu,39063,1517713226282 2018-02-04 11:00:28,243 DEBUG [M:0;ubuntu:39063] procedure.ZKProcedureUtil(272): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2018-02-04 11:00:28,245 DEBUG [M:0;ubuntu:39063] procedure.ZKProcedureCoordinator(250): Starting controller for procedure member=ubuntu,39063,1517713226282 2018-02-04 11:00:28,287 DEBUG [M:0;ubuntu:39063] executor.ExecutorService(92): Starting executor service name=MASTER_OPEN_REGION-master/ubuntu:0, corePoolSize=5, maxPoolSize=5 2018-02-04 11:00:28,287 DEBUG [M:0;ubuntu:39063] executor.ExecutorService(92): Starting executor service name=MASTER_CLOSE_REGION-master/ubuntu:0, corePoolSize=5, maxPoolSize=5 2018-02-04 11:00:28,288 DEBUG [M:0;ubuntu:39063] executor.ExecutorService(92): Starting executor service name=MASTER_SERVER_OPERATIONS-master/ubuntu:0, corePoolSize=5, maxPoolSize=5 2018-02-04 11:00:28,288 DEBUG [M:0;ubuntu:39063] executor.ExecutorService(92): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/ubuntu:0, corePoolSize=5, maxPoolSize=5 2018-02-04 11:00:28,289 DEBUG [M:0;ubuntu:39063] executor.ExecutorService(92): Starting executor service name=M_LOG_REPLAY_OPS-master/ubuntu:0, corePoolSize=10, maxPoolSize=10 2018-02-04 11:00:28,289 DEBUG [M:0;ubuntu:39063] executor.ExecutorService(92): Starting executor service name=MASTER_TABLE_OPERATIONS-master/ubuntu:0, corePoolSize=1, maxPoolSize=1 2018-02-04 11:00:28,311 INFO [M:0;ubuntu:39063] procedure2.ProcedureExecutor(512): Starting 16 Workers (bigger of cpus/4 or 16) 2018-02-04 11:00:28,324 INFO [RS:0;ubuntu:39267] regionserver.HRegionServer(870): ClusterId : 990c040f-15f4-4bef-a30c-08f39bc273d8 2018-02-04 11:00:28,325 INFO [RS:1;ubuntu:46739] regionserver.HRegionServer(870): ClusterId : 990c040f-15f4-4bef-a30c-08f39bc273d8 2018-02-04 11:00:28,328 INFO [RS:2;ubuntu:36383] regionserver.HRegionServer(870): ClusterId : 990c040f-15f4-4bef-a30c-08f39bc273d8 2018-02-04 11:00:28,340 DEBUG [RS:1;ubuntu:46739] procedure.RegionServerProcedureManagerHost(44): Procedure flush-table-proc is initializing 2018-02-04 11:00:28,340 DEBUG [RS:0;ubuntu:39267] procedure.RegionServerProcedureManagerHost(44): Procedure flush-table-proc is initializing 2018-02-04 11:00:28,341 DEBUG [RS:2;ubuntu:36383] procedure.RegionServerProcedureManagerHost(44): Procedure flush-table-proc is initializing 2018-02-04 11:00:28,348 WARN [M:0;ubuntu:39063] util.CommonFSUtils$StreamCapabilities(830): Your Hadoop installation does not include the StreamCapabilities class from HDFS-11644, so we will skip checking if any FSDataOutputStreams actually support hflush/hsync. If you are running on top of HDFS this probably just means you have an older version and this can be ignored. If you are running on top of an alternate FileSystem implementation you should manually verify that hflush and hsync are implemented; otherwise you risk data loss and hard to diagnose errors when our assumptions are violated. 2018-02-04 11:00:28,350 DEBUG [M:0;ubuntu:39063] util.CommonFSUtils$StreamCapabilities(837): The first request to check for StreamCapabilities came from this stacktrace. java.lang.ClassNotFoundException: org.apache.hadoop.fs.StreamCapabilities at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.util.CommonFSUtils$StreamCapabilities.(CommonFSUtils.java:826) at org.apache.hadoop.hbase.util.CommonFSUtils.hasCapability(CommonFSUtils.java:864) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:1033) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.recoverLease(WALProcedureStore.java:374) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.start(ProcedureExecutor.java:531) at org.apache.hadoop.hbase.master.HMaster.startProcedureExecutor(HMaster.java:1230) at org.apache.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:1153) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:851) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2025) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:561) at java.lang.Thread.run(Thread.java:748) 2018-02-04 11:00:28,354 DEBUG [RS:0;ubuntu:39267] procedure.RegionServerProcedureManagerHost(46): Procedure flush-table-proc is initialized 2018-02-04 11:00:28,355 DEBUG [RS:0;ubuntu:39267] procedure.RegionServerProcedureManagerHost(44): Procedure online-snapshot is initializing 2018-02-04 11:00:28,355 DEBUG [RS:2;ubuntu:36383] procedure.RegionServerProcedureManagerHost(46): Procedure flush-table-proc is initialized 2018-02-04 11:00:28,355 DEBUG [RS:2;ubuntu:36383] procedure.RegionServerProcedureManagerHost(44): Procedure online-snapshot is initializing 2018-02-04 11:00:28,354 DEBUG [RS:1;ubuntu:46739] procedure.RegionServerProcedureManagerHost(46): Procedure flush-table-proc is initialized 2018-02-04 11:00:28,356 DEBUG [RS:1;ubuntu:46739] procedure.RegionServerProcedureManagerHost(44): Procedure online-snapshot is initializing 2018-02-04 11:00:28,357 DEBUG [M:0;ubuntu:39063] wal.WALProcedureStore(1069): Roll new state log: 1 2018-02-04 11:00:28,359 INFO [M:0;ubuntu:39063] procedure2.ProcedureExecutor(533): Recovered WALProcedureStore lease in 30msec 2018-02-04 11:00:28,360 INFO [M:0;ubuntu:39063] procedure2.ProcedureExecutor(547): Loaded WALProcedureStore in 1msec, um pid= 2018-02-04 11:00:28,361 DEBUG [RS:2;ubuntu:36383] procedure.RegionServerProcedureManagerHost(46): Procedure online-snapshot is initialized 2018-02-04 11:00:28,361 DEBUG [RS:1;ubuntu:46739] procedure.RegionServerProcedureManagerHost(46): Procedure online-snapshot is initialized 2018-02-04 11:00:28,361 DEBUG [RS:0;ubuntu:39267] procedure.RegionServerProcedureManagerHost(46): Procedure online-snapshot is initialized 2018-02-04 11:00:28,364 INFO [RS:1;ubuntu:46739] zookeeper.ReadOnlyZKClient(133): Connect 0x702c28fc to localhost:56841 with session timeout=1000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2018-02-04 11:00:28,368 INFO [M:0;ubuntu:39063] procedure2.ProcedureExecutor$TimeoutExecutorThread(1827): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.ProcedureExecutor$CompletedProcedureCleaner; timeout=30000, timestamp=1517713258368 2018-02-04 11:00:28,368 INFO [RS:2;ubuntu:36383] zookeeper.ReadOnlyZKClient(133): Connect 0x1c128423 to localhost:56841 with session timeout=1000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2018-02-04 11:00:28,369 INFO [M:0;ubuntu:39063] procedure2.RemoteProcedureDispatcher(97): Started, threads=128, queueMaxSize=32, operationDelay=150 2018-02-04 11:00:28,370 INFO [RS:0;ubuntu:39267] zookeeper.ReadOnlyZKClient(133): Connect 0x7f64e850 to localhost:56841 with session timeout=1000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2018-02-04 11:00:28,380 DEBUG [M:0;ubuntu:39063] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2018-02-04 11:00:28,388 INFO [M:0;ubuntu:39063] zookeeper.RecoverableZooKeeper(106): Process identifier=replicationLogCleaner connecting to ZooKeeper ensemble=localhost:56841 2018-02-04 11:00:28,391 DEBUG [M:0;ubuntu:39063] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2018-02-04 11:00:28,392 DEBUG [RS:1;ubuntu:46739] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7eebd889, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=1, fallbackAllowed=true, bind address=null 2018-02-04 11:00:28,394 DEBUG [RS:2;ubuntu:36383] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@12d579d1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=1, fallbackAllowed=true, bind address=null 2018-02-04 11:00:28,394 DEBUG [RS:1;ubuntu:46739] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7ad4438c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=1, fallbackAllowed=true, bind address=ubuntu/127.0.1.1:0 2018-02-04 11:00:28,394 DEBUG [RS:2;ubuntu:36383] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@16eaef53, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=1, fallbackAllowed=true, bind address=ubuntu/127.0.1.1:0 2018-02-04 11:00:28,394 DEBUG [RS:0;ubuntu:39267] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@71a309cc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=1, fallbackAllowed=true, bind address=null 2018-02-04 11:00:28,395 DEBUG [RS:0;ubuntu:39267] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@759c8320, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=1, fallbackAllowed=true, bind address=ubuntu/127.0.1.1:0 2018-02-04 11:00:28,395 DEBUG [M:0;ubuntu:39063-EventThread] zookeeper.ZKWatcher(443): replicationLogCleaner0x0, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-02-04 11:00:28,394 DEBUG [M:0;ubuntu:39063] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2018-02-04 11:00:28,397 INFO [M:0;ubuntu:39063] cleaner.CleanerChore(113): Cleaner pool size is 1 2018-02-04 11:00:28,396 DEBUG [M:0;ubuntu:39063-EventThread] zookeeper.ZKWatcher(506): replicationLogCleaner-0x1615ec1f6c10008 connected 2018-02-04 11:00:28,401 INFO [M:0;ubuntu:39063] cleaner.LogCleaner(122): Creating OldWALs cleaners with size=2 2018-02-04 11:00:28,399 DEBUG [RS:2;ubuntu:36383] regionserver.ShutdownHook(88): Installed shutdown hook thread: Shutdownhook:RS:2;ubuntu:36383 2018-02-04 11:00:28,398 DEBUG [RS:1;ubuntu:46739] regionserver.ShutdownHook(88): Installed shutdown hook thread: Shutdownhook:RS:1;ubuntu:46739 2018-02-04 11:00:28,398 DEBUG [RS:0;ubuntu:39267] regionserver.ShutdownHook(88): Installed shutdown hook thread: Shutdownhook:RS:0;ubuntu:39267 2018-02-04 11:00:28,406 INFO [RS:2;ubuntu:36383] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2018-02-04 11:00:28,407 INFO [RS:0;ubuntu:39267] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2018-02-04 11:00:28,407 INFO [RS:0;ubuntu:39267] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2018-02-04 11:00:28,406 INFO [RS:1;ubuntu:46739] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2018-02-04 11:00:28,408 INFO [RS:1;ubuntu:46739] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2018-02-04 11:00:28,407 INFO [RS:2;ubuntu:36383] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2018-02-04 11:00:28,408 DEBUG [M:0;ubuntu:39063] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2018-02-04 11:00:28,410 INFO [RS:0;ubuntu:39267] regionserver.HRegionServer(2581): reportForDuty to master=ubuntu,39063,1517713226282 with port=39267, startcode=1517713227316 2018-02-04 11:00:28,410 INFO [RS:2;ubuntu:36383] regionserver.HRegionServer(2581): reportForDuty to master=ubuntu,39063,1517713226282 with port=36383, startcode=1517713227466 2018-02-04 11:00:28,410 INFO [RS:1;ubuntu:46739] regionserver.HRegionServer(2581): reportForDuty to master=ubuntu,39063,1517713226282 with port=46739, startcode=1517713227398 2018-02-04 11:00:28,411 DEBUG [M:0;ubuntu:39063] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2018-02-04 11:00:28,411 DEBUG [M:0;ubuntu:39063] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2018-02-04 11:00:28,419 DEBUG [M:0;ubuntu:39063] cleaner.HFileCleaner(207): Starting for large file=Thread[M:0;ubuntu:39063-HFileCleaner.large.0-1517713228419,5,FailOnTimeoutGroup] 2018-02-04 11:00:28,423 DEBUG [M:0;ubuntu:39063] cleaner.HFileCleaner(222): Starting for small files=Thread[M:0;ubuntu:39063-HFileCleaner.small.0-1517713228420,5,FailOnTimeoutGroup] 2018-02-04 11:00:28,433 INFO [M:0;ubuntu:39063] master.HMaster(861): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1517713227545, completionTime=-1 2018-02-04 11:00:28,433 INFO [M:0;ubuntu:39063] master.ServerManager(866): Waiting on regionserver count=0; waited=0ms, expecting min=3 server(s), max=3 server(s), timeout=4500ms, lastChange=0ms 2018-02-04 11:00:28,450 DEBUG [RS:2;ubuntu:36383] ipc.NettyRpcConnection(254): Connecting to ubuntu/127.0.1.1:39063 2018-02-04 11:00:28,450 DEBUG [RS:1;ubuntu:46739] ipc.NettyRpcConnection(254): Connecting to ubuntu/127.0.1.1:39063 2018-02-04 11:00:28,450 DEBUG [RS:0;ubuntu:39267] ipc.NettyRpcConnection(254): Connecting to ubuntu/127.0.1.1:39063 2018-02-04 11:00:28,514 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(528): Auth successful for zhangduo.hfs.2 (auth:SIMPLE) 2018-02-04 11:00:28,514 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(528): Auth successful for zhangduo.hfs.0 (auth:SIMPLE) 2018-02-04 11:00:28,514 INFO [RS-EventLoopGroup-1-4] ipc.ServerRpcConnection(528): Auth successful for zhangduo.hfs.1 (auth:SIMPLE) 2018-02-04 11:00:28,520 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(558): Connection from 127.0.1.1 port: 42301 with version info: version: "3.0.0-SNAPSHOT" url: "git://ubuntu/home/zhangduo/hbase/code" revision: "7fd9ea800d3ec8bfbba59879d0c98e0268d0ccde" user: "zhangduo" date: "Sat Feb 3 18:15:39 CST 2018" src_checksum: "e1f78921fcd876d508017ada55edc99a" version_major: 3 version_minor: 0 2018-02-04 11:00:28,525 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(558): Connection from 127.0.1.1 port: 44211 with version info: version: "3.0.0-SNAPSHOT" url: "git://ubuntu/home/zhangduo/hbase/code" revision: "7fd9ea800d3ec8bfbba59879d0c98e0268d0ccde" user: "zhangduo" date: "Sat Feb 3 18:15:39 CST 2018" src_checksum: "e1f78921fcd876d508017ada55edc99a" version_major: 3 version_minor: 0 2018-02-04 11:00:28,531 INFO [RS-EventLoopGroup-1-4] ipc.ServerRpcConnection(558): Connection from 127.0.1.1 port: 52601 with version info: version: "3.0.0-SNAPSHOT" url: "git://ubuntu/home/zhangduo/hbase/code" revision: "7fd9ea800d3ec8bfbba59879d0c98e0268d0ccde" user: "zhangduo" date: "Sat Feb 3 18:15:39 CST 2018" src_checksum: "e1f78921fcd876d508017ada55edc99a" version_major: 3 version_minor: 0 2018-02-04 11:00:28,558 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39063] master.ServerManager(427): Registering regionserver=ubuntu,36383,1517713227466 2018-02-04 11:00:28,560 INFO [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=39063] master.ServerManager(427): Registering regionserver=ubuntu,46739,1517713227398 2018-02-04 11:00:28,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39063] master.ServerManager(427): Registering regionserver=ubuntu,39267,1517713227316 2018-02-04 11:00:28,579 DEBUG [RS:2;ubuntu:36383] regionserver.HRegionServer(1489): Config from master: hbase.rootdir=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3 2018-02-04 11:00:28,579 DEBUG [RS:1;ubuntu:46739] regionserver.HRegionServer(1489): Config from master: hbase.rootdir=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3 2018-02-04 11:00:28,580 DEBUG [RS:2;ubuntu:36383] regionserver.HRegionServer(1489): Config from master: fs.defaultFS=hdfs://localhost:44491 2018-02-04 11:00:28,580 DEBUG [RS:1;ubuntu:46739] regionserver.HRegionServer(1489): Config from master: fs.defaultFS=hdfs://localhost:44491 2018-02-04 11:00:28,580 DEBUG [RS:1;ubuntu:46739] regionserver.HRegionServer(1489): Config from master: hbase.master.info.port=-1 2018-02-04 11:00:28,580 DEBUG [RS:2;ubuntu:36383] regionserver.HRegionServer(1489): Config from master: hbase.master.info.port=-1 2018-02-04 11:00:28,580 DEBUG [RS:0;ubuntu:39267] regionserver.HRegionServer(1489): Config from master: hbase.rootdir=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3 2018-02-04 11:00:28,584 DEBUG [RS:0;ubuntu:39267] regionserver.HRegionServer(1489): Config from master: fs.defaultFS=hdfs://localhost:44491 2018-02-04 11:00:28,584 DEBUG [RS:0;ubuntu:39267] regionserver.HRegionServer(1489): Config from master: hbase.master.info.port=-1 2018-02-04 11:00:28,585 INFO [M:0;ubuntu:39063] master.ServerManager(886): Finished waiting on RegionServer count=3; waited=152ms, expected min=3 server(s), max=3 server(s), master is running 2018-02-04 11:00:28,593 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(443): master:39063-0x1615ec1f6c10000, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-02-04 11:00:28,594 DEBUG [RS:2;ubuntu:36383] zookeeper.ZKUtil(355): regionserver:36383-0x1615ec1f6c10003, quorum=localhost:56841, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ubuntu,36383,1517713227466 2018-02-04 11:00:28,594 DEBUG [RS:1;ubuntu:46739] zookeeper.ZKUtil(355): regionserver:46739-0x1615ec1f6c10002, quorum=localhost:56841, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ubuntu,46739,1517713227398 2018-02-04 11:00:28,594 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:39063-0x1615ec1f6c10000, quorum=localhost:56841, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ubuntu,36383,1517713227466 2018-02-04 11:00:28,594 WARN [RS:2;ubuntu:36383] hbase.ZNodeClearer(62): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-02-04 11:00:28,595 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:39063-0x1615ec1f6c10000, quorum=localhost:56841, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ubuntu,46739,1517713227398 2018-02-04 11:00:28,595 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:39063-0x1615ec1f6c10000, quorum=localhost:56841, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ubuntu,39267,1517713227316 2018-02-04 11:00:28,594 DEBUG [RS:0;ubuntu:39267] zookeeper.ZKUtil(355): regionserver:39267-0x1615ec1f6c10001, quorum=localhost:56841, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ubuntu,39267,1517713227316 2018-02-04 11:00:28,598 INFO [M:0;ubuntu:39063] master.HMaster(3521): Running RecoverMetaProcedure to ensure proper hbase:meta deploy. 2018-02-04 11:00:28,594 WARN [RS:1;ubuntu:46739] hbase.ZNodeClearer(62): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-02-04 11:00:28,597 WARN [RS:0;ubuntu:39267] hbase.ZNodeClearer(62): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-02-04 11:00:28,601 INFO [RS:0;ubuntu:39267] wal.WALFactory(135): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-02-04 11:00:28,602 INFO [RS:1;ubuntu:46739] wal.WALFactory(135): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-02-04 11:00:28,601 INFO [RS:2;ubuntu:36383] wal.WALFactory(135): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-02-04 11:00:28,611 DEBUG [RS:1;ubuntu:46739] regionserver.HRegionServer(1795): logDir=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/WALs/ubuntu,46739,1517713227398 2018-02-04 11:00:28,611 DEBUG [RS:0;ubuntu:39267] regionserver.HRegionServer(1795): logDir=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/WALs/ubuntu,39267,1517713227316 2018-02-04 11:00:28,613 DEBUG [RS:2;ubuntu:36383] regionserver.HRegionServer(1795): logDir=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/WALs/ubuntu,36383,1517713227466 2018-02-04 11:00:28,654 DEBUG [RS:1;ubuntu:46739] zookeeper.ZKUtil(355): regionserver:46739-0x1615ec1f6c10002, quorum=localhost:56841, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ubuntu,36383,1517713227466 2018-02-04 11:00:28,654 DEBUG [RS:2;ubuntu:36383] zookeeper.ZKUtil(355): regionserver:36383-0x1615ec1f6c10003, quorum=localhost:56841, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ubuntu,36383,1517713227466 2018-02-04 11:00:28,655 DEBUG [RS:1;ubuntu:46739] zookeeper.ZKUtil(355): regionserver:46739-0x1615ec1f6c10002, quorum=localhost:56841, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ubuntu,46739,1517713227398 2018-02-04 11:00:28,655 DEBUG [RS:2;ubuntu:36383] zookeeper.ZKUtil(355): regionserver:36383-0x1615ec1f6c10003, quorum=localhost:56841, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ubuntu,46739,1517713227398 2018-02-04 11:00:28,655 DEBUG [RS:1;ubuntu:46739] zookeeper.ZKUtil(355): regionserver:46739-0x1615ec1f6c10002, quorum=localhost:56841, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ubuntu,39267,1517713227316 2018-02-04 11:00:28,658 DEBUG [RS:2;ubuntu:36383] zookeeper.ZKUtil(355): regionserver:36383-0x1615ec1f6c10003, quorum=localhost:56841, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ubuntu,39267,1517713227316 2018-02-04 11:00:28,658 DEBUG [RS:0;ubuntu:39267] zookeeper.ZKUtil(355): regionserver:39267-0x1615ec1f6c10001, quorum=localhost:56841, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ubuntu,36383,1517713227466 2018-02-04 11:00:28,659 DEBUG [RS:0;ubuntu:39267] zookeeper.ZKUtil(355): regionserver:39267-0x1615ec1f6c10001, quorum=localhost:56841, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ubuntu,46739,1517713227398 2018-02-04 11:00:28,659 DEBUG [RS:0;ubuntu:39267] zookeeper.ZKUtil(355): regionserver:39267-0x1615ec1f6c10001, quorum=localhost:56841, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ubuntu,39267,1517713227316 2018-02-04 11:00:28,665 INFO [RS:1;ubuntu:46739] zookeeper.ReadOnlyZKClient(133): Connect 0x46e70b17 to localhost:56841 with session timeout=1000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2018-02-04 11:00:28,665 INFO [RS:2;ubuntu:36383] zookeeper.ReadOnlyZKClient(133): Connect 0x796736be to localhost:56841 with session timeout=1000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2018-02-04 11:00:28,666 INFO [RS:0;ubuntu:39267] zookeeper.ReadOnlyZKClient(133): Connect 0x143faef3 to localhost:56841 with session timeout=1000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2018-02-04 11:00:28,696 DEBUG [RS:0;ubuntu:39267] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6d9e83ef, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=1, fallbackAllowed=true, bind address=null 2018-02-04 11:00:28,697 DEBUG [RS:1;ubuntu:46739] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@72890e82, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=1, fallbackAllowed=true, bind address=null 2018-02-04 11:00:28,697 DEBUG [RS:0;ubuntu:39267] regionserver.Replication(150): ReplicationStatisticsThread 300 2018-02-04 11:00:28,697 DEBUG [RS:1;ubuntu:46739] regionserver.Replication(150): ReplicationStatisticsThread 300 2018-02-04 11:00:28,702 DEBUG [RS:2;ubuntu:36383] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@21acc0ba, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=1, fallbackAllowed=true, bind address=null 2018-02-04 11:00:28,703 DEBUG [RS:2;ubuntu:36383] regionserver.Replication(150): ReplicationStatisticsThread 300 2018-02-04 11:00:28,712 INFO [RS:0;ubuntu:39267] regionserver.MetricsRegionServerWrapperImpl(144): Computing regionserver metrics every 5000 milliseconds 2018-02-04 11:00:28,712 INFO [RS:1;ubuntu:46739] regionserver.MetricsRegionServerWrapperImpl(144): Computing regionserver metrics every 5000 milliseconds 2018-02-04 11:00:28,712 INFO [RS:2;ubuntu:36383] regionserver.MetricsRegionServerWrapperImpl(144): Computing regionserver metrics every 5000 milliseconds 2018-02-04 11:00:28,767 INFO [RS:1;ubuntu:46739] regionserver.MemStoreFlusher(105): globalMemStoreLimit=1.0 G, globalMemStoreLimitLowMark=1012.7 M, Offheap=false 2018-02-04 11:00:28,770 INFO [RS:0;ubuntu:39267] regionserver.MemStoreFlusher(105): globalMemStoreLimit=1.0 G, globalMemStoreLimitLowMark=1012.7 M, Offheap=false 2018-02-04 11:00:28,769 INFO [RS:2;ubuntu:36383] regionserver.MemStoreFlusher(105): globalMemStoreLimit=1.0 G, globalMemStoreLimitLowMark=1012.7 M, Offheap=false 2018-02-04 11:00:28,781 INFO [RS:0;ubuntu:39267] throttle.PressureAwareCompactionThroughputController(134): Compaction throughput configurations, higher bound: 20.00 MB/sec, lower bound 10.00 MB/sec, off peak: unlimited, tuning period: 60000 ms 2018-02-04 11:00:28,781 INFO [RS:2;ubuntu:36383] throttle.PressureAwareCompactionThroughputController(134): Compaction throughput configurations, higher bound: 20.00 MB/sec, lower bound 10.00 MB/sec, off peak: unlimited, tuning period: 60000 ms 2018-02-04 11:00:28,781 INFO [RS:1;ubuntu:46739] throttle.PressureAwareCompactionThroughputController(134): Compaction throughput configurations, higher bound: 20.00 MB/sec, lower bound 10.00 MB/sec, off peak: unlimited, tuning period: 60000 ms 2018-02-04 11:00:28,786 INFO [RS:0;ubuntu:39267] regionserver.HRegionServer$CompactionChecker(1689): CompactionChecker runs every PT1S 2018-02-04 11:00:28,787 INFO [RS:2;ubuntu:36383] regionserver.HRegionServer$CompactionChecker(1689): CompactionChecker runs every PT1S 2018-02-04 11:00:28,787 INFO [RS:1;ubuntu:46739] regionserver.HRegionServer$CompactionChecker(1689): CompactionChecker runs every PT1S 2018-02-04 11:00:28,809 DEBUG [M:0;ubuntu:39063] procedure2.ProcedureExecutor(868): Stored pid=1, state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-02-04 11:00:28,815 DEBUG [RS:2;ubuntu:36383] executor.ExecutorService(92): Starting executor service name=RS_OPEN_REGION-regionserver/ubuntu:0, corePoolSize=3, maxPoolSize=3 2018-02-04 11:00:28,815 DEBUG [RS:2;ubuntu:36383] executor.ExecutorService(92): Starting executor service name=RS_OPEN_META-regionserver/ubuntu:0, corePoolSize=1, maxPoolSize=1 2018-02-04 11:00:28,815 DEBUG [RS:2;ubuntu:36383] executor.ExecutorService(92): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/ubuntu:0, corePoolSize=3, maxPoolSize=3 2018-02-04 11:00:28,815 DEBUG [RS:2;ubuntu:36383] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_REGION-regionserver/ubuntu:0, corePoolSize=3, maxPoolSize=3 2018-02-04 11:00:28,816 DEBUG [RS:2;ubuntu:36383] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_META-regionserver/ubuntu:0, corePoolSize=1, maxPoolSize=1 2018-02-04 11:00:28,816 DEBUG [RS:2;ubuntu:36383] executor.ExecutorService(92): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/ubuntu:0, corePoolSize=2, maxPoolSize=2 2018-02-04 11:00:28,816 DEBUG [RS:2;ubuntu:36383] executor.ExecutorService(92): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/ubuntu:0, corePoolSize=10, maxPoolSize=10 2018-02-04 11:00:28,816 DEBUG [RS:2;ubuntu:36383] executor.ExecutorService(92): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/ubuntu:0, corePoolSize=3, maxPoolSize=3 2018-02-04 11:00:28,816 DEBUG [RS:2;ubuntu:36383] executor.ExecutorService(92): Starting executor service name=RS_REFRESH_PEER-regionserver/ubuntu:0, corePoolSize=2, maxPoolSize=2 2018-02-04 11:00:28,815 DEBUG [RS:1;ubuntu:46739] executor.ExecutorService(92): Starting executor service name=RS_OPEN_REGION-regionserver/ubuntu:0, corePoolSize=3, maxPoolSize=3 2018-02-04 11:00:28,817 DEBUG [RS:1;ubuntu:46739] executor.ExecutorService(92): Starting executor service name=RS_OPEN_META-regionserver/ubuntu:0, corePoolSize=1, maxPoolSize=1 2018-02-04 11:00:28,817 DEBUG [RS:1;ubuntu:46739] executor.ExecutorService(92): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/ubuntu:0, corePoolSize=3, maxPoolSize=3 2018-02-04 11:00:28,818 DEBUG [RS:1;ubuntu:46739] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_REGION-regionserver/ubuntu:0, corePoolSize=3, maxPoolSize=3 2018-02-04 11:00:28,818 DEBUG [RS:1;ubuntu:46739] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_META-regionserver/ubuntu:0, corePoolSize=1, maxPoolSize=1 2018-02-04 11:00:28,818 DEBUG [RS:1;ubuntu:46739] executor.ExecutorService(92): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/ubuntu:0, corePoolSize=2, maxPoolSize=2 2018-02-04 11:00:28,818 DEBUG [RS:1;ubuntu:46739] executor.ExecutorService(92): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/ubuntu:0, corePoolSize=10, maxPoolSize=10 2018-02-04 11:00:28,818 DEBUG [RS:1;ubuntu:46739] executor.ExecutorService(92): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/ubuntu:0, corePoolSize=3, maxPoolSize=3 2018-02-04 11:00:28,818 DEBUG [RS:1;ubuntu:46739] executor.ExecutorService(92): Starting executor service name=RS_REFRESH_PEER-regionserver/ubuntu:0, corePoolSize=2, maxPoolSize=2 2018-02-04 11:00:28,819 INFO [PEWorker-1] procedure.RecoverMetaProcedure(108): Start pid=1, state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-02-04 11:00:28,819 DEBUG [RS:0;ubuntu:39267] executor.ExecutorService(92): Starting executor service name=RS_OPEN_REGION-regionserver/ubuntu:0, corePoolSize=3, maxPoolSize=3 2018-02-04 11:00:28,820 DEBUG [RS:0;ubuntu:39267] executor.ExecutorService(92): Starting executor service name=RS_OPEN_META-regionserver/ubuntu:0, corePoolSize=1, maxPoolSize=1 2018-02-04 11:00:28,820 DEBUG [RS:0;ubuntu:39267] executor.ExecutorService(92): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/ubuntu:0, corePoolSize=3, maxPoolSize=3 2018-02-04 11:00:28,820 DEBUG [RS:0;ubuntu:39267] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_REGION-regionserver/ubuntu:0, corePoolSize=3, maxPoolSize=3 2018-02-04 11:00:28,820 DEBUG [RS:0;ubuntu:39267] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_META-regionserver/ubuntu:0, corePoolSize=1, maxPoolSize=1 2018-02-04 11:00:28,820 DEBUG [RS:0;ubuntu:39267] executor.ExecutorService(92): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/ubuntu:0, corePoolSize=2, maxPoolSize=2 2018-02-04 11:00:28,820 DEBUG [RS:0;ubuntu:39267] executor.ExecutorService(92): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/ubuntu:0, corePoolSize=10, maxPoolSize=10 2018-02-04 11:00:28,820 DEBUG [RS:0;ubuntu:39267] executor.ExecutorService(92): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/ubuntu:0, corePoolSize=3, maxPoolSize=3 2018-02-04 11:00:28,820 DEBUG [RS:0;ubuntu:39267] executor.ExecutorService(92): Starting executor service name=RS_REFRESH_PEER-regionserver/ubuntu:0, corePoolSize=2, maxPoolSize=2 2018-02-04 11:00:28,820 DEBUG [PEWorker-1] zookeeper.ZKUtil(614): master:39063-0x1615ec1f6c10000, quorum=localhost:56841, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2018-02-04 11:00:28,830 DEBUG [PEWorker-1] master.MasterWalManager(169): No log files to split, proceeding... 2018-02-04 11:00:28,838 DEBUG [PEWorker-1] zookeeper.ZKUtil(614): master:39063-0x1615ec1f6c10000, quorum=localhost:56841, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2018-02-04 11:00:28,838 INFO [PEWorker-1] procedure.RecoverMetaProcedure(140): pid=1, state=RUNNABLE:RECOVER_META_ASSIGN_REGIONS; RecoverMetaProcedure failedMetaServer=null, splitWal=true; Retaining meta assignment to server=null 2018-02-04 11:00:28,844 INFO [PEWorker-1] procedure2.ProcedureExecutor(1498): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740}] 2018-02-04 11:00:28,860 INFO [PEWorker-2] procedure.MasterProcedureScheduler(883): pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740 hbase:meta hbase:meta,,1.1588230740 2018-02-04 11:00:28,866 INFO [PEWorker-2] assignment.AssignProcedure(211): Start pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740; rit=OFFLINE, location=null; forceNewPlan=false, retain=false 2018-02-04 11:00:28,902 INFO [SplitLogWorker-ubuntu:36383] regionserver.SplitLogWorker(130): SplitLogWorker ubuntu,36383,1517713227466 starting 2018-02-04 11:00:28,906 INFO [RS:1;ubuntu:46739] regionserver.HeapMemoryManager(210): Starting HeapMemoryTuner chore. 2018-02-04 11:00:28,918 INFO [RS:0;ubuntu:39267] regionserver.HeapMemoryManager(210): Starting HeapMemoryTuner chore. 2018-02-04 11:00:28,918 INFO [SplitLogWorker-ubuntu:39267] regionserver.SplitLogWorker(130): SplitLogWorker ubuntu,39267,1517713227316 starting 2018-02-04 11:00:28,918 INFO [SplitLogWorker-ubuntu:46739] regionserver.SplitLogWorker(130): SplitLogWorker ubuntu,46739,1517713227398 starting 2018-02-04 11:00:28,907 INFO [RS:2;ubuntu:36383] regionserver.HeapMemoryManager(210): Starting HeapMemoryTuner chore. 2018-02-04 11:00:29,003 INFO [RS:1;ubuntu:46739] regionserver.HRegionServer(1530): Serving as ubuntu,46739,1517713227398, RpcServer on ubuntu/127.0.1.1:46739, sessionid=0x1615ec1f6c10002 2018-02-04 11:00:29,004 DEBUG [RS:1;ubuntu:46739] procedure.RegionServerProcedureManagerHost(52): Procedure flush-table-proc is starting 2018-02-04 11:00:29,004 DEBUG [RS:1;ubuntu:46739] flush.RegionServerFlushTableProcedureManager(104): Start region server flush procedure manager ubuntu,46739,1517713227398 2018-02-04 11:00:29,004 DEBUG [RS:1;ubuntu:46739] procedure.ZKProcedureMemberRpcs(353): Starting procedure member 'ubuntu,46739,1517713227398' 2018-02-04 11:00:29,004 DEBUG [RS:1;ubuntu:46739] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2018-02-04 11:00:29,005 DEBUG [RS:1;ubuntu:46739] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2018-02-04 11:00:29,005 DEBUG [RS:1;ubuntu:46739] procedure.RegionServerProcedureManagerHost(54): Procedure flush-table-proc is started 2018-02-04 11:00:29,005 DEBUG [RS:1;ubuntu:46739] procedure.RegionServerProcedureManagerHost(52): Procedure online-snapshot is starting 2018-02-04 11:00:29,006 DEBUG [RS:1;ubuntu:46739] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager ubuntu,46739,1517713227398 2018-02-04 11:00:29,006 DEBUG [RS:1;ubuntu:46739] procedure.ZKProcedureMemberRpcs(353): Starting procedure member 'ubuntu,46739,1517713227398' 2018-02-04 11:00:29,006 DEBUG [RS:1;ubuntu:46739] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2018-02-04 11:00:29,006 DEBUG [RS:1;ubuntu:46739] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2018-02-04 11:00:29,006 DEBUG [RS:1;ubuntu:46739] procedure.RegionServerProcedureManagerHost(54): Procedure online-snapshot is started 2018-02-04 11:00:29,007 INFO [RS:1;ubuntu:46739] quotas.RegionServerRpcQuotaManager(62): Quota support disabled 2018-02-04 11:00:29,007 INFO [RS:1;ubuntu:46739] quotas.RegionServerSpaceQuotaManager(74): Quota support disabled, not starting space quota manager. 2018-02-04 11:00:29,007 INFO [RS:2;ubuntu:36383] regionserver.HRegionServer(1530): Serving as ubuntu,36383,1517713227466, RpcServer on ubuntu/127.0.1.1:36383, sessionid=0x1615ec1f6c10003 2018-02-04 11:00:29,007 DEBUG [RS:2;ubuntu:36383] procedure.RegionServerProcedureManagerHost(52): Procedure flush-table-proc is starting 2018-02-04 11:00:29,007 DEBUG [RS:2;ubuntu:36383] flush.RegionServerFlushTableProcedureManager(104): Start region server flush procedure manager ubuntu,36383,1517713227466 2018-02-04 11:00:29,011 INFO [RS:0;ubuntu:39267] regionserver.HRegionServer(1530): Serving as ubuntu,39267,1517713227316, RpcServer on ubuntu/127.0.1.1:39267, sessionid=0x1615ec1f6c10001 2018-02-04 11:00:29,011 DEBUG [RS:2;ubuntu:36383] procedure.ZKProcedureMemberRpcs(353): Starting procedure member 'ubuntu,36383,1517713227466' 2018-02-04 11:00:29,012 DEBUG [RS:2;ubuntu:36383] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2018-02-04 11:00:29,012 DEBUG [RS:2;ubuntu:36383] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2018-02-04 11:00:29,013 DEBUG [RS:2;ubuntu:36383] procedure.RegionServerProcedureManagerHost(54): Procedure flush-table-proc is started 2018-02-04 11:00:29,013 DEBUG [RS:2;ubuntu:36383] procedure.RegionServerProcedureManagerHost(52): Procedure online-snapshot is starting 2018-02-04 11:00:29,013 DEBUG [RS:2;ubuntu:36383] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager ubuntu,36383,1517713227466 2018-02-04 11:00:29,016 DEBUG [RS:0;ubuntu:39267] procedure.RegionServerProcedureManagerHost(52): Procedure flush-table-proc is starting 2018-02-04 11:00:29,016 DEBUG [RS:2;ubuntu:36383] procedure.ZKProcedureMemberRpcs(353): Starting procedure member 'ubuntu,36383,1517713227466' 2018-02-04 11:00:29,016 DEBUG [RS:2;ubuntu:36383] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2018-02-04 11:00:29,016 DEBUG [RS:0;ubuntu:39267] flush.RegionServerFlushTableProcedureManager(104): Start region server flush procedure manager ubuntu,39267,1517713227316 2018-02-04 11:00:29,016 DEBUG [RS:0;ubuntu:39267] procedure.ZKProcedureMemberRpcs(353): Starting procedure member 'ubuntu,39267,1517713227316' 2018-02-04 11:00:29,016 DEBUG [RS:0;ubuntu:39267] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2018-02-04 11:00:29,016 DEBUG [RS:2;ubuntu:36383] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2018-02-04 11:00:29,017 DEBUG [RS:0;ubuntu:39267] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2018-02-04 11:00:29,017 DEBUG [RS:2;ubuntu:36383] procedure.RegionServerProcedureManagerHost(54): Procedure online-snapshot is started 2018-02-04 11:00:29,017 INFO [RS:2;ubuntu:36383] quotas.RegionServerRpcQuotaManager(62): Quota support disabled 2018-02-04 11:00:29,019 INFO [RS:2;ubuntu:36383] quotas.RegionServerSpaceQuotaManager(74): Quota support disabled, not starting space quota manager. 2018-02-04 11:00:29,023 DEBUG [master/ubuntu:0] assignment.AssignmentManager(1745): Processing assignQueue; systemServersCount=3, allServersCount=3 2018-02-04 11:00:29,027 DEBUG [RS:0;ubuntu:39267] procedure.RegionServerProcedureManagerHost(54): Procedure flush-table-proc is started 2018-02-04 11:00:29,027 DEBUG [RS:0;ubuntu:39267] procedure.RegionServerProcedureManagerHost(52): Procedure online-snapshot is starting 2018-02-04 11:00:29,027 DEBUG [RS:0;ubuntu:39267] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager ubuntu,39267,1517713227316 2018-02-04 11:00:29,027 DEBUG [RS:0;ubuntu:39267] procedure.ZKProcedureMemberRpcs(353): Starting procedure member 'ubuntu,39267,1517713227316' 2018-02-04 11:00:29,027 DEBUG [RS:0;ubuntu:39267] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2018-02-04 11:00:29,029 DEBUG [RS:0;ubuntu:39267] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2018-02-04 11:00:29,030 DEBUG [RS:0;ubuntu:39267] procedure.RegionServerProcedureManagerHost(54): Procedure online-snapshot is started 2018-02-04 11:00:29,031 INFO [RS:0;ubuntu:39267] quotas.RegionServerRpcQuotaManager(62): Quota support disabled 2018-02-04 11:00:29,031 INFO [RS:0;ubuntu:39267] quotas.RegionServerSpaceQuotaManager(74): Quota support disabled, not starting space quota manager. 2018-02-04 11:00:29,032 DEBUG [master/ubuntu:0] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region hbase:meta,,1.1588230740 2018-02-04 11:00:29,051 INFO [PEWorker-3] assignment.AssignProcedure(239): Early suspend! pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OFFLINE, location=ubuntu,46739,1517713227398 2018-02-04 11:00:30,022 INFO [RS:1;ubuntu:46739] wal.AbstractFSWAL(425): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=ubuntu%2C46739%2C1517713227398, suffix=, logDir=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/WALs/ubuntu,46739,1517713227398, archiveDir=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/oldWALs 2018-02-04 11:00:30,026 INFO [RS:2;ubuntu:36383] wal.AbstractFSWAL(425): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=ubuntu%2C36383%2C1517713227466, suffix=, logDir=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/WALs/ubuntu,36383,1517713227466, archiveDir=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/oldWALs 2018-02-04 11:00:30,035 INFO [RS:0;ubuntu:39267] wal.AbstractFSWAL(425): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=ubuntu%2C39267%2C1517713227316, suffix=, logDir=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/WALs/ubuntu,39267,1517713227316, archiveDir=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/oldWALs 2018-02-04 11:00:30,063 DEBUG [RS:0;ubuntu:39267] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(421): org.apache.hadoop.hdfs.protocolPB.PBHelperClient not found (Hadoop is pre-2.8.0?); using class org.apache.hadoop.hdfs.protocolPB.PBHelper instead. 2018-02-04 11:00:30,118 DEBUG [RS:0;ubuntu:39267] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(494): No DfsClientConf class found, should be hadoop 2.7- java.lang.ClassNotFoundException: org.apache.hadoop.hdfs.client.impl.DfsClientConf at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createChecksumCreater(FanOutOneBlockAsyncDFSOutputHelper.java:492) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:556) at org.apache.hadoop.hbase.io.asyncfs.AsyncFSOutputHelper.createOutput(AsyncFSOutputHelper.java:51) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.initOutput(AsyncProtobufLogWriter.java:159) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:167) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:99) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:612) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:123) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:766) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:504) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.(AsyncFSWAL.java:250) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createWAL(AsyncFSWALProvider.java:66) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createWAL(AsyncFSWALProvider.java:1) at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:138) at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:1) at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:252) at org.apache.hadoop.hbase.regionserver.HRegionServer.getWAL(HRegionServer.java:2088) at org.apache.hadoop.hbase.regionserver.HRegionServer.buildServerLoad(HRegionServer.java:1309) at org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:1190) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1002) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:187) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$0(MiniHBaseCluster.java:186) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:171) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:168) at java.lang.Thread.run(Thread.java:748) 2018-02-04 11:00:30,120 DEBUG [RS:0;ubuntu:39267] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(528): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2018-02-04 11:00:30,168 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(199): No PBHelperClient class found, should be hadoop 2.7- java.lang.ClassNotFoundException: org.apache.hadoop.hdfs.protocolPB.PBHelperClient at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createPBHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:197) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:261) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:661) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$3(FanOutOneBlockAsyncDFSOutputHelper.java:656) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$14.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:720) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$14.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:1) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:638) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:676) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:552) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:304) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:748) 2018-02-04 11:00:30,179 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46483,DS-1fd5fc50-1375-4985-9716-c32596d90f81,DISK] 2018-02-04 11:00:30,180 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33785,DS-dd1fcbd6-0c82-4629-949f-0463aa9af693,DISK] 2018-02-04 11:00:30,182 DEBUG [RS-EventLoopGroup-5-5] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45015,DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619,DISK] 2018-02-04 11:00:30,197 DEBUG [RS-EventLoopGroup-5-5] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33785,DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33,DISK] 2018-02-04 11:00:30,211 DEBUG [RS-EventLoopGroup-5-6] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46483,DS-44dd590d-1f51-400a-b8c2-20d260aaa289,DISK] 2018-02-04 11:00:30,217 DEBUG [RS-EventLoopGroup-5-5] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33785,DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33,DISK] 2018-02-04 11:00:30,220 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45015,DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1,DISK] 2018-02-04 11:00:30,220 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46483,DS-44dd590d-1f51-400a-b8c2-20d260aaa289,DISK] 2018-02-04 11:00:30,225 DEBUG [RS-EventLoopGroup-5-6] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45015,DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619,DISK] 2018-02-04 11:00:30,282 INFO [RS:0;ubuntu:39267] wal.AbstractFSWAL(708): New WAL /user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/WALs/ubuntu,39267,1517713227316/ubuntu%2C39267%2C1517713227316.1517713230044 2018-02-04 11:00:30,282 DEBUG [RS:0;ubuntu:39267] wal.AbstractFSWAL(772): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45015,DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1,DISK], DatanodeInfoWithStorage[127.0.0.1:46483,DS-44dd590d-1f51-400a-b8c2-20d260aaa289,DISK], DatanodeInfoWithStorage[127.0.0.1:33785,DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33,DISK]] 2018-02-04 11:00:30,286 INFO [RS:1;ubuntu:46739] wal.AbstractFSWAL(708): New WAL /user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/WALs/ubuntu,46739,1517713227398/ubuntu%2C46739%2C1517713227398.1517713230043 2018-02-04 11:00:30,286 DEBUG [RS:1;ubuntu:46739] wal.AbstractFSWAL(772): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45015,DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619,DISK], DatanodeInfoWithStorage[127.0.0.1:46483,DS-1fd5fc50-1375-4985-9716-c32596d90f81,DISK], DatanodeInfoWithStorage[127.0.0.1:33785,DS-dd1fcbd6-0c82-4629-949f-0463aa9af693,DISK]] 2018-02-04 11:00:30,287 INFO [RS:2;ubuntu:36383] wal.AbstractFSWAL(708): New WAL /user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/WALs/ubuntu,36383,1517713227466/ubuntu%2C36383%2C1517713227466.1517713230056 2018-02-04 11:00:30,287 DEBUG [RS:2;ubuntu:36383] wal.AbstractFSWAL(772): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:46483,DS-44dd590d-1f51-400a-b8c2-20d260aaa289,DISK], DatanodeInfoWithStorage[127.0.0.1:33785,DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33,DISK], DatanodeInfoWithStorage[127.0.0.1:45015,DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619,DISK]] 2018-02-04 11:00:30,303 INFO [PEWorker-4] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as ubuntu,46739,1517713227398 2018-02-04 11:00:30,315 DEBUG [PEWorker-4] zookeeper.MetaTableLocator(466): META region location doesn't exist, create it 2018-02-04 11:00:30,319 INFO [PEWorker-4] assignment.RegionTransitionProcedure(208): Dispatch pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=ubuntu,46739,1517713227398 2018-02-04 11:00:30,496 DEBUG [RSProcedureDispatcher-pool3-t1] master.ServerManager(778): New admin connection to ubuntu,46739,1517713227398 2018-02-04 11:00:30,501 DEBUG [RSProcedureDispatcher-pool3-t1] ipc.NettyRpcConnection(254): Connecting to ubuntu/127.0.1.1:46739 2018-02-04 11:00:30,512 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(528): Auth successful for zhangduo (auth:SIMPLE) 2018-02-04 11:00:30,512 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(558): Connection from 127.0.0.1 port: 53360 with version info: version: "3.0.0-SNAPSHOT" url: "git://ubuntu/home/zhangduo/hbase/code" revision: "7fd9ea800d3ec8bfbba59879d0c98e0268d0ccde" user: "zhangduo" date: "Sat Feb 3 18:15:39 CST 2018" src_checksum: "e1f78921fcd876d508017ada55edc99a" version_major: 3 version_minor: 0 2018-02-04 11:00:30,516 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=46739] regionserver.RSRpcServices(1912): Open hbase:meta,,1.1588230740 2018-02-04 11:00:30,520 INFO [RS_OPEN_META-regionserver/ubuntu:0-0] wal.WALFactory(135): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-02-04 11:00:30,528 INFO [RS_OPEN_META-regionserver/ubuntu:0-0] wal.AbstractFSWAL(425): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=ubuntu%2C46739%2C1517713227398.meta, suffix=.meta, logDir=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/WALs/ubuntu,46739,1517713227398, archiveDir=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/oldWALs 2018-02-04 11:00:30,538 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33785,DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33,DISK] 2018-02-04 11:00:30,563 DEBUG [RS-EventLoopGroup-5-4] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46483,DS-44dd590d-1f51-400a-b8c2-20d260aaa289,DISK] 2018-02-04 11:00:30,564 DEBUG [RS-EventLoopGroup-5-7] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45015,DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619,DISK] 2018-02-04 11:00:30,585 INFO [RS_OPEN_META-regionserver/ubuntu:0-0] wal.AbstractFSWAL(708): New WAL /user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/WALs/ubuntu,46739,1517713227398/ubuntu%2C46739%2C1517713227398.meta.1517713230529.meta 2018-02-04 11:00:30,585 DEBUG [RS_OPEN_META-regionserver/ubuntu:0-0] wal.AbstractFSWAL(772): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:33785,DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33,DISK], DatanodeInfoWithStorage[127.0.0.1:46483,DS-44dd590d-1f51-400a-b8c2-20d260aaa289,DISK], DatanodeInfoWithStorage[127.0.0.1:45015,DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619,DISK]] 2018-02-04 11:00:30,585 DEBUG [RS_OPEN_META-regionserver/ubuntu:0-0] regionserver.HRegion(6969): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2018-02-04 11:00:30,606 DEBUG [RS_OPEN_META-regionserver/ubuntu:0-0] coprocessor.CoprocessorHost(200): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2018-02-04 11:00:30,624 DEBUG [RS_OPEN_META-regionserver/ubuntu:0-0] regionserver.HRegion(7947): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2018-02-04 11:00:30,643 INFO [RS_OPEN_META-regionserver/ubuntu:0-0] regionserver.RegionCoprocessorHost(427): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2018-02-04 11:00:30,650 DEBUG [RS_OPEN_META-regionserver/ubuntu:0-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table meta 1588230740 2018-02-04 11:00:30,650 DEBUG [RS_OPEN_META-regionserver/ubuntu:0-0] regionserver.HRegion(809): Instantiated hbase:meta,,1.1588230740 2018-02-04 11:00:30,666 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/hbase/meta/1588230740/info 2018-02-04 11:00:30,666 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/hbase/meta/1588230740/info 2018-02-04 11:00:30,667 INFO [StoreOpener-1588230740-1] regionserver.HStore(294): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-02-04 11:00:30,667 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=0, currentSize=819720, freeSize=1116962296, maxSize=1117782016, heapSize=819720, minSize=1061892928, minFactor=0.95, multiSize=530946464, multiFactor=0.5, singleSize=265473232, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-02-04 11:00:30,668 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-02-04 11:00:30,672 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/hbase/meta/1588230740/rep_barrier 2018-02-04 11:00:30,673 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/hbase/meta/1588230740/rep_barrier 2018-02-04 11:00:30,673 INFO [StoreOpener-1588230740-1] regionserver.HStore(294): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-02-04 11:00:30,673 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_barrier: blockCache=LruBlockCache{blockCount=0, currentSize=819720, freeSize=1116962296, maxSize=1117782016, heapSize=819720, minSize=1061892928, minFactor=0.95, multiSize=530946464, multiFactor=0.5, singleSize=265473232, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-02-04 11:00:30,674 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-02-04 11:00:30,678 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/hbase/meta/1588230740/rep_meta 2018-02-04 11:00:30,679 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/hbase/meta/1588230740/rep_meta 2018-02-04 11:00:30,679 INFO [StoreOpener-1588230740-1] regionserver.HStore(294): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-02-04 11:00:30,679 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_meta: blockCache=LruBlockCache{blockCount=0, currentSize=819720, freeSize=1116962296, maxSize=1117782016, heapSize=819720, minSize=1061892928, minFactor=0.95, multiSize=530946464, multiFactor=0.5, singleSize=265473232, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-02-04 11:00:30,679 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-02-04 11:00:30,683 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/hbase/meta/1588230740/rep_position 2018-02-04 11:00:30,683 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/hbase/meta/1588230740/rep_position 2018-02-04 11:00:30,683 INFO [StoreOpener-1588230740-1] regionserver.HStore(294): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-02-04 11:00:30,683 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_position: blockCache=LruBlockCache{blockCount=0, currentSize=819720, freeSize=1116962296, maxSize=1117782016, heapSize=819720, minSize=1061892928, minFactor=0.95, multiSize=530946464, multiFactor=0.5, singleSize=265473232, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-02-04 11:00:30,683 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-02-04 11:00:30,686 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/hbase/meta/1588230740/table 2018-02-04 11:00:30,686 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/hbase/meta/1588230740/table 2018-02-04 11:00:30,686 INFO [StoreOpener-1588230740-1] regionserver.HStore(294): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-02-04 11:00:30,686 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for table: blockCache=LruBlockCache{blockCount=0, currentSize=819720, freeSize=1116962296, maxSize=1117782016, heapSize=819720, minSize=1061892928, minFactor=0.95, multiSize=530946464, multiFactor=0.5, singleSize=265473232, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-02-04 11:00:30,687 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-02-04 11:00:30,692 DEBUG [RS_OPEN_META-regionserver/ubuntu:0-0] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/hbase/meta/1588230740 2018-02-04 11:00:30,697 DEBUG [RS_OPEN_META-regionserver/ubuntu:0-0] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushSize/# of families (25.6M)) instead. 2018-02-04 11:00:30,704 DEBUG [RS_OPEN_META-regionserver/ubuntu:0-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/hbase/meta/1588230740/recovered.edits/3.seqid, newSeqId=3, maxSeqId=2 2018-02-04 11:00:30,705 INFO [RS_OPEN_META-regionserver/ubuntu:0-0] regionserver.HRegion(962): Onlined 1588230740; next sequenceid=3 2018-02-04 11:00:30,748 INFO [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(2166): Post open deploy tasks for hbase:meta,,1.1588230740 2018-02-04 11:00:30,765 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=39063] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=3, pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=ubuntu,46739,1517713227398 2018-02-04 11:00:30,767 INFO [PEWorker-5] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as ubuntu,46739,1517713227398 2018-02-04 11:00:30,771 DEBUG [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(2190): Finished post open deploy task for hbase:meta,,1.1588230740 2018-02-04 11:00:30,773 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(443): master:39063-0x1615ec1f6c10000, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2018-02-04 11:00:30,774 DEBUG [RS_OPEN_META-regionserver/ubuntu:0-0] handler.OpenRegionHandler(128): Opened hbase:meta,,1.1588230740 on ubuntu,46739,1517713227398 2018-02-04 11:00:30,782 INFO [PEWorker-5] procedure2.ProcedureExecutor(1617): Finished subprocedure(s) of pid=1, state=RUNNABLE; RecoverMetaProcedure failedMetaServer=null, splitWal=true; resume parent processing. 2018-02-04 11:00:30,782 INFO [PEWorker-5] procedure2.ProcedureExecutor(1249): Finished pid=2, ppid=1, state=SUCCESS; AssignProcedure table=hbase:meta, region=1588230740 in 1.9300sec 2018-02-04 11:00:30,786 INFO [PEWorker-6] procedure2.ProcedureExecutor(1249): Finished pid=1, state=SUCCESS; RecoverMetaProcedure failedMetaServer=null, splitWal=true in 2.1830sec 2018-02-04 11:00:30,786 INFO [M:0;ubuntu:39063] master.HMaster(3524): hbase:meta deployed at=ubuntu,46739,1517713227398 2018-02-04 11:00:30,831 DEBUG [hconnection-0x60afc9db-shared-pool8-t1] ipc.NettyRpcConnection(254): Connecting to ubuntu/127.0.1.1:46739 2018-02-04 11:00:30,841 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(528): Auth successful for zhangduo (auth:SIMPLE) 2018-02-04 11:00:30,841 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(558): Connection from 127.0.0.1 port: 53368 with version info: version: "3.0.0-SNAPSHOT" url: "git://ubuntu/home/zhangduo/hbase/code" revision: "7fd9ea800d3ec8bfbba59879d0c98e0268d0ccde" user: "zhangduo" date: "Sat Feb 3 18:15:39 CST 2018" src_checksum: "e1f78921fcd876d508017ada55edc99a" version_major: 3 version_minor: 0 2018-02-04 11:00:30,900 INFO [M:0;ubuntu:39063] master.MasterWalManager(189): Log folder hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/WALs/ubuntu,36383,1517713227466 belongs to an existing region server 2018-02-04 11:00:30,902 INFO [M:0;ubuntu:39063] master.MasterWalManager(189): Log folder hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/WALs/ubuntu,39267,1517713227316 belongs to an existing region server 2018-02-04 11:00:30,903 INFO [M:0;ubuntu:39063] master.MasterWalManager(189): Log folder hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/WALs/ubuntu,46739,1517713227398 belongs to an existing region server 2018-02-04 11:00:30,903 DEBUG [M:0;ubuntu:39063] assignment.AssignmentManager(1186): Joining cluster... 2018-02-04 11:00:30,914 INFO [M:0;ubuntu:39063] assignment.AssignmentManager(1196): Number of RegionServers=3 2018-02-04 11:00:30,914 INFO [M:0;ubuntu:39063] procedure2.ProcedureExecutor$TimeoutExecutorThread(1827): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=5000, timestamp=1517713235914 2018-02-04 11:00:30,914 INFO [M:0;ubuntu:39063] assignment.AssignmentManager(1203): Joined the cluster in 11msec, failover=true 2018-02-04 11:00:30,959 INFO [M:0;ubuntu:39063] master.TableNamespaceManager(95): Namespace table not found. Creating... 2018-02-04 11:00:30,964 INFO [M:0;ubuntu:39063] master.HMaster(1783): Client=null/null create 'hbase:namespace', {NAME => 'info', VERSIONS => '10', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} 2018-02-04 11:00:31,085 DEBUG [M:0;ubuntu:39063] procedure2.ProcedureExecutor(868): Stored pid=3, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2018-02-04 11:00:31,193 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741834_1010{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|FINALIZED]]} size 0 2018-02-04 11:00:31,194 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741834_1010{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|FINALIZED]]} size 0 2018-02-04 11:00:31,194 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741834_1010{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|FINALIZED], ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|FINALIZED]]} size 0 2018-02-04 11:00:31,201 DEBUG [PEWorker-7] util.FSTableDescriptors(716): Wrote into hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2018-02-04 11:00:31,206 INFO [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(6792): creating HRegion hbase:namespace HTD == 'hbase:namespace', {NAME => 'info', VERSIONS => '10', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} RootDir = hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/.tmp Table name == hbase:namespace 2018-02-04 11:00:31,242 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741835_1011{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|FINALIZED]]} size 0 2018-02-04 11:00:31,242 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741835_1011{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|FINALIZED], ReplicaUC[[DISK]DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619:NORMAL:127.0.0.1:45015|FINALIZED]]} size 0 2018-02-04 11:00:31,243 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741835_1011{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|FINALIZED], ReplicaUC[[DISK]DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619:NORMAL:127.0.0.1:45015|FINALIZED]]} size 0 2018-02-04 11:00:31,244 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(809): Instantiated hbase:namespace,,1517713230960.cf37e425ee5be11cb09b53c3671e1599. 2018-02-04 11:00:31,245 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1483): Closing hbase:namespace,,1517713230960.cf37e425ee5be11cb09b53c3671e1599.: disabling compactions & flushes 2018-02-04 11:00:31,245 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1522): Updates disabled for region hbase:namespace,,1517713230960.cf37e425ee5be11cb09b53c3671e1599. 2018-02-04 11:00:31,245 INFO [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1635): Closed hbase:namespace,,1517713230960.cf37e425ee5be11cb09b53c3671e1599. 2018-02-04 11:00:31,348 DEBUG [PEWorker-7] hbase.MetaTableAccessor(2146): Put {"totalColumns":2,"row":"hbase:namespace,,1517713230960.cf37e425ee5be11cb09b53c3671e1599.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-02-04 11:00:31,431 INFO [PEWorker-7] hbase.MetaTableAccessor(1638): Added 1 regions to meta. 2018-02-04 11:00:31,439 DEBUG [PEWorker-7] hbase.MetaTableAccessor(2146): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1517713231436}]},"ts":1517713231436} 2018-02-04 11:00:31,448 INFO [PEWorker-7] hbase.MetaTableAccessor(1801): Updated table hbase:namespace state to ENABLING in META 2018-02-04 11:00:31,448 DEBUG [PEWorker-7] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region hbase:namespace,,1517713230960.cf37e425ee5be11cb09b53c3671e1599. 2018-02-04 11:00:31,460 INFO [PEWorker-7] procedure2.ProcedureExecutor(1498): Initialized subprocedures=[{pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=cf37e425ee5be11cb09b53c3671e1599, target=ubuntu,39267,1517713227316}] 2018-02-04 11:00:31,464 INFO [PEWorker-7] procedure.MasterProcedureScheduler(883): pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=cf37e425ee5be11cb09b53c3671e1599, target=ubuntu,39267,1517713227316 hbase:namespace hbase:namespace,,1517713230960.cf37e425ee5be11cb09b53c3671e1599. 2018-02-04 11:00:31,476 INFO [PEWorker-7] assignment.AssignProcedure(211): Start pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=cf37e425ee5be11cb09b53c3671e1599, target=ubuntu,39267,1517713227316; rit=OFFLINE, location=ubuntu,39267,1517713227316; forceNewPlan=false, retain=false 2018-02-04 11:00:31,649 INFO [master/ubuntu:0] balancer.BaseLoadBalancer(1470): Reassigned 1 regions. 1 retained the pre-restart assignment. 2018-02-04 11:00:31,652 INFO [PEWorker-9] assignment.RegionStateStore(201): pid=4 updating hbase:meta row=hbase:namespace,,1517713230960.cf37e425ee5be11cb09b53c3671e1599., regionState=OPENING, regionLocation=ubuntu,39267,1517713227316 2018-02-04 11:00:31,655 INFO [PEWorker-9] zookeeper.ReadOnlyZKClient(133): Connect 0x5db12a7f to localhost:56841 with session timeout=1000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2018-02-04 11:00:31,668 DEBUG [PEWorker-9] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1c8c2044, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=1, fallbackAllowed=true, bind address=null 2018-02-04 11:00:31,673 DEBUG [MultiHConnection-shared--pool9-t1] ipc.NettyRpcConnection(254): Connecting to ubuntu/127.0.1.1:46739 2018-02-04 11:00:31,685 INFO [RS-EventLoopGroup-4-4] ipc.ServerRpcConnection(528): Auth successful for zhangduo (auth:SIMPLE) 2018-02-04 11:00:31,686 INFO [RS-EventLoopGroup-4-4] ipc.ServerRpcConnection(558): Connection from 127.0.0.1 port: 53386 with version info: version: "3.0.0-SNAPSHOT" url: "git://ubuntu/home/zhangduo/hbase/code" revision: "7fd9ea800d3ec8bfbba59879d0c98e0268d0ccde" user: "zhangduo" date: "Sat Feb 3 18:15:39 CST 2018" src_checksum: "e1f78921fcd876d508017ada55edc99a" version_major: 3 version_minor: 0 2018-02-04 11:00:31,695 INFO [PEWorker-9] assignment.RegionTransitionProcedure(208): Dispatch pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:namespace, region=cf37e425ee5be11cb09b53c3671e1599, target=ubuntu,39267,1517713227316; rit=OPENING, location=ubuntu,39267,1517713227316 2018-02-04 11:00:31,851 DEBUG [RSProcedureDispatcher-pool3-t2] master.ServerManager(778): New admin connection to ubuntu,39267,1517713227316 2018-02-04 11:00:31,853 DEBUG [RSProcedureDispatcher-pool3-t2] ipc.NettyRpcConnection(254): Connecting to ubuntu/127.0.1.1:39267 2018-02-04 11:00:31,869 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(528): Auth successful for zhangduo (auth:SIMPLE) 2018-02-04 11:00:31,870 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(558): Connection from 127.0.0.1 port: 35566 with version info: version: "3.0.0-SNAPSHOT" url: "git://ubuntu/home/zhangduo/hbase/code" revision: "7fd9ea800d3ec8bfbba59879d0c98e0268d0ccde" user: "zhangduo" date: "Sat Feb 3 18:15:39 CST 2018" src_checksum: "e1f78921fcd876d508017ada55edc99a" version_major: 3 version_minor: 0 2018-02-04 11:00:31,871 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=39267] regionserver.RSRpcServices(1912): Open hbase:namespace,,1517713230960.cf37e425ee5be11cb09b53c3671e1599. 2018-02-04 11:00:31,891 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(6969): Opening region: {ENCODED => cf37e425ee5be11cb09b53c3671e1599, NAME => 'hbase:namespace,,1517713230960.cf37e425ee5be11cb09b53c3671e1599.', STARTKEY => '', ENDKEY => ''} 2018-02-04 11:00:31,894 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/ubuntu:0-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table namespace cf37e425ee5be11cb09b53c3671e1599 2018-02-04 11:00:31,894 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(809): Instantiated hbase:namespace,,1517713230960.cf37e425ee5be11cb09b53c3671e1599. 2018-02-04 11:00:31,903 DEBUG [StoreOpener-cf37e425ee5be11cb09b53c3671e1599-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/hbase/namespace/cf37e425ee5be11cb09b53c3671e1599/info 2018-02-04 11:00:31,904 DEBUG [StoreOpener-cf37e425ee5be11cb09b53c3671e1599-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/hbase/namespace/cf37e425ee5be11cb09b53c3671e1599/info 2018-02-04 11:00:31,904 INFO [StoreOpener-cf37e425ee5be11cb09b53c3671e1599-1] regionserver.HStore(294): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-02-04 11:00:31,904 INFO [StoreOpener-cf37e425ee5be11cb09b53c3671e1599-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=0, currentSize=819720, freeSize=1116962296, maxSize=1117782016, heapSize=819720, minSize=1061892928, minFactor=0.95, multiSize=530946464, multiFactor=0.5, singleSize=265473232, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-02-04 11:00:31,905 INFO [StoreOpener-cf37e425ee5be11cb09b53c3671e1599-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-02-04 11:00:31,910 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/hbase/namespace/cf37e425ee5be11cb09b53c3671e1599 2018-02-04 11:00:31,922 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/ubuntu:0-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/hbase/namespace/cf37e425ee5be11cb09b53c3671e1599/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-02-04 11:00:31,922 INFO [RS_OPEN_PRIORITY_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(962): Onlined cf37e425ee5be11cb09b53c3671e1599; next sequenceid=2 2018-02-04 11:00:31,927 INFO [PostOpenDeployTasks:cf37e425ee5be11cb09b53c3671e1599] regionserver.HRegionServer(2166): Post open deploy tasks for hbase:namespace,,1517713230960.cf37e425ee5be11cb09b53c3671e1599. 2018-02-04 11:00:31,931 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=39063] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:namespace, region=cf37e425ee5be11cb09b53c3671e1599, target=ubuntu,39267,1517713227316; rit=OPENING, location=ubuntu,39267,1517713227316 2018-02-04 11:00:31,932 INFO [PEWorker-10] assignment.RegionStateStore(201): pid=4 updating hbase:meta row=hbase:namespace,,1517713230960.cf37e425ee5be11cb09b53c3671e1599., regionState=OPEN, openSeqNum=2, regionLocation=ubuntu,39267,1517713227316 2018-02-04 11:00:31,935 DEBUG [PostOpenDeployTasks:cf37e425ee5be11cb09b53c3671e1599] regionserver.HRegionServer(2190): Finished post open deploy task for hbase:namespace,,1517713230960.cf37e425ee5be11cb09b53c3671e1599. 2018-02-04 11:00:31,937 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/ubuntu:0-0] handler.OpenRegionHandler(128): Opened hbase:namespace,,1517713230960.cf37e425ee5be11cb09b53c3671e1599. on ubuntu,39267,1517713227316 2018-02-04 11:00:31,952 INFO [PEWorker-10] procedure2.ProcedureExecutor(1617): Finished subprocedure(s) of pid=3, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=hbase:namespace; resume parent processing. 2018-02-04 11:00:31,952 INFO [PEWorker-10] procedure2.ProcedureExecutor(1249): Finished pid=4, ppid=3, state=SUCCESS; AssignProcedure table=hbase:namespace, region=cf37e425ee5be11cb09b53c3671e1599, target=ubuntu,39267,1517713227316 in 484msec 2018-02-04 11:00:31,952 DEBUG [PEWorker-11] hbase.MetaTableAccessor(2146): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1517713231952}]},"ts":1517713231952} 2018-02-04 11:00:31,962 INFO [PEWorker-11] hbase.MetaTableAccessor(1801): Updated table hbase:namespace state to ENABLED in META 2018-02-04 11:00:31,971 INFO [PEWorker-11] procedure2.ProcedureExecutor(1249): Finished pid=3, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 1.0010sec 2018-02-04 11:00:32,049 DEBUG [M:0;ubuntu:39063] zookeeper.ZKUtil(357): master:39063-0x1615ec1f6c10000, quorum=localhost:56841, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2018-02-04 11:00:32,052 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(443): master:39063-0x1615ec1f6c10000, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2018-02-04 11:00:32,078 DEBUG [M:0;ubuntu:39063] ipc.NettyRpcConnection(254): Connecting to ubuntu/127.0.1.1:39267 2018-02-04 11:00:32,084 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(528): Auth successful for zhangduo (auth:SIMPLE) 2018-02-04 11:00:32,084 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(558): Connection from 127.0.0.1 port: 35570 with version info: version: "3.0.0-SNAPSHOT" url: "git://ubuntu/home/zhangduo/hbase/code" revision: "7fd9ea800d3ec8bfbba59879d0c98e0268d0ccde" user: "zhangduo" date: "Sat Feb 3 18:15:39 CST 2018" src_checksum: "e1f78921fcd876d508017ada55edc99a" version_major: 3 version_minor: 0 2018-02-04 11:00:32,216 DEBUG [M:0;ubuntu:39063] procedure2.ProcedureExecutor(868): Stored pid=5, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2018-02-04 11:00:32,297 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(443): master:39063-0x1615ec1f6c10000, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2018-02-04 11:00:32,313 INFO [PEWorker-12] procedure2.ProcedureExecutor(1249): Finished pid=5, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 218msec 2018-02-04 11:00:32,436 DEBUG [M:0;ubuntu:39063] procedure2.ProcedureExecutor(868): Stored pid=6, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2018-02-04 11:00:32,473 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(443): master:39063-0x1615ec1f6c10000, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2018-02-04 11:00:32,482 INFO [PEWorker-13] procedure2.ProcedureExecutor(1249): Finished pid=6, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 151msec 2018-02-04 11:00:32,496 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(443): master:39063-0x1615ec1f6c10000, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2018-02-04 11:00:32,499 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(443): master:39063-0x1615ec1f6c10000, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2018-02-04 11:00:32,499 INFO [M:0;ubuntu:39063] master.HMaster(937): Master has completed initialization 4.942sec 2018-02-04 11:00:32,502 INFO [M:0;ubuntu:39063] quotas.MasterQuotaManager(80): Quota support disabled 2018-02-04 11:00:32,502 INFO [M:0;ubuntu:39063] zookeeper.ZKWatcher(170): not a secure deployment, proceeding 2018-02-04 11:00:32,571 INFO [Time-limited test] zookeeper.ReadOnlyZKClient(133): Connect 0x4107e73b to localhost:56841 with session timeout=1000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2018-02-04 11:00:32,597 DEBUG [Time-limited test] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4281eecf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=1, fallbackAllowed=true, bind address=null 2018-02-04 11:00:32,620 DEBUG [hconnection-0x78b0f3e2-shared-pool13-t1] ipc.NettyRpcConnection(254): Connecting to ubuntu/127.0.1.1:46739 2018-02-04 11:00:32,636 INFO [RS-EventLoopGroup-4-5] ipc.ServerRpcConnection(528): Auth successful for zhangduo (auth:SIMPLE) 2018-02-04 11:00:32,636 INFO [RS-EventLoopGroup-4-5] ipc.ServerRpcConnection(558): Connection from 127.0.0.1 port: 53396 with version info: version: "3.0.0-SNAPSHOT" url: "git://ubuntu/home/zhangduo/hbase/code" revision: "7fd9ea800d3ec8bfbba59879d0c98e0268d0ccde" user: "zhangduo" date: "Sat Feb 3 18:15:39 CST 2018" src_checksum: "e1f78921fcd876d508017ada55edc99a" version_major: 3 version_minor: 0 2018-02-04 11:00:32,640 INFO [Time-limited test] hbase.HBaseTestingUtility(1051): Minicluster is up; activeMaster=ubuntu,39063,1517713226282 2018-02-04 11:00:32,662 DEBUG [Time-limited test] ipc.NettyRpcConnection(254): Connecting to ubuntu/127.0.1.1:39063 2018-02-04 11:00:32,684 INFO [RS-EventLoopGroup-1-5] ipc.ServerRpcConnection(528): Auth successful for zhangduo (auth:SIMPLE) 2018-02-04 11:00:32,685 INFO [RS-EventLoopGroup-1-5] ipc.ServerRpcConnection(558): Connection from 127.0.0.1 port: 52916 with version info: version: "3.0.0-SNAPSHOT" url: "git://ubuntu/home/zhangduo/hbase/code" revision: "7fd9ea800d3ec8bfbba59879d0c98e0268d0ccde" user: "zhangduo" date: "Sat Feb 3 18:15:39 CST 2018" src_checksum: "e1f78921fcd876d508017ada55edc99a" version_major: 3 version_minor: 0 2018-02-04 11:00:32,704 INFO [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=39063] master.HMaster$3(1750): Client=zhangduo//127.0.0.1 create 'tabletest', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2018-02-04 11:00:32,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=39063] procedure2.ProcedureExecutor(868): Stored pid=7, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=tabletest 2018-02-04 11:00:32,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=39063] master.MasterRpcServices(1157): Checking to see if procedure is done pid=7 2018-02-04 11:00:32,846 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741836_1012{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|FINALIZED]]} size 0 2018-02-04 11:00:32,848 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741836_1012{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|FINALIZED], ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|FINALIZED]]} size 0 2018-02-04 11:00:32,850 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741836_1012{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|FINALIZED], ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|FINALIZED]]} size 0 2018-02-04 11:00:32,859 DEBUG [PEWorker-14] util.FSTableDescriptors(716): Wrote into hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/.tmp/data/default/tabletest/.tabledesc/.tableinfo.0000000001 2018-02-04 11:00:32,866 INFO [RegionOpenAndInitThread-tabletest-1] regionserver.HRegion(6792): creating HRegion tabletest HTD == 'tabletest', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/.tmp Table name == tabletest 2018-02-04 11:00:32,867 INFO [RegionOpenAndInitThread-tabletest-2] regionserver.HRegion(6792): creating HRegion tabletest HTD == 'tabletest', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/.tmp Table name == tabletest 2018-02-04 11:00:32,866 INFO [RegionOpenAndInitThread-tabletest-4] regionserver.HRegion(6792): creating HRegion tabletest HTD == 'tabletest', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/.tmp Table name == tabletest 2018-02-04 11:00:32,866 INFO [RegionOpenAndInitThread-tabletest-3] regionserver.HRegion(6792): creating HRegion tabletest HTD == 'tabletest', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/.tmp Table name == tabletest 2018-02-04 11:00:32,873 INFO [RegionOpenAndInitThread-tabletest-6] regionserver.HRegion(6792): creating HRegion tabletest HTD == 'tabletest', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/.tmp Table name == tabletest 2018-02-04 11:00:32,876 INFO [RegionOpenAndInitThread-tabletest-7] regionserver.HRegion(6792): creating HRegion tabletest HTD == 'tabletest', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/.tmp Table name == tabletest 2018-02-04 11:00:32,874 INFO [RegionOpenAndInitThread-tabletest-9] regionserver.HRegion(6792): creating HRegion tabletest HTD == 'tabletest', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/.tmp Table name == tabletest 2018-02-04 11:00:32,874 INFO [RegionOpenAndInitThread-tabletest-5] regionserver.HRegion(6792): creating HRegion tabletest HTD == 'tabletest', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/.tmp Table name == tabletest 2018-02-04 11:00:32,895 INFO [RegionOpenAndInitThread-tabletest-13] regionserver.HRegion(6792): creating HRegion tabletest HTD == 'tabletest', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/.tmp Table name == tabletest 2018-02-04 11:00:32,886 INFO [RegionOpenAndInitThread-tabletest-16] regionserver.HRegion(6792): creating HRegion tabletest HTD == 'tabletest', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/.tmp Table name == tabletest 2018-02-04 11:00:32,885 INFO [RegionOpenAndInitThread-tabletest-15] regionserver.HRegion(6792): creating HRegion tabletest HTD == 'tabletest', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/.tmp Table name == tabletest 2018-02-04 11:00:32,884 INFO [RegionOpenAndInitThread-tabletest-14] regionserver.HRegion(6792): creating HRegion tabletest HTD == 'tabletest', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/.tmp Table name == tabletest 2018-02-04 11:00:32,884 INFO [RegionOpenAndInitThread-tabletest-12] regionserver.HRegion(6792): creating HRegion tabletest HTD == 'tabletest', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/.tmp Table name == tabletest 2018-02-04 11:00:32,878 INFO [RegionOpenAndInitThread-tabletest-8] regionserver.HRegion(6792): creating HRegion tabletest HTD == 'tabletest', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/.tmp Table name == tabletest 2018-02-04 11:00:32,877 INFO [RegionOpenAndInitThread-tabletest-10] regionserver.HRegion(6792): creating HRegion tabletest HTD == 'tabletest', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/.tmp Table name == tabletest 2018-02-04 11:00:32,895 INFO [RegionOpenAndInitThread-tabletest-11] regionserver.HRegion(6792): creating HRegion tabletest HTD == 'tabletest', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/.tmp Table name == tabletest 2018-02-04 11:00:32,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=39063] master.MasterRpcServices(1157): Checking to see if procedure is done pid=7 2018-02-04 11:00:33,030 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741838_1014{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW]]} size 0 2018-02-04 11:00:33,049 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741838_1014 size 50 2018-02-04 11:00:33,049 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741838_1014 size 50 2018-02-04 11:00:33,049 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741839_1015{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW]]} size 50 2018-02-04 11:00:33,049 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741842_1018{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|RBW]]} size 50 2018-02-04 11:00:33,050 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741842_1018 size 50 2018-02-04 11:00:33,050 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741839_1015 size 50 2018-02-04 11:00:33,050 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741839_1015 size 50 2018-02-04 11:00:33,050 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741842_1018 size 50 2018-02-04 11:00:33,050 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741837_1013{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619:NORMAL:127.0.0.1:45015|RBW]]} size 0 2018-02-04 11:00:33,051 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741837_1013{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619:NORMAL:127.0.0.1:45015|RBW]]} size 0 2018-02-04 11:00:33,051 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741837_1013{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619:NORMAL:127.0.0.1:45015|RBW]]} size 0 2018-02-04 11:00:33,067 DEBUG [RegionOpenAndInitThread-tabletest-1] regionserver.HRegion(809): Instantiated tabletest,,1517713232695.4f8ea7a76697ee6c4cd92147cc852db7. 2018-02-04 11:00:33,068 DEBUG [RegionOpenAndInitThread-tabletest-1] regionserver.HRegion(1483): Closing tabletest,,1517713232695.4f8ea7a76697ee6c4cd92147cc852db7.: disabling compactions & flushes 2018-02-04 11:00:33,068 DEBUG [RegionOpenAndInitThread-tabletest-1] regionserver.HRegion(1522): Updates disabled for region tabletest,,1517713232695.4f8ea7a76697ee6c4cd92147cc852db7. 2018-02-04 11:00:33,068 INFO [RegionOpenAndInitThread-tabletest-1] regionserver.HRegion(1635): Closed tabletest,,1517713232695.4f8ea7a76697ee6c4cd92147cc852db7. 2018-02-04 11:00:33,069 INFO [RegionOpenAndInitThread-tabletest-1] regionserver.HRegion(6792): creating HRegion tabletest HTD == 'tabletest', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/.tmp Table name == tabletest 2018-02-04 11:00:33,072 DEBUG [RegionOpenAndInitThread-tabletest-3] regionserver.HRegion(809): Instantiated tabletest,ccc,1517713232695.cd5c06da9ebf17554a264cbdd6fcd9ef. 2018-02-04 11:00:33,072 DEBUG [RegionOpenAndInitThread-tabletest-3] regionserver.HRegion(1483): Closing tabletest,ccc,1517713232695.cd5c06da9ebf17554a264cbdd6fcd9ef.: disabling compactions & flushes 2018-02-04 11:00:33,072 DEBUG [RegionOpenAndInitThread-tabletest-3] regionserver.HRegion(1522): Updates disabled for region tabletest,ccc,1517713232695.cd5c06da9ebf17554a264cbdd6fcd9ef. 2018-02-04 11:00:33,072 INFO [RegionOpenAndInitThread-tabletest-3] regionserver.HRegion(1635): Closed tabletest,ccc,1517713232695.cd5c06da9ebf17554a264cbdd6fcd9ef. 2018-02-04 11:00:33,082 INFO [RegionOpenAndInitThread-tabletest-3] regionserver.HRegion(6792): creating HRegion tabletest HTD == 'tabletest', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/.tmp Table name == tabletest 2018-02-04 11:00:33,092 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741843_1019{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619:NORMAL:127.0.0.1:45015|RBW]]} size 0 2018-02-04 11:00:33,096 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741844_1020{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619:NORMAL:127.0.0.1:45015|RBW]]} size 0 2018-02-04 11:00:33,096 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741850_1026{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619:NORMAL:127.0.0.1:45015|RBW]]} size 0 2018-02-04 11:00:33,096 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741841_1017{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|RBW]]} size 0 2018-02-04 11:00:33,102 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741848_1024{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW]]} size 0 2018-02-04 11:00:33,102 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741850_1026{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619:NORMAL:127.0.0.1:45015|RBW]]} size 0 2018-02-04 11:00:33,102 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741843_1019 size 50 2018-02-04 11:00:33,102 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741844_1020{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619:NORMAL:127.0.0.1:45015|RBW]]} size 0 2018-02-04 11:00:33,102 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741845_1021{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW]]} size 0 2018-02-04 11:00:33,102 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741852_1028{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|RBW]]} size 0 2018-02-04 11:00:33,102 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741841_1017{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|RBW]]} size 0 2018-02-04 11:00:33,102 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741848_1024{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW]]} size 0 2018-02-04 11:00:33,105 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741848_1024{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|FINALIZED]]} size 0 2018-02-04 11:00:33,106 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741843_1019 size 50 2018-02-04 11:00:33,107 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741844_1020{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619:NORMAL:127.0.0.1:45015|RBW]]} size 0 2018-02-04 11:00:33,107 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741850_1026 size 50 2018-02-04 11:00:33,107 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741841_1017 size 50 2018-02-04 11:00:33,107 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741845_1021 size 50 2018-02-04 11:00:33,107 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741845_1021 size 50 2018-02-04 11:00:33,112 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741852_1028 size 50 2018-02-04 11:00:33,112 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741852_1028 size 50 2018-02-04 11:00:33,114 WARN [DataStreamer for file /user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/.tmp/data/default/tabletest/756819a3f903733cb342d3913aab52a6/.regioninfo] hdfs.DFSOutputStream$DataStreamer(718): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1252) at java.lang.Thread.join(Thread.java:1326) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.closeResponder(DFSOutputStream.java:716) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.endBlock(DFSOutputStream.java:476) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:652) 2018-02-04 11:00:33,115 DEBUG [RegionOpenAndInitThread-tabletest-2] regionserver.HRegion(809): Instantiated tabletest,bbb,1517713232695.756819a3f903733cb342d3913aab52a6. 2018-02-04 11:00:33,115 DEBUG [RegionOpenAndInitThread-tabletest-2] regionserver.HRegion(1483): Closing tabletest,bbb,1517713232695.756819a3f903733cb342d3913aab52a6.: disabling compactions & flushes 2018-02-04 11:00:33,115 DEBUG [RegionOpenAndInitThread-tabletest-2] regionserver.HRegion(1522): Updates disabled for region tabletest,bbb,1517713232695.756819a3f903733cb342d3913aab52a6. 2018-02-04 11:00:33,115 INFO [RegionOpenAndInitThread-tabletest-2] regionserver.HRegion(1635): Closed tabletest,bbb,1517713232695.756819a3f903733cb342d3913aab52a6. 2018-02-04 11:00:33,116 INFO [RegionOpenAndInitThread-tabletest-2] regionserver.HRegion(6792): creating HRegion tabletest HTD == 'tabletest', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/.tmp Table name == tabletest 2018-02-04 11:00:33,124 DEBUG [RegionOpenAndInitThread-tabletest-5] regionserver.HRegion(809): Instantiated tabletest,eee,1517713232695.14343105008c3cdd39a78990296fbe94. 2018-02-04 11:00:33,124 DEBUG [RegionOpenAndInitThread-tabletest-5] regionserver.HRegion(1483): Closing tabletest,eee,1517713232695.14343105008c3cdd39a78990296fbe94.: disabling compactions & flushes 2018-02-04 11:00:33,124 DEBUG [RegionOpenAndInitThread-tabletest-5] regionserver.HRegion(1522): Updates disabled for region tabletest,eee,1517713232695.14343105008c3cdd39a78990296fbe94. 2018-02-04 11:00:33,124 INFO [RegionOpenAndInitThread-tabletest-5] regionserver.HRegion(1635): Closed tabletest,eee,1517713232695.14343105008c3cdd39a78990296fbe94. 2018-02-04 11:00:33,125 INFO [RegionOpenAndInitThread-tabletest-5] regionserver.HRegion(6792): creating HRegion tabletest HTD == 'tabletest', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/.tmp Table name == tabletest 2018-02-04 11:00:33,125 DEBUG [RegionOpenAndInitThread-tabletest-10] regionserver.HRegion(809): Instantiated tabletest,jjj,1517713232695.e4481a648e0b377eb5bf59bfb99dfd88. 2018-02-04 11:00:33,125 DEBUG [RegionOpenAndInitThread-tabletest-10] regionserver.HRegion(1483): Closing tabletest,jjj,1517713232695.e4481a648e0b377eb5bf59bfb99dfd88.: disabling compactions & flushes 2018-02-04 11:00:33,125 DEBUG [RegionOpenAndInitThread-tabletest-10] regionserver.HRegion(1522): Updates disabled for region tabletest,jjj,1517713232695.e4481a648e0b377eb5bf59bfb99dfd88. 2018-02-04 11:00:33,125 INFO [RegionOpenAndInitThread-tabletest-10] regionserver.HRegion(1635): Closed tabletest,jjj,1517713232695.e4481a648e0b377eb5bf59bfb99dfd88. 2018-02-04 11:00:33,125 INFO [RegionOpenAndInitThread-tabletest-10] regionserver.HRegion(6792): creating HRegion tabletest HTD == 'tabletest', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/.tmp Table name == tabletest 2018-02-04 11:00:33,136 DEBUG [RegionOpenAndInitThread-tabletest-16] regionserver.HRegion(809): Instantiated tabletest,ppp,1517713232695.12bc4be2348528b0bd4575ae03200fc3. 2018-02-04 11:00:33,136 DEBUG [RegionOpenAndInitThread-tabletest-16] regionserver.HRegion(1483): Closing tabletest,ppp,1517713232695.12bc4be2348528b0bd4575ae03200fc3.: disabling compactions & flushes 2018-02-04 11:00:33,137 DEBUG [RegionOpenAndInitThread-tabletest-16] regionserver.HRegion(1522): Updates disabled for region tabletest,ppp,1517713232695.12bc4be2348528b0bd4575ae03200fc3. 2018-02-04 11:00:33,137 INFO [RegionOpenAndInitThread-tabletest-16] regionserver.HRegion(1635): Closed tabletest,ppp,1517713232695.12bc4be2348528b0bd4575ae03200fc3. 2018-02-04 11:00:33,138 INFO [RegionOpenAndInitThread-tabletest-16] regionserver.HRegion(6792): creating HRegion tabletest HTD == 'tabletest', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/.tmp Table name == tabletest 2018-02-04 11:00:33,147 DEBUG [RegionOpenAndInitThread-tabletest-9] regionserver.HRegion(809): Instantiated tabletest,iii,1517713232695.f1f365dfd0648c0fb5d9848c1db452cc. 2018-02-04 11:00:33,147 DEBUG [RegionOpenAndInitThread-tabletest-9] regionserver.HRegion(1483): Closing tabletest,iii,1517713232695.f1f365dfd0648c0fb5d9848c1db452cc.: disabling compactions & flushes 2018-02-04 11:00:33,147 DEBUG [RegionOpenAndInitThread-tabletest-9] regionserver.HRegion(1522): Updates disabled for region tabletest,iii,1517713232695.f1f365dfd0648c0fb5d9848c1db452cc. 2018-02-04 11:00:33,147 INFO [RegionOpenAndInitThread-tabletest-9] regionserver.HRegion(1635): Closed tabletest,iii,1517713232695.f1f365dfd0648c0fb5d9848c1db452cc. 2018-02-04 11:00:33,148 INFO [RegionOpenAndInitThread-tabletest-9] regionserver.HRegion(6792): creating HRegion tabletest HTD == 'tabletest', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/.tmp Table name == tabletest 2018-02-04 11:00:33,155 DEBUG [RegionOpenAndInitThread-tabletest-15] regionserver.HRegion(809): Instantiated tabletest,ooo,1517713232695.41f447d2065af74b219164f0a70d08b3. 2018-02-04 11:00:33,155 DEBUG [RegionOpenAndInitThread-tabletest-15] regionserver.HRegion(1483): Closing tabletest,ooo,1517713232695.41f447d2065af74b219164f0a70d08b3.: disabling compactions & flushes 2018-02-04 11:00:33,156 DEBUG [RegionOpenAndInitThread-tabletest-15] regionserver.HRegion(1522): Updates disabled for region tabletest,ooo,1517713232695.41f447d2065af74b219164f0a70d08b3. 2018-02-04 11:00:33,156 INFO [RegionOpenAndInitThread-tabletest-15] regionserver.HRegion(1635): Closed tabletest,ooo,1517713232695.41f447d2065af74b219164f0a70d08b3. 2018-02-04 11:00:33,156 INFO [RegionOpenAndInitThread-tabletest-15] regionserver.HRegion(6792): creating HRegion tabletest HTD == 'tabletest', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/.tmp Table name == tabletest 2018-02-04 11:00:33,174 DEBUG [RegionOpenAndInitThread-tabletest-14] regionserver.HRegion(809): Instantiated tabletest,nnn,1517713232695.7429e32b4c4e0aadde8a8154cbd1f071. 2018-02-04 11:00:33,174 DEBUG [RegionOpenAndInitThread-tabletest-14] regionserver.HRegion(1483): Closing tabletest,nnn,1517713232695.7429e32b4c4e0aadde8a8154cbd1f071.: disabling compactions & flushes 2018-02-04 11:00:33,175 DEBUG [RegionOpenAndInitThread-tabletest-14] regionserver.HRegion(1522): Updates disabled for region tabletest,nnn,1517713232695.7429e32b4c4e0aadde8a8154cbd1f071. 2018-02-04 11:00:33,175 INFO [RegionOpenAndInitThread-tabletest-14] regionserver.HRegion(1635): Closed tabletest,nnn,1517713232695.7429e32b4c4e0aadde8a8154cbd1f071. 2018-02-04 11:00:33,176 INFO [RegionOpenAndInitThread-tabletest-14] regionserver.HRegion(6792): creating HRegion tabletest HTD == 'tabletest', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/.tmp Table name == tabletest 2018-02-04 11:00:33,186 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741851_1027{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|RBW]]} size 0 2018-02-04 11:00:33,187 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741851_1027 size 50 2018-02-04 11:00:33,187 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741851_1027 size 50 2018-02-04 11:00:33,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=39063] master.MasterRpcServices(1157): Checking to see if procedure is done pid=7 2018-02-04 11:00:33,193 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741849_1025{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|RBW]]} size 50 2018-02-04 11:00:33,193 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741847_1023{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|RBW]]} size 0 2018-02-04 11:00:33,193 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741849_1025 size 50 2018-02-04 11:00:33,194 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741849_1025 size 50 2018-02-04 11:00:33,196 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741847_1023{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|RBW]]} size 0 2018-02-04 11:00:33,198 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741840_1016{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW]]} size 0 2018-02-04 11:00:33,198 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741847_1023{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|RBW]]} size 0 2018-02-04 11:00:33,198 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741846_1022{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW]]} size 0 2018-02-04 11:00:33,199 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741840_1016{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW]]} size 0 2018-02-04 11:00:33,203 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741846_1022{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW]]} size 0 2018-02-04 11:00:33,203 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741846_1022{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW]]} size 0 2018-02-04 11:00:33,203 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741840_1016{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW]]} size 0 2018-02-04 11:00:33,226 DEBUG [RegionOpenAndInitThread-tabletest-8] regionserver.HRegion(809): Instantiated tabletest,hhh,1517713232695.ad4f7bcab0a803ddae20b2866466d222. 2018-02-04 11:00:33,227 DEBUG [RegionOpenAndInitThread-tabletest-8] regionserver.HRegion(1483): Closing tabletest,hhh,1517713232695.ad4f7bcab0a803ddae20b2866466d222.: disabling compactions & flushes 2018-02-04 11:00:33,227 DEBUG [RegionOpenAndInitThread-tabletest-8] regionserver.HRegion(1522): Updates disabled for region tabletest,hhh,1517713232695.ad4f7bcab0a803ddae20b2866466d222. 2018-02-04 11:00:33,227 INFO [RegionOpenAndInitThread-tabletest-8] regionserver.HRegion(1635): Closed tabletest,hhh,1517713232695.ad4f7bcab0a803ddae20b2866466d222. 2018-02-04 11:00:33,227 INFO [RegionOpenAndInitThread-tabletest-8] regionserver.HRegion(6792): creating HRegion tabletest HTD == 'tabletest', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/.tmp Table name == tabletest 2018-02-04 11:00:33,239 DEBUG [RegionOpenAndInitThread-tabletest-12] regionserver.HRegion(809): Instantiated tabletest,lll,1517713232695.85d5a900ae52e80b00b89bb74335621c. 2018-02-04 11:00:33,240 DEBUG [RegionOpenAndInitThread-tabletest-12] regionserver.HRegion(1483): Closing tabletest,lll,1517713232695.85d5a900ae52e80b00b89bb74335621c.: disabling compactions & flushes 2018-02-04 11:00:33,240 DEBUG [RegionOpenAndInitThread-tabletest-12] regionserver.HRegion(1522): Updates disabled for region tabletest,lll,1517713232695.85d5a900ae52e80b00b89bb74335621c. 2018-02-04 11:00:33,240 INFO [RegionOpenAndInitThread-tabletest-12] regionserver.HRegion(1635): Closed tabletest,lll,1517713232695.85d5a900ae52e80b00b89bb74335621c. 2018-02-04 11:00:33,240 DEBUG [RegionOpenAndInitThread-tabletest-13] regionserver.HRegion(809): Instantiated tabletest,mmm,1517713232695.a19e4bf3b69ad0ea43c87985b14ec9fe. 2018-02-04 11:00:33,240 DEBUG [RegionOpenAndInitThread-tabletest-13] regionserver.HRegion(1483): Closing tabletest,mmm,1517713232695.a19e4bf3b69ad0ea43c87985b14ec9fe.: disabling compactions & flushes 2018-02-04 11:00:33,240 DEBUG [RegionOpenAndInitThread-tabletest-13] regionserver.HRegion(1522): Updates disabled for region tabletest,mmm,1517713232695.a19e4bf3b69ad0ea43c87985b14ec9fe. 2018-02-04 11:00:33,241 INFO [RegionOpenAndInitThread-tabletest-13] regionserver.HRegion(1635): Closed tabletest,mmm,1517713232695.a19e4bf3b69ad0ea43c87985b14ec9fe. 2018-02-04 11:00:33,241 DEBUG [RegionOpenAndInitThread-tabletest-6] regionserver.HRegion(809): Instantiated tabletest,fff,1517713232695.7b96ff3277efd9751a2175b9c2b4d481. 2018-02-04 11:00:33,242 DEBUG [RegionOpenAndInitThread-tabletest-6] regionserver.HRegion(1483): Closing tabletest,fff,1517713232695.7b96ff3277efd9751a2175b9c2b4d481.: disabling compactions & flushes 2018-02-04 11:00:33,242 DEBUG [RegionOpenAndInitThread-tabletest-6] regionserver.HRegion(1522): Updates disabled for region tabletest,fff,1517713232695.7b96ff3277efd9751a2175b9c2b4d481. 2018-02-04 11:00:33,242 INFO [RegionOpenAndInitThread-tabletest-6] regionserver.HRegion(1635): Closed tabletest,fff,1517713232695.7b96ff3277efd9751a2175b9c2b4d481. 2018-02-04 11:00:33,249 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741854_1030{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619:NORMAL:127.0.0.1:45015|RBW]]} size 0 2018-02-04 11:00:33,251 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741854_1030 size 50 2018-02-04 11:00:33,252 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741854_1030 size 50 2018-02-04 11:00:33,261 DEBUG [RegionOpenAndInitThread-tabletest-3] regionserver.HRegion(809): Instantiated tabletest,rrr,1517713232695.1b8fb6f4917edac2a9ec29ba9ebed5ef. 2018-02-04 11:00:33,262 DEBUG [RegionOpenAndInitThread-tabletest-3] regionserver.HRegion(1483): Closing tabletest,rrr,1517713232695.1b8fb6f4917edac2a9ec29ba9ebed5ef.: disabling compactions & flushes 2018-02-04 11:00:33,262 DEBUG [RegionOpenAndInitThread-tabletest-3] regionserver.HRegion(1522): Updates disabled for region tabletest,rrr,1517713232695.1b8fb6f4917edac2a9ec29ba9ebed5ef. 2018-02-04 11:00:33,262 INFO [RegionOpenAndInitThread-tabletest-3] regionserver.HRegion(1635): Closed tabletest,rrr,1517713232695.1b8fb6f4917edac2a9ec29ba9ebed5ef. 2018-02-04 11:00:33,263 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741855_1031{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619:NORMAL:127.0.0.1:45015|RBW]]} size 0 2018-02-04 11:00:33,269 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741855_1031 size 50 2018-02-04 11:00:33,269 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741855_1031 size 50 2018-02-04 11:00:33,280 DEBUG [RegionOpenAndInitThread-tabletest-10] regionserver.HRegion(809): Instantiated tabletest,uuu,1517713232695.9579a7c2f22113dfe574ba9569101416. 2018-02-04 11:00:33,281 DEBUG [RegionOpenAndInitThread-tabletest-10] regionserver.HRegion(1483): Closing tabletest,uuu,1517713232695.9579a7c2f22113dfe574ba9569101416.: disabling compactions & flushes 2018-02-04 11:00:33,281 DEBUG [RegionOpenAndInitThread-tabletest-10] regionserver.HRegion(1522): Updates disabled for region tabletest,uuu,1517713232695.9579a7c2f22113dfe574ba9569101416. 2018-02-04 11:00:33,281 INFO [RegionOpenAndInitThread-tabletest-10] regionserver.HRegion(1635): Closed tabletest,uuu,1517713232695.9579a7c2f22113dfe574ba9569101416. 2018-02-04 11:00:33,287 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741858_1034{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW]]} size 50 2018-02-04 11:00:33,288 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741858_1034 size 50 2018-02-04 11:00:33,288 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741858_1034 size 50 2018-02-04 11:00:33,306 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741856_1032{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW]]} size 50 2018-02-04 11:00:33,313 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741853_1029{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW]]} size 0 2018-02-04 11:00:33,315 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741856_1032 size 50 2018-02-04 11:00:33,317 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741853_1029{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW]]} size 0 2018-02-04 11:00:33,320 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741856_1032 size 50 2018-02-04 11:00:33,321 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741859_1035{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW]]} size 0 2018-02-04 11:00:33,324 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741859_1035{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW]]} size 0 2018-02-04 11:00:33,324 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741857_1033{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619:NORMAL:127.0.0.1:45015|RBW]]} size 0 2018-02-04 11:00:33,325 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741857_1033{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619:NORMAL:127.0.0.1:45015|RBW]]} size 0 2018-02-04 11:00:33,325 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741859_1035{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW]]} size 0 2018-02-04 11:00:33,327 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741853_1029 size 50 2018-02-04 11:00:33,336 DEBUG [RegionOpenAndInitThread-tabletest-2] regionserver.HRegion(809): Instantiated tabletest,sss,1517713232695.e063e6b2c2640eb6eafc8737a71fb5f5. 2018-02-04 11:00:33,336 DEBUG [RegionOpenAndInitThread-tabletest-2] regionserver.HRegion(1483): Closing tabletest,sss,1517713232695.e063e6b2c2640eb6eafc8737a71fb5f5.: disabling compactions & flushes 2018-02-04 11:00:33,336 DEBUG [RegionOpenAndInitThread-tabletest-2] regionserver.HRegion(1522): Updates disabled for region tabletest,sss,1517713232695.e063e6b2c2640eb6eafc8737a71fb5f5. 2018-02-04 11:00:33,336 INFO [RegionOpenAndInitThread-tabletest-2] regionserver.HRegion(1635): Closed tabletest,sss,1517713232695.e063e6b2c2640eb6eafc8737a71fb5f5. 2018-02-04 11:00:33,336 DEBUG [RegionOpenAndInitThread-tabletest-1] regionserver.HRegion(809): Instantiated tabletest,qqq,1517713232695.6a7b42fe3d02ea7aca1d05fa666e87ac. 2018-02-04 11:00:33,338 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741857_1033 size 50 2018-02-04 11:00:33,338 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741860_1036{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW]]} size 0 2018-02-04 11:00:33,338 DEBUG [RegionOpenAndInitThread-tabletest-9] regionserver.HRegion(809): Instantiated tabletest,www,1517713232695.7514f60148f3a526baae5c7ce2c78c1b. 2018-02-04 11:00:33,339 DEBUG [RegionOpenAndInitThread-tabletest-9] regionserver.HRegion(1483): Closing tabletest,www,1517713232695.7514f60148f3a526baae5c7ce2c78c1b.: disabling compactions & flushes 2018-02-04 11:00:33,339 DEBUG [RegionOpenAndInitThread-tabletest-9] regionserver.HRegion(1522): Updates disabled for region tabletest,www,1517713232695.7514f60148f3a526baae5c7ce2c78c1b. 2018-02-04 11:00:33,339 INFO [RegionOpenAndInitThread-tabletest-9] regionserver.HRegion(1635): Closed tabletest,www,1517713232695.7514f60148f3a526baae5c7ce2c78c1b. 2018-02-04 11:00:33,341 DEBUG [RegionOpenAndInitThread-tabletest-1] regionserver.HRegion(1483): Closing tabletest,qqq,1517713232695.6a7b42fe3d02ea7aca1d05fa666e87ac.: disabling compactions & flushes 2018-02-04 11:00:33,346 DEBUG [RegionOpenAndInitThread-tabletest-1] regionserver.HRegion(1522): Updates disabled for region tabletest,qqq,1517713232695.6a7b42fe3d02ea7aca1d05fa666e87ac. 2018-02-04 11:00:33,352 INFO [RegionOpenAndInitThread-tabletest-1] regionserver.HRegion(1635): Closed tabletest,qqq,1517713232695.6a7b42fe3d02ea7aca1d05fa666e87ac. 2018-02-04 11:00:33,357 DEBUG [RegionOpenAndInitThread-tabletest-15] regionserver.HRegion(809): Instantiated tabletest,xxx,1517713232695.66ce5e96488227599f1cd0e5e39a2982. 2018-02-04 11:00:33,357 DEBUG [RegionOpenAndInitThread-tabletest-15] regionserver.HRegion(1483): Closing tabletest,xxx,1517713232695.66ce5e96488227599f1cd0e5e39a2982.: disabling compactions & flushes 2018-02-04 11:00:33,357 DEBUG [RegionOpenAndInitThread-tabletest-15] regionserver.HRegion(1522): Updates disabled for region tabletest,xxx,1517713232695.66ce5e96488227599f1cd0e5e39a2982. 2018-02-04 11:00:33,357 INFO [RegionOpenAndInitThread-tabletest-15] regionserver.HRegion(1635): Closed tabletest,xxx,1517713232695.66ce5e96488227599f1cd0e5e39a2982. 2018-02-04 11:00:33,361 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741860_1036 size 50 2018-02-04 11:00:33,363 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741860_1036 size 50 2018-02-04 11:00:33,363 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741862_1038{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|RBW]]} size 50 2018-02-04 11:00:33,364 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741862_1038 size 50 2018-02-04 11:00:33,364 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741862_1038 size 50 2018-02-04 11:00:33,368 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741861_1037{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW]]} size 0 2018-02-04 11:00:33,368 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741861_1037{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW]]} size 0 2018-02-04 11:00:33,370 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741861_1037 size 47 2018-02-04 11:00:33,370 DEBUG [RegionOpenAndInitThread-tabletest-8] regionserver.HRegion(809): Instantiated tabletest,zzz,1517713232695.1da7d397be67579013b1f635bfbcb064. 2018-02-04 11:00:33,370 DEBUG [RegionOpenAndInitThread-tabletest-8] regionserver.HRegion(1483): Closing tabletest,zzz,1517713232695.1da7d397be67579013b1f635bfbcb064.: disabling compactions & flushes 2018-02-04 11:00:33,370 DEBUG [RegionOpenAndInitThread-tabletest-8] regionserver.HRegion(1522): Updates disabled for region tabletest,zzz,1517713232695.1da7d397be67579013b1f635bfbcb064. 2018-02-04 11:00:33,370 INFO [RegionOpenAndInitThread-tabletest-8] regionserver.HRegion(1635): Closed tabletest,zzz,1517713232695.1da7d397be67579013b1f635bfbcb064. 2018-02-04 11:00:33,452 DEBUG [RegionOpenAndInitThread-tabletest-7] regionserver.HRegion(809): Instantiated tabletest,ggg,1517713232695.879a95042a6098ee953dbfe218826b40. 2018-02-04 11:00:33,452 DEBUG [RegionOpenAndInitThread-tabletest-4] regionserver.HRegion(809): Instantiated tabletest,ddd,1517713232695.68c9676b485038d63fd2cf8a435420e5. 2018-02-04 11:00:33,452 DEBUG [RegionOpenAndInitThread-tabletest-7] regionserver.HRegion(1483): Closing tabletest,ggg,1517713232695.879a95042a6098ee953dbfe218826b40.: disabling compactions & flushes 2018-02-04 11:00:33,453 DEBUG [RegionOpenAndInitThread-tabletest-7] regionserver.HRegion(1522): Updates disabled for region tabletest,ggg,1517713232695.879a95042a6098ee953dbfe218826b40. 2018-02-04 11:00:33,453 INFO [RegionOpenAndInitThread-tabletest-7] regionserver.HRegion(1635): Closed tabletest,ggg,1517713232695.879a95042a6098ee953dbfe218826b40. 2018-02-04 11:00:33,453 DEBUG [RegionOpenAndInitThread-tabletest-4] regionserver.HRegion(1483): Closing tabletest,ddd,1517713232695.68c9676b485038d63fd2cf8a435420e5.: disabling compactions & flushes 2018-02-04 11:00:33,454 DEBUG [RegionOpenAndInitThread-tabletest-4] regionserver.HRegion(1522): Updates disabled for region tabletest,ddd,1517713232695.68c9676b485038d63fd2cf8a435420e5. 2018-02-04 11:00:33,454 INFO [RegionOpenAndInitThread-tabletest-4] regionserver.HRegion(1635): Closed tabletest,ddd,1517713232695.68c9676b485038d63fd2cf8a435420e5. 2018-02-04 11:00:33,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39063] master.MasterRpcServices(1157): Checking to see if procedure is done pid=7 2018-02-04 11:00:33,591 DEBUG [RegionOpenAndInitThread-tabletest-11] regionserver.HRegion(809): Instantiated tabletest,kkk,1517713232695.a127c3f989561b0b55d12be9bd499407. 2018-02-04 11:00:33,591 DEBUG [RegionOpenAndInitThread-tabletest-11] regionserver.HRegion(1483): Closing tabletest,kkk,1517713232695.a127c3f989561b0b55d12be9bd499407.: disabling compactions & flushes 2018-02-04 11:00:33,591 DEBUG [RegionOpenAndInitThread-tabletest-11] regionserver.HRegion(1522): Updates disabled for region tabletest,kkk,1517713232695.a127c3f989561b0b55d12be9bd499407. 2018-02-04 11:00:33,592 INFO [RegionOpenAndInitThread-tabletest-11] regionserver.HRegion(1635): Closed tabletest,kkk,1517713232695.a127c3f989561b0b55d12be9bd499407. 2018-02-04 11:00:33,692 DEBUG [RegionOpenAndInitThread-tabletest-16] regionserver.HRegion(809): Instantiated tabletest,vvv,1517713232695.2a83d01250e1489bfa6bcdc2057bb1b5. 2018-02-04 11:00:33,693 DEBUG [RegionOpenAndInitThread-tabletest-16] regionserver.HRegion(1483): Closing tabletest,vvv,1517713232695.2a83d01250e1489bfa6bcdc2057bb1b5.: disabling compactions & flushes 2018-02-04 11:00:33,693 DEBUG [RegionOpenAndInitThread-tabletest-16] regionserver.HRegion(1522): Updates disabled for region tabletest,vvv,1517713232695.2a83d01250e1489bfa6bcdc2057bb1b5. 2018-02-04 11:00:33,693 INFO [RegionOpenAndInitThread-tabletest-16] regionserver.HRegion(1635): Closed tabletest,vvv,1517713232695.2a83d01250e1489bfa6bcdc2057bb1b5. 2018-02-04 11:00:33,711 DEBUG [RegionOpenAndInitThread-tabletest-5] regionserver.HRegion(809): Instantiated tabletest,ttt,1517713232695.31b3e09f2db65dd6057a91f45cb8b163. 2018-02-04 11:00:33,714 DEBUG [RegionOpenAndInitThread-tabletest-5] regionserver.HRegion(1483): Closing tabletest,ttt,1517713232695.31b3e09f2db65dd6057a91f45cb8b163.: disabling compactions & flushes 2018-02-04 11:00:33,714 DEBUG [RegionOpenAndInitThread-tabletest-5] regionserver.HRegion(1522): Updates disabled for region tabletest,ttt,1517713232695.31b3e09f2db65dd6057a91f45cb8b163. 2018-02-04 11:00:33,714 INFO [RegionOpenAndInitThread-tabletest-5] regionserver.HRegion(1635): Closed tabletest,ttt,1517713232695.31b3e09f2db65dd6057a91f45cb8b163. 2018-02-04 11:00:33,759 DEBUG [RegionOpenAndInitThread-tabletest-14] regionserver.HRegion(809): Instantiated tabletest,yyy,1517713232695.20457e81e436062b31548a08737ebba4. 2018-02-04 11:00:33,760 DEBUG [RegionOpenAndInitThread-tabletest-14] regionserver.HRegion(1483): Closing tabletest,yyy,1517713232695.20457e81e436062b31548a08737ebba4.: disabling compactions & flushes 2018-02-04 11:00:33,760 DEBUG [RegionOpenAndInitThread-tabletest-14] regionserver.HRegion(1522): Updates disabled for region tabletest,yyy,1517713232695.20457e81e436062b31548a08737ebba4. 2018-02-04 11:00:33,760 INFO [RegionOpenAndInitThread-tabletest-14] regionserver.HRegion(1635): Closed tabletest,yyy,1517713232695.20457e81e436062b31548a08737ebba4. 2018-02-04 11:00:33,784 DEBUG [PEWorker-14] hbase.MetaTableAccessor(2146): Put {"totalColumns":2,"row":"tabletest,,1517713232695.4f8ea7a76697ee6c4cd92147cc852db7.","families":{"info":[{"qualifier":"regioninfo","vlen":46,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-02-04 11:00:33,785 DEBUG [PEWorker-14] hbase.MetaTableAccessor(2146): Put {"totalColumns":2,"row":"tabletest,ccc,1517713232695.cd5c06da9ebf17554a264cbdd6fcd9ef.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-02-04 11:00:33,785 DEBUG [PEWorker-14] hbase.MetaTableAccessor(2146): Put {"totalColumns":2,"row":"tabletest,bbb,1517713232695.756819a3f903733cb342d3913aab52a6.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-02-04 11:00:33,785 DEBUG [PEWorker-14] hbase.MetaTableAccessor(2146): Put {"totalColumns":2,"row":"tabletest,eee,1517713232695.14343105008c3cdd39a78990296fbe94.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-02-04 11:00:33,785 DEBUG [PEWorker-14] hbase.MetaTableAccessor(2146): Put {"totalColumns":2,"row":"tabletest,jjj,1517713232695.e4481a648e0b377eb5bf59bfb99dfd88.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-02-04 11:00:33,785 DEBUG [PEWorker-14] hbase.MetaTableAccessor(2146): Put {"totalColumns":2,"row":"tabletest,ppp,1517713232695.12bc4be2348528b0bd4575ae03200fc3.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-02-04 11:00:33,786 DEBUG [PEWorker-14] hbase.MetaTableAccessor(2146): Put {"totalColumns":2,"row":"tabletest,iii,1517713232695.f1f365dfd0648c0fb5d9848c1db452cc.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-02-04 11:00:33,786 DEBUG [PEWorker-14] hbase.MetaTableAccessor(2146): Put {"totalColumns":2,"row":"tabletest,ooo,1517713232695.41f447d2065af74b219164f0a70d08b3.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-02-04 11:00:33,786 DEBUG [PEWorker-14] hbase.MetaTableAccessor(2146): Put {"totalColumns":2,"row":"tabletest,nnn,1517713232695.7429e32b4c4e0aadde8a8154cbd1f071.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-02-04 11:00:33,786 DEBUG [PEWorker-14] hbase.MetaTableAccessor(2146): Put {"totalColumns":2,"row":"tabletest,hhh,1517713232695.ad4f7bcab0a803ddae20b2866466d222.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-02-04 11:00:33,786 DEBUG [PEWorker-14] hbase.MetaTableAccessor(2146): Put {"totalColumns":2,"row":"tabletest,lll,1517713232695.85d5a900ae52e80b00b89bb74335621c.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-02-04 11:00:33,786 DEBUG [PEWorker-14] hbase.MetaTableAccessor(2146): Put {"totalColumns":2,"row":"tabletest,mmm,1517713232695.a19e4bf3b69ad0ea43c87985b14ec9fe.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-02-04 11:00:33,787 DEBUG [PEWorker-14] hbase.MetaTableAccessor(2146): Put {"totalColumns":2,"row":"tabletest,fff,1517713232695.7b96ff3277efd9751a2175b9c2b4d481.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-02-04 11:00:33,787 DEBUG [PEWorker-14] hbase.MetaTableAccessor(2146): Put {"totalColumns":2,"row":"tabletest,rrr,1517713232695.1b8fb6f4917edac2a9ec29ba9ebed5ef.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-02-04 11:00:33,787 DEBUG [PEWorker-14] hbase.MetaTableAccessor(2146): Put {"totalColumns":2,"row":"tabletest,uuu,1517713232695.9579a7c2f22113dfe574ba9569101416.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-02-04 11:00:33,787 DEBUG [PEWorker-14] hbase.MetaTableAccessor(2146): Put {"totalColumns":2,"row":"tabletest,sss,1517713232695.e063e6b2c2640eb6eafc8737a71fb5f5.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-02-04 11:00:33,787 DEBUG [PEWorker-14] hbase.MetaTableAccessor(2146): Put {"totalColumns":2,"row":"tabletest,www,1517713232695.7514f60148f3a526baae5c7ce2c78c1b.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-02-04 11:00:33,788 DEBUG [PEWorker-14] hbase.MetaTableAccessor(2146): Put {"totalColumns":2,"row":"tabletest,qqq,1517713232695.6a7b42fe3d02ea7aca1d05fa666e87ac.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-02-04 11:00:33,788 DEBUG [PEWorker-14] hbase.MetaTableAccessor(2146): Put {"totalColumns":2,"row":"tabletest,xxx,1517713232695.66ce5e96488227599f1cd0e5e39a2982.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-02-04 11:00:33,788 DEBUG [PEWorker-14] hbase.MetaTableAccessor(2146): Put {"totalColumns":2,"row":"tabletest,zzz,1517713232695.1da7d397be67579013b1f635bfbcb064.","families":{"info":[{"qualifier":"regioninfo","vlen":46,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-02-04 11:00:33,788 DEBUG [PEWorker-14] hbase.MetaTableAccessor(2146): Put {"totalColumns":2,"row":"tabletest,ggg,1517713232695.879a95042a6098ee953dbfe218826b40.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-02-04 11:00:33,788 DEBUG [PEWorker-14] hbase.MetaTableAccessor(2146): Put {"totalColumns":2,"row":"tabletest,ddd,1517713232695.68c9676b485038d63fd2cf8a435420e5.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-02-04 11:00:33,788 DEBUG [PEWorker-14] hbase.MetaTableAccessor(2146): Put {"totalColumns":2,"row":"tabletest,kkk,1517713232695.a127c3f989561b0b55d12be9bd499407.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-02-04 11:00:33,789 DEBUG [PEWorker-14] hbase.MetaTableAccessor(2146): Put {"totalColumns":2,"row":"tabletest,vvv,1517713232695.2a83d01250e1489bfa6bcdc2057bb1b5.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-02-04 11:00:33,789 DEBUG [PEWorker-14] hbase.MetaTableAccessor(2146): Put {"totalColumns":2,"row":"tabletest,ttt,1517713232695.31b3e09f2db65dd6057a91f45cb8b163.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-02-04 11:00:33,789 DEBUG [PEWorker-14] hbase.MetaTableAccessor(2146): Put {"totalColumns":2,"row":"tabletest,yyy,1517713232695.20457e81e436062b31548a08737ebba4.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-02-04 11:00:33,813 INFO [PEWorker-14] hbase.MetaTableAccessor(1638): Added 26 regions to meta. 2018-02-04 11:00:33,818 DEBUG [PEWorker-14] hbase.MetaTableAccessor(2146): Put {"totalColumns":1,"row":"tabletest","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1517713233818}]},"ts":1517713233818} 2018-02-04 11:00:33,826 INFO [PEWorker-14] hbase.MetaTableAccessor(1801): Updated table tabletest state to ENABLING in META 2018-02-04 11:00:33,827 DEBUG [PEWorker-14] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region tabletest,,1517713232695.4f8ea7a76697ee6c4cd92147cc852db7. 2018-02-04 11:00:33,838 DEBUG [PEWorker-14] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region tabletest,ccc,1517713232695.cd5c06da9ebf17554a264cbdd6fcd9ef. 2018-02-04 11:00:33,840 DEBUG [PEWorker-14] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region tabletest,bbb,1517713232695.756819a3f903733cb342d3913aab52a6. 2018-02-04 11:00:33,842 DEBUG [PEWorker-14] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region tabletest,eee,1517713232695.14343105008c3cdd39a78990296fbe94. 2018-02-04 11:00:33,845 DEBUG [PEWorker-14] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region tabletest,jjj,1517713232695.e4481a648e0b377eb5bf59bfb99dfd88. 2018-02-04 11:00:33,847 DEBUG [PEWorker-14] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region tabletest,ppp,1517713232695.12bc4be2348528b0bd4575ae03200fc3. 2018-02-04 11:00:33,851 DEBUG [PEWorker-14] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region tabletest,iii,1517713232695.f1f365dfd0648c0fb5d9848c1db452cc. 2018-02-04 11:00:33,854 DEBUG [PEWorker-14] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region tabletest,ooo,1517713232695.41f447d2065af74b219164f0a70d08b3. 2018-02-04 11:00:33,855 DEBUG [PEWorker-14] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region tabletest,nnn,1517713232695.7429e32b4c4e0aadde8a8154cbd1f071. 2018-02-04 11:00:33,857 DEBUG [PEWorker-14] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region tabletest,hhh,1517713232695.ad4f7bcab0a803ddae20b2866466d222. 2018-02-04 11:00:33,858 DEBUG [PEWorker-14] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region tabletest,lll,1517713232695.85d5a900ae52e80b00b89bb74335621c. 2018-02-04 11:00:33,859 DEBUG [PEWorker-14] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region tabletest,mmm,1517713232695.a19e4bf3b69ad0ea43c87985b14ec9fe. 2018-02-04 11:00:33,861 DEBUG [PEWorker-14] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region tabletest,fff,1517713232695.7b96ff3277efd9751a2175b9c2b4d481. 2018-02-04 11:00:33,862 DEBUG [PEWorker-14] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region tabletest,rrr,1517713232695.1b8fb6f4917edac2a9ec29ba9ebed5ef. 2018-02-04 11:00:33,863 DEBUG [PEWorker-14] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region tabletest,uuu,1517713232695.9579a7c2f22113dfe574ba9569101416. 2018-02-04 11:00:33,865 DEBUG [PEWorker-14] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region tabletest,sss,1517713232695.e063e6b2c2640eb6eafc8737a71fb5f5. 2018-02-04 11:00:33,868 DEBUG [PEWorker-14] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region tabletest,www,1517713232695.7514f60148f3a526baae5c7ce2c78c1b. 2018-02-04 11:00:33,870 DEBUG [PEWorker-14] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region tabletest,qqq,1517713232695.6a7b42fe3d02ea7aca1d05fa666e87ac. 2018-02-04 11:00:33,871 DEBUG [PEWorker-14] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region tabletest,xxx,1517713232695.66ce5e96488227599f1cd0e5e39a2982. 2018-02-04 11:00:33,876 DEBUG [PEWorker-14] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region tabletest,zzz,1517713232695.1da7d397be67579013b1f635bfbcb064. 2018-02-04 11:00:33,878 DEBUG [PEWorker-14] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region tabletest,ggg,1517713232695.879a95042a6098ee953dbfe218826b40. 2018-02-04 11:00:33,880 DEBUG [PEWorker-14] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region tabletest,ddd,1517713232695.68c9676b485038d63fd2cf8a435420e5. 2018-02-04 11:00:33,882 DEBUG [PEWorker-14] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region tabletest,kkk,1517713232695.a127c3f989561b0b55d12be9bd499407. 2018-02-04 11:00:33,883 DEBUG [PEWorker-14] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region tabletest,vvv,1517713232695.2a83d01250e1489bfa6bcdc2057bb1b5. 2018-02-04 11:00:33,884 DEBUG [PEWorker-14] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region tabletest,ttt,1517713232695.31b3e09f2db65dd6057a91f45cb8b163. 2018-02-04 11:00:33,886 DEBUG [PEWorker-14] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region tabletest,yyy,1517713232695.20457e81e436062b31548a08737ebba4. 2018-02-04 11:00:33,893 INFO [PEWorker-14] procedure2.ProcedureExecutor(1498): Initialized subprocedures=[{pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=4f8ea7a76697ee6c4cd92147cc852db7, target=ubuntu,39267,1517713227316}, {pid=9, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=756819a3f903733cb342d3913aab52a6, target=ubuntu,36383,1517713227466}, {pid=10, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=cd5c06da9ebf17554a264cbdd6fcd9ef, target=ubuntu,46739,1517713227398}, {pid=11, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=68c9676b485038d63fd2cf8a435420e5, target=ubuntu,39267,1517713227316}, {pid=12, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=14343105008c3cdd39a78990296fbe94, target=ubuntu,39267,1517713227316}, {pid=13, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=7b96ff3277efd9751a2175b9c2b4d481, target=ubuntu,39267,1517713227316}, {pid=14, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=879a95042a6098ee953dbfe218826b40, target=ubuntu,36383,1517713227466}, {pid=15, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=ad4f7bcab0a803ddae20b2866466d222, target=ubuntu,39267,1517713227316}, {pid=16, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=f1f365dfd0648c0fb5d9848c1db452cc, target=ubuntu,39267,1517713227316}, {pid=17, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=e4481a648e0b377eb5bf59bfb99dfd88, target=ubuntu,46739,1517713227398}, {pid=18, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=a127c3f989561b0b55d12be9bd499407, target=ubuntu,46739,1517713227398}, {pid=19, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=85d5a900ae52e80b00b89bb74335621c, target=ubuntu,46739,1517713227398}, {pid=20, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=a19e4bf3b69ad0ea43c87985b14ec9fe, target=ubuntu,36383,1517713227466}, {pid=21, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=7429e32b4c4e0aadde8a8154cbd1f071, target=ubuntu,36383,1517713227466}, {pid=22, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=41f447d2065af74b219164f0a70d08b3, target=ubuntu,46739,1517713227398}, {pid=23, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=12bc4be2348528b0bd4575ae03200fc3, target=ubuntu,36383,1517713227466}, {pid=24, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=6a7b42fe3d02ea7aca1d05fa666e87ac, target=ubuntu,36383,1517713227466}, {pid=25, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=1b8fb6f4917edac2a9ec29ba9ebed5ef, target=ubuntu,46739,1517713227398}, {pid=26, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=e063e6b2c2640eb6eafc8737a71fb5f5, target=ubuntu,39267,1517713227316}, {pid=27, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=31b3e09f2db65dd6057a91f45cb8b163, target=ubuntu,39267,1517713227316}, {pid=28, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=9579a7c2f22113dfe574ba9569101416, target=ubuntu,36383,1517713227466}, {pid=29, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=2a83d01250e1489bfa6bcdc2057bb1b5, target=ubuntu,36383,1517713227466}, {pid=30, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=7514f60148f3a526baae5c7ce2c78c1b, target=ubuntu,46739,1517713227398}, {pid=31, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=66ce5e96488227599f1cd0e5e39a2982, target=ubuntu,39267,1517713227316}, {pid=32, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=20457e81e436062b31548a08737ebba4, target=ubuntu,46739,1517713227398}, {pid=33, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=1da7d397be67579013b1f635bfbcb064, target=ubuntu,46739,1517713227398}] 2018-02-04 11:00:33,911 INFO [PEWorker-15] procedure.MasterProcedureScheduler(883): pid=14, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=879a95042a6098ee953dbfe218826b40, target=ubuntu,36383,1517713227466 tabletest tabletest,ggg,1517713232695.879a95042a6098ee953dbfe218826b40. 2018-02-04 11:00:33,912 INFO [PEWorker-2] procedure.MasterProcedureScheduler(883): pid=11, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=68c9676b485038d63fd2cf8a435420e5, target=ubuntu,39267,1517713227316 tabletest tabletest,ddd,1517713232695.68c9676b485038d63fd2cf8a435420e5. 2018-02-04 11:00:33,912 INFO [PEWorker-3] procedure.MasterProcedureScheduler(883): pid=10, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=cd5c06da9ebf17554a264cbdd6fcd9ef, target=ubuntu,46739,1517713227398 tabletest tabletest,ccc,1517713232695.cd5c06da9ebf17554a264cbdd6fcd9ef. 2018-02-04 11:00:33,913 INFO [PEWorker-4] procedure.MasterProcedureScheduler(883): pid=9, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=756819a3f903733cb342d3913aab52a6, target=ubuntu,36383,1517713227466 tabletest tabletest,bbb,1517713232695.756819a3f903733cb342d3913aab52a6. 2018-02-04 11:00:33,913 INFO [PEWorker-5] procedure.MasterProcedureScheduler(883): pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=4f8ea7a76697ee6c4cd92147cc852db7, target=ubuntu,39267,1517713227316 tabletest tabletest,,1517713232695.4f8ea7a76697ee6c4cd92147cc852db7. 2018-02-04 11:00:33,914 INFO [PEWorker-15] assignment.AssignProcedure(211): Start pid=14, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=879a95042a6098ee953dbfe218826b40, target=ubuntu,36383,1517713227466; rit=OFFLINE, location=ubuntu,36383,1517713227466; forceNewPlan=false, retain=false 2018-02-04 11:00:33,917 INFO [PEWorker-2] assignment.AssignProcedure(211): Start pid=11, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=68c9676b485038d63fd2cf8a435420e5, target=ubuntu,39267,1517713227316; rit=OFFLINE, location=ubuntu,39267,1517713227316; forceNewPlan=false, retain=false 2018-02-04 11:00:33,917 INFO [PEWorker-16] procedure.MasterProcedureScheduler(883): pid=13, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=7b96ff3277efd9751a2175b9c2b4d481, target=ubuntu,39267,1517713227316 tabletest tabletest,fff,1517713232695.7b96ff3277efd9751a2175b9c2b4d481. 2018-02-04 11:00:33,918 INFO [PEWorker-1] procedure.MasterProcedureScheduler(883): pid=12, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=14343105008c3cdd39a78990296fbe94, target=ubuntu,39267,1517713227316 tabletest tabletest,eee,1517713232695.14343105008c3cdd39a78990296fbe94. 2018-02-04 11:00:33,918 INFO [PEWorker-6] procedure.MasterProcedureScheduler(883): pid=15, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=ad4f7bcab0a803ddae20b2866466d222, target=ubuntu,39267,1517713227316 tabletest tabletest,hhh,1517713232695.ad4f7bcab0a803ddae20b2866466d222. 2018-02-04 11:00:33,919 INFO [PEWorker-5] assignment.AssignProcedure(211): Start pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=4f8ea7a76697ee6c4cd92147cc852db7, target=ubuntu,39267,1517713227316; rit=OFFLINE, location=ubuntu,39267,1517713227316; forceNewPlan=false, retain=false 2018-02-04 11:00:33,920 INFO [PEWorker-4] assignment.AssignProcedure(211): Start pid=9, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=756819a3f903733cb342d3913aab52a6, target=ubuntu,36383,1517713227466; rit=OFFLINE, location=ubuntu,36383,1517713227466; forceNewPlan=false, retain=false 2018-02-04 11:00:33,922 INFO [PEWorker-8] procedure.MasterProcedureScheduler(883): pid=16, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=f1f365dfd0648c0fb5d9848c1db452cc, target=ubuntu,39267,1517713227316 tabletest tabletest,iii,1517713232695.f1f365dfd0648c0fb5d9848c1db452cc. 2018-02-04 11:00:33,923 INFO [PEWorker-7] procedure.MasterProcedureScheduler(883): pid=17, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=e4481a648e0b377eb5bf59bfb99dfd88, target=ubuntu,46739,1517713227398 tabletest tabletest,jjj,1517713232695.e4481a648e0b377eb5bf59bfb99dfd88. 2018-02-04 11:00:33,924 INFO [PEWorker-1] assignment.AssignProcedure(211): Start pid=12, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=14343105008c3cdd39a78990296fbe94, target=ubuntu,39267,1517713227316; rit=OFFLINE, location=ubuntu,39267,1517713227316; forceNewPlan=false, retain=false 2018-02-04 11:00:33,925 INFO [PEWorker-3] assignment.AssignProcedure(211): Start pid=10, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=cd5c06da9ebf17554a264cbdd6fcd9ef, target=ubuntu,46739,1517713227398; rit=OFFLINE, location=ubuntu,46739,1517713227398; forceNewPlan=false, retain=false 2018-02-04 11:00:33,925 INFO [PEWorker-16] assignment.AssignProcedure(211): Start pid=13, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=7b96ff3277efd9751a2175b9c2b4d481, target=ubuntu,39267,1517713227316; rit=OFFLINE, location=ubuntu,39267,1517713227316; forceNewPlan=false, retain=false 2018-02-04 11:00:33,927 INFO [PEWorker-8] assignment.AssignProcedure(211): Start pid=16, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=f1f365dfd0648c0fb5d9848c1db452cc, target=ubuntu,39267,1517713227316; rit=OFFLINE, location=ubuntu,39267,1517713227316; forceNewPlan=false, retain=false 2018-02-04 11:00:33,927 INFO [PEWorker-9] procedure.MasterProcedureScheduler(883): pid=18, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=a127c3f989561b0b55d12be9bd499407, target=ubuntu,46739,1517713227398 tabletest tabletest,kkk,1517713232695.a127c3f989561b0b55d12be9bd499407. 2018-02-04 11:00:33,929 INFO [PEWorker-6] assignment.AssignProcedure(211): Start pid=15, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=ad4f7bcab0a803ddae20b2866466d222, target=ubuntu,39267,1517713227316; rit=OFFLINE, location=ubuntu,39267,1517713227316; forceNewPlan=false, retain=false 2018-02-04 11:00:33,930 INFO [PEWorker-7] assignment.AssignProcedure(211): Start pid=17, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=e4481a648e0b377eb5bf59bfb99dfd88, target=ubuntu,46739,1517713227398; rit=OFFLINE, location=ubuntu,46739,1517713227398; forceNewPlan=false, retain=false 2018-02-04 11:00:33,932 INFO [PEWorker-10] procedure.MasterProcedureScheduler(883): pid=19, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=85d5a900ae52e80b00b89bb74335621c, target=ubuntu,46739,1517713227398 tabletest tabletest,lll,1517713232695.85d5a900ae52e80b00b89bb74335621c. 2018-02-04 11:00:33,935 INFO [PEWorker-9] assignment.AssignProcedure(211): Start pid=18, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=a127c3f989561b0b55d12be9bd499407, target=ubuntu,46739,1517713227398; rit=OFFLINE, location=ubuntu,46739,1517713227398; forceNewPlan=false, retain=false 2018-02-04 11:00:33,936 INFO [PEWorker-11] procedure.MasterProcedureScheduler(883): pid=20, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=a19e4bf3b69ad0ea43c87985b14ec9fe, target=ubuntu,36383,1517713227466 tabletest tabletest,mmm,1517713232695.a19e4bf3b69ad0ea43c87985b14ec9fe. 2018-02-04 11:00:33,937 INFO [PEWorker-12] procedure.MasterProcedureScheduler(883): pid=21, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=7429e32b4c4e0aadde8a8154cbd1f071, target=ubuntu,36383,1517713227466 tabletest tabletest,nnn,1517713232695.7429e32b4c4e0aadde8a8154cbd1f071. 2018-02-04 11:00:33,938 INFO [PEWorker-13] procedure.MasterProcedureScheduler(883): pid=22, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=41f447d2065af74b219164f0a70d08b3, target=ubuntu,46739,1517713227398 tabletest tabletest,ooo,1517713232695.41f447d2065af74b219164f0a70d08b3. 2018-02-04 11:00:33,938 INFO [PEWorker-10] assignment.AssignProcedure(211): Start pid=19, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=85d5a900ae52e80b00b89bb74335621c, target=ubuntu,46739,1517713227398; rit=OFFLINE, location=ubuntu,46739,1517713227398; forceNewPlan=false, retain=false 2018-02-04 11:00:33,938 INFO [PEWorker-15] procedure.MasterProcedureScheduler(883): pid=23, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=12bc4be2348528b0bd4575ae03200fc3, target=ubuntu,36383,1517713227466 tabletest tabletest,ppp,1517713232695.12bc4be2348528b0bd4575ae03200fc3. 2018-02-04 11:00:33,939 INFO [PEWorker-2] procedure.MasterProcedureScheduler(883): pid=24, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=6a7b42fe3d02ea7aca1d05fa666e87ac, target=ubuntu,36383,1517713227466 tabletest tabletest,qqq,1517713232695.6a7b42fe3d02ea7aca1d05fa666e87ac. 2018-02-04 11:00:33,939 INFO [PEWorker-4] procedure.MasterProcedureScheduler(883): pid=28, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=9579a7c2f22113dfe574ba9569101416, target=ubuntu,36383,1517713227466 tabletest tabletest,uuu,1517713232695.9579a7c2f22113dfe574ba9569101416. 2018-02-04 11:00:33,940 INFO [PEWorker-3] procedure.MasterProcedureScheduler(883): pid=26, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=e063e6b2c2640eb6eafc8737a71fb5f5, target=ubuntu,39267,1517713227316 tabletest tabletest,sss,1517713232695.e063e6b2c2640eb6eafc8737a71fb5f5. 2018-02-04 11:00:33,940 INFO [PEWorker-5] procedure.MasterProcedureScheduler(883): pid=29, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=2a83d01250e1489bfa6bcdc2057bb1b5, target=ubuntu,36383,1517713227466 tabletest tabletest,vvv,1517713232695.2a83d01250e1489bfa6bcdc2057bb1b5. 2018-02-04 11:00:33,940 INFO [PEWorker-1] procedure.MasterProcedureScheduler(883): pid=27, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=31b3e09f2db65dd6057a91f45cb8b163, target=ubuntu,39267,1517713227316 tabletest tabletest,ttt,1517713232695.31b3e09f2db65dd6057a91f45cb8b163. 2018-02-04 11:00:33,940 INFO [PEWorker-16] procedure.MasterProcedureScheduler(883): pid=25, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=1b8fb6f4917edac2a9ec29ba9ebed5ef, target=ubuntu,46739,1517713227398 tabletest tabletest,rrr,1517713232695.1b8fb6f4917edac2a9ec29ba9ebed5ef. 2018-02-04 11:00:33,941 INFO [PEWorker-8] procedure.MasterProcedureScheduler(883): pid=30, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=7514f60148f3a526baae5c7ce2c78c1b, target=ubuntu,46739,1517713227398 tabletest tabletest,www,1517713232695.7514f60148f3a526baae5c7ce2c78c1b. 2018-02-04 11:00:33,941 INFO [PEWorker-14] procedure.MasterProcedureScheduler(883): pid=33, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=1da7d397be67579013b1f635bfbcb064, target=ubuntu,46739,1517713227398 tabletest tabletest,zzz,1517713232695.1da7d397be67579013b1f635bfbcb064. 2018-02-04 11:00:33,942 INFO [PEWorker-11] assignment.AssignProcedure(211): Start pid=20, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=a19e4bf3b69ad0ea43c87985b14ec9fe, target=ubuntu,36383,1517713227466; rit=OFFLINE, location=ubuntu,36383,1517713227466; forceNewPlan=false, retain=false 2018-02-04 11:00:33,944 INFO [PEWorker-12] assignment.AssignProcedure(211): Start pid=21, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=7429e32b4c4e0aadde8a8154cbd1f071, target=ubuntu,36383,1517713227466; rit=OFFLINE, location=ubuntu,36383,1517713227466; forceNewPlan=false, retain=false 2018-02-04 11:00:33,945 INFO [PEWorker-13] assignment.AssignProcedure(211): Start pid=22, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=41f447d2065af74b219164f0a70d08b3, target=ubuntu,46739,1517713227398; rit=OFFLINE, location=ubuntu,46739,1517713227398; forceNewPlan=false, retain=false 2018-02-04 11:00:33,946 INFO [PEWorker-15] assignment.AssignProcedure(211): Start pid=23, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=12bc4be2348528b0bd4575ae03200fc3, target=ubuntu,36383,1517713227466; rit=OFFLINE, location=ubuntu,36383,1517713227466; forceNewPlan=false, retain=false 2018-02-04 11:00:33,947 INFO [PEWorker-2] assignment.AssignProcedure(211): Start pid=24, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=6a7b42fe3d02ea7aca1d05fa666e87ac, target=ubuntu,36383,1517713227466; rit=OFFLINE, location=ubuntu,36383,1517713227466; forceNewPlan=false, retain=false 2018-02-04 11:00:33,955 INFO [PEWorker-6] procedure.MasterProcedureScheduler(883): pid=32, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=20457e81e436062b31548a08737ebba4, target=ubuntu,46739,1517713227398 tabletest tabletest,yyy,1517713232695.20457e81e436062b31548a08737ebba4. 2018-02-04 11:00:33,956 INFO [PEWorker-7] procedure.MasterProcedureScheduler(883): pid=31, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=66ce5e96488227599f1cd0e5e39a2982, target=ubuntu,39267,1517713227316 tabletest tabletest,xxx,1517713232695.66ce5e96488227599f1cd0e5e39a2982. 2018-02-04 11:00:33,957 INFO [PEWorker-4] assignment.AssignProcedure(211): Start pid=28, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=9579a7c2f22113dfe574ba9569101416, target=ubuntu,36383,1517713227466; rit=OFFLINE, location=ubuntu,36383,1517713227466; forceNewPlan=false, retain=false 2018-02-04 11:00:33,958 INFO [PEWorker-5] assignment.AssignProcedure(211): Start pid=29, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=2a83d01250e1489bfa6bcdc2057bb1b5, target=ubuntu,36383,1517713227466; rit=OFFLINE, location=ubuntu,36383,1517713227466; forceNewPlan=false, retain=false 2018-02-04 11:00:33,958 INFO [PEWorker-1] assignment.AssignProcedure(211): Start pid=27, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=31b3e09f2db65dd6057a91f45cb8b163, target=ubuntu,39267,1517713227316; rit=OFFLINE, location=ubuntu,39267,1517713227316; forceNewPlan=false, retain=false 2018-02-04 11:00:33,958 INFO [PEWorker-3] assignment.AssignProcedure(211): Start pid=26, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=e063e6b2c2640eb6eafc8737a71fb5f5, target=ubuntu,39267,1517713227316; rit=OFFLINE, location=ubuntu,39267,1517713227316; forceNewPlan=false, retain=false 2018-02-04 11:00:33,959 INFO [PEWorker-16] assignment.AssignProcedure(211): Start pid=25, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=1b8fb6f4917edac2a9ec29ba9ebed5ef, target=ubuntu,46739,1517713227398; rit=OFFLINE, location=ubuntu,46739,1517713227398; forceNewPlan=false, retain=false 2018-02-04 11:00:33,959 INFO [PEWorker-8] assignment.AssignProcedure(211): Start pid=30, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=7514f60148f3a526baae5c7ce2c78c1b, target=ubuntu,46739,1517713227398; rit=OFFLINE, location=ubuntu,46739,1517713227398; forceNewPlan=false, retain=false 2018-02-04 11:00:33,959 INFO [PEWorker-14] assignment.AssignProcedure(211): Start pid=33, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=1da7d397be67579013b1f635bfbcb064, target=ubuntu,46739,1517713227398; rit=OFFLINE, location=ubuntu,46739,1517713227398; forceNewPlan=false, retain=false 2018-02-04 11:00:33,962 INFO [PEWorker-6] assignment.AssignProcedure(211): Start pid=32, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=20457e81e436062b31548a08737ebba4, target=ubuntu,46739,1517713227398; rit=OFFLINE, location=ubuntu,46739,1517713227398; forceNewPlan=false, retain=false 2018-02-04 11:00:33,964 INFO [PEWorker-7] assignment.AssignProcedure(211): Start pid=31, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=tabletest, region=66ce5e96488227599f1cd0e5e39a2982, target=ubuntu,39267,1517713227316; rit=OFFLINE, location=ubuntu,39267,1517713227316; forceNewPlan=false, retain=false 2018-02-04 11:00:34,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39063] master.MasterRpcServices(1157): Checking to see if procedure is done pid=7 2018-02-04 11:00:34,090 INFO [master/ubuntu:0] balancer.BaseLoadBalancer(1470): Reassigned 26 regions. 26 retained the pre-restart assignment. 2018-02-04 11:00:34,118 INFO [PEWorker-9] assignment.RegionStateStore(201): pid=17 updating hbase:meta row=tabletest,jjj,1517713232695.e4481a648e0b377eb5bf59bfb99dfd88., regionState=OPENING, regionLocation=ubuntu,46739,1517713227398 2018-02-04 11:00:34,119 INFO [PEWorker-10] assignment.RegionStateStore(201): pid=19 updating hbase:meta row=tabletest,lll,1517713232695.85d5a900ae52e80b00b89bb74335621c., regionState=OPENING, regionLocation=ubuntu,46739,1517713227398 2018-02-04 11:00:34,120 INFO [PEWorker-11] assignment.RegionStateStore(201): pid=22 updating hbase:meta row=tabletest,ooo,1517713232695.41f447d2065af74b219164f0a70d08b3., regionState=OPENING, regionLocation=ubuntu,46739,1517713227398 2018-02-04 11:00:34,121 INFO [PEWorker-12] assignment.RegionStateStore(201): pid=30 updating hbase:meta row=tabletest,www,1517713232695.7514f60148f3a526baae5c7ce2c78c1b., regionState=OPENING, regionLocation=ubuntu,46739,1517713227398 2018-02-04 11:00:34,121 INFO [PEWorker-13] assignment.RegionStateStore(201): pid=10 updating hbase:meta row=tabletest,ccc,1517713232695.cd5c06da9ebf17554a264cbdd6fcd9ef., regionState=OPENING, regionLocation=ubuntu,46739,1517713227398 2018-02-04 11:00:34,122 INFO [PEWorker-15] assignment.RegionStateStore(201): pid=18 updating hbase:meta row=tabletest,kkk,1517713232695.a127c3f989561b0b55d12be9bd499407., regionState=OPENING, regionLocation=ubuntu,46739,1517713227398 2018-02-04 11:00:34,123 INFO [PEWorker-2] assignment.RegionStateStore(201): pid=32 updating hbase:meta row=tabletest,yyy,1517713232695.20457e81e436062b31548a08737ebba4., regionState=OPENING, regionLocation=ubuntu,46739,1517713227398 2018-02-04 11:00:34,124 INFO [PEWorker-4] assignment.RegionStateStore(201): pid=25 updating hbase:meta row=tabletest,rrr,1517713232695.1b8fb6f4917edac2a9ec29ba9ebed5ef., regionState=OPENING, regionLocation=ubuntu,46739,1517713227398 2018-02-04 11:00:34,125 INFO [PEWorker-1] assignment.RegionStateStore(201): pid=33 updating hbase:meta row=tabletest,zzz,1517713232695.1da7d397be67579013b1f635bfbcb064., regionState=OPENING, regionLocation=ubuntu,46739,1517713227398 2018-02-04 11:00:34,126 INFO [PEWorker-5] assignment.RegionStateStore(201): pid=8 updating hbase:meta row=tabletest,,1517713232695.4f8ea7a76697ee6c4cd92147cc852db7., regionState=OPENING, regionLocation=ubuntu,39267,1517713227316 2018-02-04 11:00:34,128 INFO [PEWorker-3] assignment.RegionStateStore(201): pid=11 updating hbase:meta row=tabletest,ddd,1517713232695.68c9676b485038d63fd2cf8a435420e5., regionState=OPENING, regionLocation=ubuntu,39267,1517713227316 2018-02-04 11:00:34,129 INFO [PEWorker-8] assignment.RegionStateStore(201): pid=26 updating hbase:meta row=tabletest,sss,1517713232695.e063e6b2c2640eb6eafc8737a71fb5f5., regionState=OPENING, regionLocation=ubuntu,39267,1517713227316 2018-02-04 11:00:34,131 INFO [PEWorker-14] assignment.RegionStateStore(201): pid=16 updating hbase:meta row=tabletest,iii,1517713232695.f1f365dfd0648c0fb5d9848c1db452cc., regionState=OPENING, regionLocation=ubuntu,39267,1517713227316 2018-02-04 11:00:34,132 INFO [PEWorker-6] assignment.RegionStateStore(201): pid=31 updating hbase:meta row=tabletest,xxx,1517713232695.66ce5e96488227599f1cd0e5e39a2982., regionState=OPENING, regionLocation=ubuntu,39267,1517713227316 2018-02-04 11:00:34,134 INFO [PEWorker-7] assignment.RegionStateStore(201): pid=12 updating hbase:meta row=tabletest,eee,1517713232695.14343105008c3cdd39a78990296fbe94., regionState=OPENING, regionLocation=ubuntu,39267,1517713227316 2018-02-04 11:00:34,128 INFO [PEWorker-16] assignment.RegionStateStore(201): pid=13 updating hbase:meta row=tabletest,fff,1517713232695.7b96ff3277efd9751a2175b9c2b4d481., regionState=OPENING, regionLocation=ubuntu,39267,1517713227316 2018-02-04 11:00:34,175 INFO [PEWorker-9] assignment.RegionTransitionProcedure(208): Dispatch pid=17, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=tabletest, region=e4481a648e0b377eb5bf59bfb99dfd88, target=ubuntu,46739,1517713227398; rit=OPENING, location=ubuntu,46739,1517713227398 2018-02-04 11:00:34,177 INFO [PEWorker-5] assignment.RegionTransitionProcedure(208): Dispatch pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=tabletest, region=4f8ea7a76697ee6c4cd92147cc852db7, target=ubuntu,39267,1517713227316; rit=OPENING, location=ubuntu,39267,1517713227316 2018-02-04 11:00:34,178 INFO [PEWorker-4] assignment.RegionTransitionProcedure(208): Dispatch pid=25, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=tabletest, region=1b8fb6f4917edac2a9ec29ba9ebed5ef, target=ubuntu,46739,1517713227398; rit=OPENING, location=ubuntu,46739,1517713227398 2018-02-04 11:00:34,179 INFO [PEWorker-2] assignment.RegionTransitionProcedure(208): Dispatch pid=32, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=tabletest, region=20457e81e436062b31548a08737ebba4, target=ubuntu,46739,1517713227398; rit=OPENING, location=ubuntu,46739,1517713227398 2018-02-04 11:00:34,186 INFO [PEWorker-1] assignment.RegionTransitionProcedure(208): Dispatch pid=33, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=tabletest, region=1da7d397be67579013b1f635bfbcb064, target=ubuntu,46739,1517713227398; rit=OPENING, location=ubuntu,46739,1517713227398 2018-02-04 11:00:34,190 INFO [PEWorker-10] assignment.RegionTransitionProcedure(208): Dispatch pid=19, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=tabletest, region=85d5a900ae52e80b00b89bb74335621c, target=ubuntu,46739,1517713227398; rit=OPENING, location=ubuntu,46739,1517713227398 2018-02-04 11:00:34,192 INFO [PEWorker-3] assignment.RegionTransitionProcedure(208): Dispatch pid=11, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=tabletest, region=68c9676b485038d63fd2cf8a435420e5, target=ubuntu,39267,1517713227316; rit=OPENING, location=ubuntu,39267,1517713227316 2018-02-04 11:00:34,196 INFO [PEWorker-15] assignment.RegionTransitionProcedure(208): Dispatch pid=18, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=tabletest, region=a127c3f989561b0b55d12be9bd499407, target=ubuntu,46739,1517713227398; rit=OPENING, location=ubuntu,46739,1517713227398 2018-02-04 11:00:34,196 INFO [PEWorker-16] assignment.RegionTransitionProcedure(208): Dispatch pid=13, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=tabletest, region=7b96ff3277efd9751a2175b9c2b4d481, target=ubuntu,39267,1517713227316; rit=OPENING, location=ubuntu,39267,1517713227316 2018-02-04 11:00:34,196 INFO [PEWorker-13] assignment.RegionTransitionProcedure(208): Dispatch pid=10, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=tabletest, region=cd5c06da9ebf17554a264cbdd6fcd9ef, target=ubuntu,46739,1517713227398; rit=OPENING, location=ubuntu,46739,1517713227398 2018-02-04 11:00:34,205 INFO [PEWorker-6] assignment.RegionTransitionProcedure(208): Dispatch pid=31, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=tabletest, region=66ce5e96488227599f1cd0e5e39a2982, target=ubuntu,39267,1517713227316; rit=OPENING, location=ubuntu,39267,1517713227316 2018-02-04 11:00:34,208 INFO [PEWorker-14] assignment.RegionTransitionProcedure(208): Dispatch pid=16, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=tabletest, region=f1f365dfd0648c0fb5d9848c1db452cc, target=ubuntu,39267,1517713227316; rit=OPENING, location=ubuntu,39267,1517713227316 2018-02-04 11:00:34,208 INFO [PEWorker-7] assignment.RegionTransitionProcedure(208): Dispatch pid=12, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=tabletest, region=14343105008c3cdd39a78990296fbe94, target=ubuntu,39267,1517713227316; rit=OPENING, location=ubuntu,39267,1517713227316 2018-02-04 11:00:34,208 INFO [PEWorker-8] assignment.RegionTransitionProcedure(208): Dispatch pid=26, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=tabletest, region=e063e6b2c2640eb6eafc8737a71fb5f5, target=ubuntu,39267,1517713227316; rit=OPENING, location=ubuntu,39267,1517713227316 2018-02-04 11:00:34,208 INFO [PEWorker-11] assignment.RegionTransitionProcedure(208): Dispatch pid=22, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=tabletest, region=41f447d2065af74b219164f0a70d08b3, target=ubuntu,46739,1517713227398; rit=OPENING, location=ubuntu,46739,1517713227398 2018-02-04 11:00:34,209 INFO [PEWorker-12] assignment.RegionTransitionProcedure(208): Dispatch pid=30, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=tabletest, region=7514f60148f3a526baae5c7ce2c78c1b, target=ubuntu,46739,1517713227398; rit=OPENING, location=ubuntu,46739,1517713227398 2018-02-04 11:00:34,213 INFO [PEWorker-9] assignment.RegionStateStore(201): pid=27 updating hbase:meta row=tabletest,ttt,1517713232695.31b3e09f2db65dd6057a91f45cb8b163., regionState=OPENING, regionLocation=ubuntu,39267,1517713227316 2018-02-04 11:00:34,213 INFO [PEWorker-10] assignment.RegionStateStore(201): pid=21 updating hbase:meta row=tabletest,nnn,1517713232695.7429e32b4c4e0aadde8a8154cbd1f071., regionState=OPENING, regionLocation=ubuntu,36383,1517713227466 2018-02-04 11:00:34,213 INFO [PEWorker-4] assignment.RegionStateStore(201): pid=24 updating hbase:meta row=tabletest,qqq,1517713232695.6a7b42fe3d02ea7aca1d05fa666e87ac., regionState=OPENING, regionLocation=ubuntu,36383,1517713227466 2018-02-04 11:00:34,213 INFO [PEWorker-1] assignment.RegionStateStore(201): pid=20 updating hbase:meta row=tabletest,mmm,1517713232695.a19e4bf3b69ad0ea43c87985b14ec9fe., regionState=OPENING, regionLocation=ubuntu,36383,1517713227466 2018-02-04 11:00:34,213 INFO [PEWorker-3] assignment.RegionStateStore(201): pid=28 updating hbase:meta row=tabletest,uuu,1517713232695.9579a7c2f22113dfe574ba9569101416., regionState=OPENING, regionLocation=ubuntu,36383,1517713227466 2018-02-04 11:00:34,213 INFO [PEWorker-5] assignment.RegionStateStore(201): pid=15 updating hbase:meta row=tabletest,hhh,1517713232695.ad4f7bcab0a803ddae20b2866466d222., regionState=OPENING, regionLocation=ubuntu,39267,1517713227316 2018-02-04 11:00:34,213 INFO [PEWorker-2] assignment.RegionStateStore(201): pid=14 updating hbase:meta row=tabletest,ggg,1517713232695.879a95042a6098ee953dbfe218826b40., regionState=OPENING, regionLocation=ubuntu,36383,1517713227466 2018-02-04 11:00:34,213 INFO [PEWorker-16] assignment.RegionStateStore(201): pid=29 updating hbase:meta row=tabletest,vvv,1517713232695.2a83d01250e1489bfa6bcdc2057bb1b5., regionState=OPENING, regionLocation=ubuntu,36383,1517713227466 2018-02-04 11:00:34,214 INFO [PEWorker-13] assignment.RegionStateStore(201): pid=23 updating hbase:meta row=tabletest,ppp,1517713232695.12bc4be2348528b0bd4575ae03200fc3., regionState=OPENING, regionLocation=ubuntu,36383,1517713227466 2018-02-04 11:00:34,214 INFO [PEWorker-15] assignment.RegionStateStore(201): pid=9 updating hbase:meta row=tabletest,bbb,1517713232695.756819a3f903733cb342d3913aab52a6., regionState=OPENING, regionLocation=ubuntu,36383,1517713227466 2018-02-04 11:00:34,220 INFO [PEWorker-9] assignment.RegionTransitionProcedure(208): Dispatch pid=27, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=tabletest, region=31b3e09f2db65dd6057a91f45cb8b163, target=ubuntu,39267,1517713227316; rit=OPENING, location=ubuntu,39267,1517713227316 2018-02-04 11:00:34,225 INFO [PEWorker-4] assignment.RegionTransitionProcedure(208): Dispatch pid=24, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=tabletest, region=6a7b42fe3d02ea7aca1d05fa666e87ac, target=ubuntu,36383,1517713227466; rit=OPENING, location=ubuntu,36383,1517713227466 2018-02-04 11:00:34,225 INFO [PEWorker-10] assignment.RegionTransitionProcedure(208): Dispatch pid=21, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=tabletest, region=7429e32b4c4e0aadde8a8154cbd1f071, target=ubuntu,36383,1517713227466; rit=OPENING, location=ubuntu,36383,1517713227466 2018-02-04 11:00:34,229 INFO [PEWorker-13] assignment.RegionTransitionProcedure(208): Dispatch pid=23, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=tabletest, region=12bc4be2348528b0bd4575ae03200fc3, target=ubuntu,36383,1517713227466; rit=OPENING, location=ubuntu,36383,1517713227466 2018-02-04 11:00:34,229 INFO [PEWorker-1] assignment.RegionTransitionProcedure(208): Dispatch pid=20, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=tabletest, region=a19e4bf3b69ad0ea43c87985b14ec9fe, target=ubuntu,36383,1517713227466; rit=OPENING, location=ubuntu,36383,1517713227466 2018-02-04 11:00:34,230 INFO [PEWorker-5] assignment.RegionTransitionProcedure(208): Dispatch pid=15, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=tabletest, region=ad4f7bcab0a803ddae20b2866466d222, target=ubuntu,39267,1517713227316; rit=OPENING, location=ubuntu,39267,1517713227316 2018-02-04 11:00:34,230 INFO [PEWorker-2] assignment.RegionTransitionProcedure(208): Dispatch pid=14, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=tabletest, region=879a95042a6098ee953dbfe218826b40, target=ubuntu,36383,1517713227466; rit=OPENING, location=ubuntu,36383,1517713227466 2018-02-04 11:00:34,230 INFO [PEWorker-15] assignment.RegionTransitionProcedure(208): Dispatch pid=9, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=tabletest, region=756819a3f903733cb342d3913aab52a6, target=ubuntu,36383,1517713227466; rit=OPENING, location=ubuntu,36383,1517713227466 2018-02-04 11:00:34,230 INFO [PEWorker-16] assignment.RegionTransitionProcedure(208): Dispatch pid=29, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=tabletest, region=2a83d01250e1489bfa6bcdc2057bb1b5, target=ubuntu,36383,1517713227466; rit=OPENING, location=ubuntu,36383,1517713227466 2018-02-04 11:00:34,231 INFO [PEWorker-3] assignment.RegionTransitionProcedure(208): Dispatch pid=28, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=tabletest, region=9579a7c2f22113dfe574ba9569101416, target=ubuntu,36383,1517713227466; rit=OPENING, location=ubuntu,36383,1517713227466 2018-02-04 11:00:34,335 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=39267] regionserver.RSRpcServices(1912): Open tabletest,xxx,1517713232695.66ce5e96488227599f1cd0e5e39a2982. 2018-02-04 11:00:34,337 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=46739] regionserver.RSRpcServices(1912): Open tabletest,rrr,1517713232695.1b8fb6f4917edac2a9ec29ba9ebed5ef. 2018-02-04 11:00:34,348 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=46739] regionserver.RSRpcServices(1912): Open tabletest,yyy,1517713232695.20457e81e436062b31548a08737ebba4. 2018-02-04 11:00:34,352 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(6969): Opening region: {ENCODED => 1b8fb6f4917edac2a9ec29ba9ebed5ef, NAME => 'tabletest,rrr,1517713232695.1b8fb6f4917edac2a9ec29ba9ebed5ef.', STARTKEY => 'rrr', ENDKEY => 'sss'} 2018-02-04 11:00:34,353 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table tabletest 1b8fb6f4917edac2a9ec29ba9ebed5ef 2018-02-04 11:00:34,353 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(809): Instantiated tabletest,rrr,1517713232695.1b8fb6f4917edac2a9ec29ba9ebed5ef. 2018-02-04 11:00:34,356 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=39267] regionserver.RSRpcServices(1912): Open tabletest,iii,1517713232695.f1f365dfd0648c0fb5d9848c1db452cc. 2018-02-04 11:00:34,360 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=46739] regionserver.RSRpcServices(1912): Open tabletest,kkk,1517713232695.a127c3f989561b0b55d12be9bd499407. 2018-02-04 11:00:34,362 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=46739] regionserver.RSRpcServices(1912): Open tabletest,ooo,1517713232695.41f447d2065af74b219164f0a70d08b3. 2018-02-04 11:00:34,362 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=46739] regionserver.RSRpcServices(1912): Open tabletest,jjj,1517713232695.e4481a648e0b377eb5bf59bfb99dfd88. 2018-02-04 11:00:34,363 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=46739] regionserver.RSRpcServices(1912): Open tabletest,ccc,1517713232695.cd5c06da9ebf17554a264cbdd6fcd9ef. 2018-02-04 11:00:34,363 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=46739] regionserver.RSRpcServices(1912): Open tabletest,www,1517713232695.7514f60148f3a526baae5c7ce2c78c1b. 2018-02-04 11:00:34,364 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=46739] regionserver.RSRpcServices(1912): Open tabletest,zzz,1517713232695.1da7d397be67579013b1f635bfbcb064. 2018-02-04 11:00:34,364 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=46739] regionserver.RSRpcServices(1912): Open tabletest,lll,1517713232695.85d5a900ae52e80b00b89bb74335621c. 2018-02-04 11:00:34,365 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(6969): Opening region: {ENCODED => 66ce5e96488227599f1cd0e5e39a2982, NAME => 'tabletest,xxx,1517713232695.66ce5e96488227599f1cd0e5e39a2982.', STARTKEY => 'xxx', ENDKEY => 'yyy'} 2018-02-04 11:00:34,367 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=39267] regionserver.RSRpcServices(1912): Open tabletest,ttt,1517713232695.31b3e09f2db65dd6057a91f45cb8b163. 2018-02-04 11:00:34,373 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(6969): Opening region: {ENCODED => 20457e81e436062b31548a08737ebba4, NAME => 'tabletest,yyy,1517713232695.20457e81e436062b31548a08737ebba4.', STARTKEY => 'yyy', ENDKEY => 'zzz'} 2018-02-04 11:00:34,373 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(6969): Opening region: {ENCODED => f1f365dfd0648c0fb5d9848c1db452cc, NAME => 'tabletest,iii,1517713232695.f1f365dfd0648c0fb5d9848c1db452cc.', STARTKEY => 'iii', ENDKEY => 'jjj'} 2018-02-04 11:00:34,373 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table tabletest 66ce5e96488227599f1cd0e5e39a2982 2018-02-04 11:00:34,373 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table tabletest f1f365dfd0648c0fb5d9848c1db452cc 2018-02-04 11:00:34,373 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table tabletest 20457e81e436062b31548a08737ebba4 2018-02-04 11:00:34,374 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=39267] regionserver.RSRpcServices(1912): Open tabletest,sss,1517713232695.e063e6b2c2640eb6eafc8737a71fb5f5. 2018-02-04 11:00:34,374 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=39267] regionserver.RSRpcServices(1912): Open tabletest,ddd,1517713232695.68c9676b485038d63fd2cf8a435420e5. 2018-02-04 11:00:34,374 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=39267] regionserver.RSRpcServices(1912): Open tabletest,fff,1517713232695.7b96ff3277efd9751a2175b9c2b4d481. 2018-02-04 11:00:34,374 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=39267] regionserver.RSRpcServices(1912): Open tabletest,,1517713232695.4f8ea7a76697ee6c4cd92147cc852db7. 2018-02-04 11:00:34,374 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=39267] regionserver.RSRpcServices(1912): Open tabletest,eee,1517713232695.14343105008c3cdd39a78990296fbe94. 2018-02-04 11:00:34,374 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=39267] regionserver.RSRpcServices(1912): Open tabletest,hhh,1517713232695.ad4f7bcab0a803ddae20b2866466d222. 2018-02-04 11:00:34,374 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(809): Instantiated tabletest,yyy,1517713232695.20457e81e436062b31548a08737ebba4. 2018-02-04 11:00:34,376 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(809): Instantiated tabletest,xxx,1517713232695.66ce5e96488227599f1cd0e5e39a2982. 2018-02-04 11:00:34,376 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(809): Instantiated tabletest,iii,1517713232695.f1f365dfd0648c0fb5d9848c1db452cc. 2018-02-04 11:00:34,378 DEBUG [RSProcedureDispatcher-pool3-t5] master.ServerManager(778): New admin connection to ubuntu,36383,1517713227466 2018-02-04 11:00:34,381 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(6969): Opening region: {ENCODED => 31b3e09f2db65dd6057a91f45cb8b163, NAME => 'tabletest,ttt,1517713232695.31b3e09f2db65dd6057a91f45cb8b163.', STARTKEY => 'ttt', ENDKEY => 'uuu'} 2018-02-04 11:00:34,382 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table tabletest 31b3e09f2db65dd6057a91f45cb8b163 2018-02-04 11:00:34,383 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(809): Instantiated tabletest,ttt,1517713232695.31b3e09f2db65dd6057a91f45cb8b163. 2018-02-04 11:00:34,382 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(6969): Opening region: {ENCODED => a127c3f989561b0b55d12be9bd499407, NAME => 'tabletest,kkk,1517713232695.a127c3f989561b0b55d12be9bd499407.', STARTKEY => 'kkk', ENDKEY => 'lll'} 2018-02-04 11:00:34,384 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table tabletest a127c3f989561b0b55d12be9bd499407 2018-02-04 11:00:34,384 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(809): Instantiated tabletest,kkk,1517713232695.a127c3f989561b0b55d12be9bd499407. 2018-02-04 11:00:34,386 DEBUG [RSProcedureDispatcher-pool3-t5] ipc.NettyRpcConnection(254): Connecting to ubuntu/127.0.1.1:36383 2018-02-04 11:00:34,395 DEBUG [StoreOpener-1b8fb6f4917edac2a9ec29ba9ebed5ef-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/1b8fb6f4917edac2a9ec29ba9ebed5ef/family 2018-02-04 11:00:34,396 DEBUG [StoreOpener-1b8fb6f4917edac2a9ec29ba9ebed5ef-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/1b8fb6f4917edac2a9ec29ba9ebed5ef/family 2018-02-04 11:00:34,406 DEBUG [StoreOpener-f1f365dfd0648c0fb5d9848c1db452cc-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/f1f365dfd0648c0fb5d9848c1db452cc/family 2018-02-04 11:00:34,407 DEBUG [StoreOpener-f1f365dfd0648c0fb5d9848c1db452cc-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/f1f365dfd0648c0fb5d9848c1db452cc/family 2018-02-04 11:00:34,423 DEBUG [StoreOpener-20457e81e436062b31548a08737ebba4-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/20457e81e436062b31548a08737ebba4/family 2018-02-04 11:00:34,423 DEBUG [StoreOpener-20457e81e436062b31548a08737ebba4-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/20457e81e436062b31548a08737ebba4/family 2018-02-04 11:00:34,424 DEBUG [StoreOpener-66ce5e96488227599f1cd0e5e39a2982-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/66ce5e96488227599f1cd0e5e39a2982/family 2018-02-04 11:00:34,424 DEBUG [StoreOpener-66ce5e96488227599f1cd0e5e39a2982-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/66ce5e96488227599f1cd0e5e39a2982/family 2018-02-04 11:00:34,436 INFO [RS-EventLoopGroup-5-4] ipc.ServerRpcConnection(528): Auth successful for zhangduo (auth:SIMPLE) 2018-02-04 11:00:34,437 DEBUG [StoreOpener-31b3e09f2db65dd6057a91f45cb8b163-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/31b3e09f2db65dd6057a91f45cb8b163/family 2018-02-04 11:00:34,437 DEBUG [StoreOpener-31b3e09f2db65dd6057a91f45cb8b163-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/31b3e09f2db65dd6057a91f45cb8b163/family 2018-02-04 11:00:34,438 INFO [RS-EventLoopGroup-5-4] ipc.ServerRpcConnection(558): Connection from 127.0.0.1 port: 49794 with version info: version: "3.0.0-SNAPSHOT" url: "git://ubuntu/home/zhangduo/hbase/code" revision: "7fd9ea800d3ec8bfbba59879d0c98e0268d0ccde" user: "zhangduo" date: "Sat Feb 3 18:15:39 CST 2018" src_checksum: "e1f78921fcd876d508017ada55edc99a" version_major: 3 version_minor: 0 2018-02-04 11:00:34,439 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=36383] regionserver.RSRpcServices(1912): Open tabletest,uuu,1517713232695.9579a7c2f22113dfe574ba9569101416. 2018-02-04 11:00:34,440 DEBUG [StoreOpener-a127c3f989561b0b55d12be9bd499407-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/a127c3f989561b0b55d12be9bd499407/family 2018-02-04 11:00:34,442 DEBUG [StoreOpener-a127c3f989561b0b55d12be9bd499407-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/a127c3f989561b0b55d12be9bd499407/family 2018-02-04 11:00:34,463 INFO [StoreOpener-a127c3f989561b0b55d12be9bd499407-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-02-04 11:00:34,463 INFO [StoreOpener-a127c3f989561b0b55d12be9bd499407-1] regionserver.HStore(294): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-02-04 11:00:34,463 INFO [StoreOpener-a127c3f989561b0b55d12be9bd499407-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=819720, freeSize=1116962296, maxSize=1117782016, heapSize=819720, minSize=1061892928, minFactor=0.95, multiSize=530946464, multiFactor=0.5, singleSize=265473232, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-02-04 11:00:34,464 INFO [StoreOpener-a127c3f989561b0b55d12be9bd499407-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-02-04 11:00:34,465 INFO [StoreOpener-f1f365dfd0648c0fb5d9848c1db452cc-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-02-04 11:00:34,465 INFO [StoreOpener-f1f365dfd0648c0fb5d9848c1db452cc-1] regionserver.HStore(294): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-02-04 11:00:34,465 INFO [StoreOpener-f1f365dfd0648c0fb5d9848c1db452cc-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=819720, freeSize=1116962296, maxSize=1117782016, heapSize=819720, minSize=1061892928, minFactor=0.95, multiSize=530946464, multiFactor=0.5, singleSize=265473232, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-02-04 11:00:34,466 INFO [StoreOpener-f1f365dfd0648c0fb5d9848c1db452cc-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-02-04 11:00:34,466 INFO [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(1035): Setting FlushNonSloppyStoresFirstPolicy for the region=tabletest,kkk,1517713232695.a127c3f989561b0b55d12be9bd499407. 2018-02-04 11:00:34,468 INFO [StoreOpener-20457e81e436062b31548a08737ebba4-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-02-04 11:00:34,468 INFO [StoreOpener-20457e81e436062b31548a08737ebba4-1] regionserver.HStore(294): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-02-04 11:00:34,469 INFO [StoreOpener-20457e81e436062b31548a08737ebba4-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=819720, freeSize=1116962296, maxSize=1117782016, heapSize=819720, minSize=1061892928, minFactor=0.95, multiSize=530946464, multiFactor=0.5, singleSize=265473232, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-02-04 11:00:34,469 INFO [StoreOpener-20457e81e436062b31548a08737ebba4-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-02-04 11:00:34,470 INFO [StoreOpener-1b8fb6f4917edac2a9ec29ba9ebed5ef-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-02-04 11:00:34,470 INFO [StoreOpener-1b8fb6f4917edac2a9ec29ba9ebed5ef-1] regionserver.HStore(294): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-02-04 11:00:34,470 INFO [StoreOpener-1b8fb6f4917edac2a9ec29ba9ebed5ef-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=819720, freeSize=1116962296, maxSize=1117782016, heapSize=819720, minSize=1061892928, minFactor=0.95, multiSize=530946464, multiFactor=0.5, singleSize=265473232, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-02-04 11:00:34,471 INFO [StoreOpener-1b8fb6f4917edac2a9ec29ba9ebed5ef-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-02-04 11:00:34,474 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=36383] regionserver.RSRpcServices(1912): Open tabletest,nnn,1517713232695.7429e32b4c4e0aadde8a8154cbd1f071. 2018-02-04 11:00:34,475 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=36383] regionserver.RSRpcServices(1912): Open tabletest,mmm,1517713232695.a19e4bf3b69ad0ea43c87985b14ec9fe. 2018-02-04 11:00:34,478 INFO [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(1035): Setting FlushNonSloppyStoresFirstPolicy for the region=tabletest,yyy,1517713232695.20457e81e436062b31548a08737ebba4. 2018-02-04 11:00:34,480 INFO [StoreOpener-31b3e09f2db65dd6057a91f45cb8b163-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-02-04 11:00:34,481 INFO [StoreOpener-31b3e09f2db65dd6057a91f45cb8b163-1] regionserver.HStore(294): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-02-04 11:00:34,481 INFO [StoreOpener-31b3e09f2db65dd6057a91f45cb8b163-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=819720, freeSize=1116962296, maxSize=1117782016, heapSize=819720, minSize=1061892928, minFactor=0.95, multiSize=530946464, multiFactor=0.5, singleSize=265473232, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-02-04 11:00:34,484 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=36383] regionserver.RSRpcServices(1912): Open tabletest,ggg,1517713232695.879a95042a6098ee953dbfe218826b40. 2018-02-04 11:00:34,485 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=36383] regionserver.RSRpcServices(1912): Open tabletest,bbb,1517713232695.756819a3f903733cb342d3913aab52a6. 2018-02-04 11:00:34,485 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=36383] regionserver.RSRpcServices(1912): Open tabletest,vvv,1517713232695.2a83d01250e1489bfa6bcdc2057bb1b5. 2018-02-04 11:00:34,486 INFO [StoreOpener-66ce5e96488227599f1cd0e5e39a2982-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-02-04 11:00:34,486 INFO [StoreOpener-66ce5e96488227599f1cd0e5e39a2982-1] regionserver.HStore(294): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-02-04 11:00:34,487 INFO [StoreOpener-66ce5e96488227599f1cd0e5e39a2982-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=819720, freeSize=1116962296, maxSize=1117782016, heapSize=819720, minSize=1061892928, minFactor=0.95, multiSize=530946464, multiFactor=0.5, singleSize=265473232, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-02-04 11:00:34,495 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(6969): Opening region: {ENCODED => 9579a7c2f22113dfe574ba9569101416, NAME => 'tabletest,uuu,1517713232695.9579a7c2f22113dfe574ba9569101416.', STARTKEY => 'uuu', ENDKEY => 'vvv'} 2018-02-04 11:00:34,498 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/a127c3f989561b0b55d12be9bd499407 2018-02-04 11:00:34,499 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table tabletest 9579a7c2f22113dfe574ba9569101416 2018-02-04 11:00:34,499 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(809): Instantiated tabletest,uuu,1517713232695.9579a7c2f22113dfe574ba9569101416. 2018-02-04 11:00:34,498 INFO [StoreOpener-66ce5e96488227599f1cd0e5e39a2982-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-02-04 11:00:34,503 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(6969): Opening region: {ENCODED => a19e4bf3b69ad0ea43c87985b14ec9fe, NAME => 'tabletest,mmm,1517713232695.a19e4bf3b69ad0ea43c87985b14ec9fe.', STARTKEY => 'mmm', ENDKEY => 'nnn'} 2018-02-04 11:00:34,496 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(6969): Opening region: {ENCODED => 7429e32b4c4e0aadde8a8154cbd1f071, NAME => 'tabletest,nnn,1517713232695.7429e32b4c4e0aadde8a8154cbd1f071.', STARTKEY => 'nnn', ENDKEY => 'ooo'} 2018-02-04 11:00:34,506 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table tabletest 7429e32b4c4e0aadde8a8154cbd1f071 2018-02-04 11:00:34,506 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(809): Instantiated tabletest,nnn,1517713232695.7429e32b4c4e0aadde8a8154cbd1f071. 2018-02-04 11:00:34,510 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table tabletest a19e4bf3b69ad0ea43c87985b14ec9fe 2018-02-04 11:00:34,510 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(809): Instantiated tabletest,mmm,1517713232695.a19e4bf3b69ad0ea43c87985b14ec9fe. 2018-02-04 11:00:34,512 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table tabletest descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-02-04 11:00:34,513 DEBUG [StoreOpener-9579a7c2f22113dfe574ba9569101416-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/9579a7c2f22113dfe574ba9569101416/family 2018-02-04 11:00:34,514 DEBUG [StoreOpener-9579a7c2f22113dfe574ba9569101416-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/9579a7c2f22113dfe574ba9569101416/family 2018-02-04 11:00:34,515 INFO [StoreOpener-9579a7c2f22113dfe574ba9569101416-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-02-04 11:00:34,517 INFO [StoreOpener-9579a7c2f22113dfe574ba9569101416-1] regionserver.HStore(294): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-02-04 11:00:34,517 INFO [StoreOpener-9579a7c2f22113dfe574ba9569101416-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=819720, freeSize=1116962296, maxSize=1117782016, heapSize=819720, minSize=1061892928, minFactor=0.95, multiSize=530946464, multiFactor=0.5, singleSize=265473232, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-02-04 11:00:34,518 INFO [StoreOpener-9579a7c2f22113dfe574ba9569101416-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-02-04 11:00:34,521 INFO [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(1035): Setting FlushNonSloppyStoresFirstPolicy for the region=tabletest,uuu,1517713232695.9579a7c2f22113dfe574ba9569101416. 2018-02-04 11:00:34,498 INFO [StoreOpener-31b3e09f2db65dd6057a91f45cb8b163-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-02-04 11:00:34,498 INFO [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(1035): Setting FlushNonSloppyStoresFirstPolicy for the region=tabletest,iii,1517713232695.f1f365dfd0648c0fb5d9848c1db452cc. 2018-02-04 11:00:34,498 INFO [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(1035): Setting FlushNonSloppyStoresFirstPolicy for the region=tabletest,rrr,1517713232695.1b8fb6f4917edac2a9ec29ba9ebed5ef. 2018-02-04 11:00:34,496 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=36383] regionserver.RSRpcServices(1912): Open tabletest,qqq,1517713232695.6a7b42fe3d02ea7aca1d05fa666e87ac. 2018-02-04 11:00:34,506 INFO [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(1035): Setting FlushNonSloppyStoresFirstPolicy for the region=tabletest,xxx,1517713232695.66ce5e96488227599f1cd0e5e39a2982. 2018-02-04 11:00:34,499 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/20457e81e436062b31548a08737ebba4 2018-02-04 11:00:34,544 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] wal.WALSplitter(675): Wrote file=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/a127c3f989561b0b55d12be9bd499407/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-02-04 11:00:34,544 INFO [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(962): Onlined a127c3f989561b0b55d12be9bd499407; next sequenceid=2 2018-02-04 11:00:34,548 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/f1f365dfd0648c0fb5d9848c1db452cc 2018-02-04 11:00:34,544 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=36383] regionserver.RSRpcServices(1912): Open tabletest,ppp,1517713232695.12bc4be2348528b0bd4575ae03200fc3. 2018-02-04 11:00:34,545 DEBUG [StoreOpener-7429e32b4c4e0aadde8a8154cbd1f071-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/7429e32b4c4e0aadde8a8154cbd1f071/family 2018-02-04 11:00:34,562 DEBUG [StoreOpener-7429e32b4c4e0aadde8a8154cbd1f071-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/7429e32b4c4e0aadde8a8154cbd1f071/family 2018-02-04 11:00:34,568 INFO [StoreOpener-7429e32b4c4e0aadde8a8154cbd1f071-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-02-04 11:00:34,568 INFO [StoreOpener-7429e32b4c4e0aadde8a8154cbd1f071-1] regionserver.HStore(294): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-02-04 11:00:34,569 INFO [StoreOpener-7429e32b4c4e0aadde8a8154cbd1f071-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=819720, freeSize=1116962296, maxSize=1117782016, heapSize=819720, minSize=1061892928, minFactor=0.95, multiSize=530946464, multiFactor=0.5, singleSize=265473232, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-02-04 11:00:34,569 INFO [StoreOpener-7429e32b4c4e0aadde8a8154cbd1f071-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-02-04 11:00:34,567 INFO [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(1035): Setting FlushNonSloppyStoresFirstPolicy for the region=tabletest,ttt,1517713232695.31b3e09f2db65dd6057a91f45cb8b163. 2018-02-04 11:00:34,567 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/1b8fb6f4917edac2a9ec29ba9ebed5ef 2018-02-04 11:00:34,567 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/9579a7c2f22113dfe574ba9569101416 2018-02-04 11:00:34,566 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/66ce5e96488227599f1cd0e5e39a2982 2018-02-04 11:00:34,571 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table tabletest descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-02-04 11:00:34,571 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/31b3e09f2db65dd6057a91f45cb8b163 2018-02-04 11:00:34,580 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table tabletest descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-02-04 11:00:34,580 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table tabletest descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-02-04 11:00:34,584 INFO [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(1035): Setting FlushNonSloppyStoresFirstPolicy for the region=tabletest,nnn,1517713232695.7429e32b4c4e0aadde8a8154cbd1f071. 2018-02-04 11:00:34,585 DEBUG [StoreOpener-a19e4bf3b69ad0ea43c87985b14ec9fe-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/a19e4bf3b69ad0ea43c87985b14ec9fe/family 2018-02-04 11:00:34,585 DEBUG [StoreOpener-a19e4bf3b69ad0ea43c87985b14ec9fe-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/a19e4bf3b69ad0ea43c87985b14ec9fe/family 2018-02-04 11:00:34,585 INFO [StoreOpener-a19e4bf3b69ad0ea43c87985b14ec9fe-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-02-04 11:00:34,586 INFO [StoreOpener-a19e4bf3b69ad0ea43c87985b14ec9fe-1] regionserver.HStore(294): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-02-04 11:00:34,586 INFO [StoreOpener-a19e4bf3b69ad0ea43c87985b14ec9fe-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=819720, freeSize=1116962296, maxSize=1117782016, heapSize=819720, minSize=1061892928, minFactor=0.95, multiSize=530946464, multiFactor=0.5, singleSize=265473232, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-02-04 11:00:34,586 INFO [StoreOpener-a19e4bf3b69ad0ea43c87985b14ec9fe-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-02-04 11:00:34,587 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table tabletest descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-02-04 11:00:34,589 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/7429e32b4c4e0aadde8a8154cbd1f071 2018-02-04 11:00:34,589 INFO [PostOpenDeployTasks:a127c3f989561b0b55d12be9bd499407] regionserver.HRegionServer(2166): Post open deploy tasks for tabletest,kkk,1517713232695.a127c3f989561b0b55d12be9bd499407. 2018-02-04 11:00:34,596 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table tabletest descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-02-04 11:00:34,601 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table tabletest descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-02-04 11:00:34,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=39063] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=18, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=tabletest, region=a127c3f989561b0b55d12be9bd499407, target=ubuntu,46739,1517713227398; rit=OPENING, location=ubuntu,46739,1517713227398 2018-02-04 11:00:34,612 INFO [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(1035): Setting FlushNonSloppyStoresFirstPolicy for the region=tabletest,mmm,1517713232695.a19e4bf3b69ad0ea43c87985b14ec9fe. 2018-02-04 11:00:34,614 INFO [PEWorker-6] assignment.RegionStateStore(201): pid=18 updating hbase:meta row=tabletest,kkk,1517713232695.a127c3f989561b0b55d12be9bd499407., regionState=OPEN, openSeqNum=2, regionLocation=ubuntu,46739,1517713227398 2018-02-04 11:00:34,614 DEBUG [PostOpenDeployTasks:a127c3f989561b0b55d12be9bd499407] regionserver.HRegionServer(2190): Finished post open deploy task for tabletest,kkk,1517713232695.a127c3f989561b0b55d12be9bd499407. 2018-02-04 11:00:34,626 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table tabletest descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-02-04 11:00:34,637 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/a19e4bf3b69ad0ea43c87985b14ec9fe 2018-02-04 11:00:34,648 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] handler.OpenRegionHandler(128): Opened tabletest,kkk,1517713232695.a127c3f989561b0b55d12be9bd499407. on ubuntu,46739,1517713227398 2018-02-04 11:00:34,655 INFO [PEWorker-6] procedure2.ProcedureExecutor(1249): Finished pid=18, ppid=7, state=SUCCESS; AssignProcedure table=tabletest, region=a127c3f989561b0b55d12be9bd499407, target=ubuntu,46739,1517713227398 in 736msec 2018-02-04 11:00:34,656 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/66ce5e96488227599f1cd0e5e39a2982/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-02-04 11:00:34,657 INFO [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(962): Onlined 66ce5e96488227599f1cd0e5e39a2982; next sequenceid=2 2018-02-04 11:00:34,658 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(6969): Opening region: {ENCODED => 41f447d2065af74b219164f0a70d08b3, NAME => 'tabletest,ooo,1517713232695.41f447d2065af74b219164f0a70d08b3.', STARTKEY => 'ooo', ENDKEY => 'ppp'} 2018-02-04 11:00:34,659 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table tabletest 41f447d2065af74b219164f0a70d08b3 2018-02-04 11:00:34,660 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(809): Instantiated tabletest,ooo,1517713232695.41f447d2065af74b219164f0a70d08b3. 2018-02-04 11:00:34,668 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table tabletest descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-02-04 11:00:34,674 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/1b8fb6f4917edac2a9ec29ba9ebed5ef/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-02-04 11:00:34,674 INFO [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(962): Onlined 1b8fb6f4917edac2a9ec29ba9ebed5ef; next sequenceid=2 2018-02-04 11:00:34,675 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] wal.WALSplitter(675): Wrote file=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/f1f365dfd0648c0fb5d9848c1db452cc/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-02-04 11:00:34,675 INFO [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(962): Onlined f1f365dfd0648c0fb5d9848c1db452cc; next sequenceid=2 2018-02-04 11:00:34,678 INFO [PostOpenDeployTasks:66ce5e96488227599f1cd0e5e39a2982] regionserver.HRegionServer(2166): Post open deploy tasks for tabletest,xxx,1517713232695.66ce5e96488227599f1cd0e5e39a2982. 2018-02-04 11:00:34,679 INFO [PostOpenDeployTasks:1b8fb6f4917edac2a9ec29ba9ebed5ef] regionserver.HRegionServer(2166): Post open deploy tasks for tabletest,rrr,1517713232695.1b8fb6f4917edac2a9ec29ba9ebed5ef. 2018-02-04 11:00:34,681 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] wal.WALSplitter(675): Wrote file=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/7429e32b4c4e0aadde8a8154cbd1f071/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-02-04 11:00:34,681 INFO [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(962): Onlined 7429e32b4c4e0aadde8a8154cbd1f071; next sequenceid=2 2018-02-04 11:00:34,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=39063] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=25, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=tabletest, region=1b8fb6f4917edac2a9ec29ba9ebed5ef, target=ubuntu,46739,1517713227398; rit=OPENING, location=ubuntu,46739,1517713227398 2018-02-04 11:00:34,683 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] wal.WALSplitter(675): Wrote file=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/31b3e09f2db65dd6057a91f45cb8b163/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-02-04 11:00:34,684 INFO [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(962): Onlined 31b3e09f2db65dd6057a91f45cb8b163; next sequenceid=2 2018-02-04 11:00:34,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39063] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=31, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=tabletest, region=66ce5e96488227599f1cd0e5e39a2982, target=ubuntu,39267,1517713227316; rit=OPENING, location=ubuntu,39267,1517713227316 2018-02-04 11:00:34,684 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] wal.WALSplitter(675): Wrote file=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/20457e81e436062b31548a08737ebba4/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-02-04 11:00:34,685 INFO [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(962): Onlined 20457e81e436062b31548a08737ebba4; next sequenceid=2 2018-02-04 11:00:34,685 INFO [PEWorker-14] assignment.RegionStateStore(201): pid=25 updating hbase:meta row=tabletest,rrr,1517713232695.1b8fb6f4917edac2a9ec29ba9ebed5ef., regionState=OPEN, openSeqNum=2, regionLocation=ubuntu,46739,1517713227398 2018-02-04 11:00:34,686 INFO [PEWorker-7] assignment.RegionStateStore(201): pid=31 updating hbase:meta row=tabletest,xxx,1517713232695.66ce5e96488227599f1cd0e5e39a2982., regionState=OPEN, openSeqNum=2, regionLocation=ubuntu,39267,1517713227316 2018-02-04 11:00:34,687 DEBUG [PostOpenDeployTasks:66ce5e96488227599f1cd0e5e39a2982] regionserver.HRegionServer(2190): Finished post open deploy task for tabletest,xxx,1517713232695.66ce5e96488227599f1cd0e5e39a2982. 2018-02-04 11:00:34,686 DEBUG [PostOpenDeployTasks:1b8fb6f4917edac2a9ec29ba9ebed5ef] regionserver.HRegionServer(2190): Finished post open deploy task for tabletest,rrr,1517713232695.1b8fb6f4917edac2a9ec29ba9ebed5ef. 2018-02-04 11:00:34,687 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] handler.OpenRegionHandler(128): Opened tabletest,rrr,1517713232695.1b8fb6f4917edac2a9ec29ba9ebed5ef. on ubuntu,46739,1517713227398 2018-02-04 11:00:34,688 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(6969): Opening region: {ENCODED => e4481a648e0b377eb5bf59bfb99dfd88, NAME => 'tabletest,jjj,1517713232695.e4481a648e0b377eb5bf59bfb99dfd88.', STARTKEY => 'jjj', ENDKEY => 'kkk'} 2018-02-04 11:00:34,689 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table tabletest e4481a648e0b377eb5bf59bfb99dfd88 2018-02-04 11:00:34,689 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(809): Instantiated tabletest,jjj,1517713232695.e4481a648e0b377eb5bf59bfb99dfd88. 2018-02-04 11:00:34,699 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] handler.OpenRegionHandler(128): Opened tabletest,xxx,1517713232695.66ce5e96488227599f1cd0e5e39a2982. on ubuntu,39267,1517713227316 2018-02-04 11:00:34,700 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(6969): Opening region: {ENCODED => e063e6b2c2640eb6eafc8737a71fb5f5, NAME => 'tabletest,sss,1517713232695.e063e6b2c2640eb6eafc8737a71fb5f5.', STARTKEY => 'sss', ENDKEY => 'ttt'} 2018-02-04 11:00:34,701 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table tabletest e063e6b2c2640eb6eafc8737a71fb5f5 2018-02-04 11:00:34,701 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(809): Instantiated tabletest,sss,1517713232695.e063e6b2c2640eb6eafc8737a71fb5f5. 2018-02-04 11:00:34,703 INFO [PostOpenDeployTasks:f1f365dfd0648c0fb5d9848c1db452cc] regionserver.HRegionServer(2166): Post open deploy tasks for tabletest,iii,1517713232695.f1f365dfd0648c0fb5d9848c1db452cc. 2018-02-04 11:00:34,703 INFO [PostOpenDeployTasks:7429e32b4c4e0aadde8a8154cbd1f071] regionserver.HRegionServer(2166): Post open deploy tasks for tabletest,nnn,1517713232695.7429e32b4c4e0aadde8a8154cbd1f071. 2018-02-04 11:00:34,703 INFO [PostOpenDeployTasks:20457e81e436062b31548a08737ebba4] regionserver.HRegionServer(2166): Post open deploy tasks for tabletest,yyy,1517713232695.20457e81e436062b31548a08737ebba4. 2018-02-04 11:00:34,705 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] wal.WALSplitter(675): Wrote file=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/a19e4bf3b69ad0ea43c87985b14ec9fe/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-02-04 11:00:34,705 INFO [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(962): Onlined a19e4bf3b69ad0ea43c87985b14ec9fe; next sequenceid=2 2018-02-04 11:00:34,705 INFO [PostOpenDeployTasks:31b3e09f2db65dd6057a91f45cb8b163] regionserver.HRegionServer(2166): Post open deploy tasks for tabletest,ttt,1517713232695.31b3e09f2db65dd6057a91f45cb8b163. 2018-02-04 11:00:34,707 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/9579a7c2f22113dfe574ba9569101416/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-02-04 11:00:34,707 INFO [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(962): Onlined 9579a7c2f22113dfe574ba9569101416; next sequenceid=2 2018-02-04 11:00:34,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39063] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=16, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=tabletest, region=f1f365dfd0648c0fb5d9848c1db452cc, target=ubuntu,39267,1517713227316; rit=OPENING, location=ubuntu,39267,1517713227316 2018-02-04 11:00:34,708 INFO [PEWorker-8] assignment.RegionStateStore(201): pid=16 updating hbase:meta row=tabletest,iii,1517713232695.f1f365dfd0648c0fb5d9848c1db452cc., regionState=OPEN, openSeqNum=2, regionLocation=ubuntu,39267,1517713227316 2018-02-04 11:00:34,709 DEBUG [PostOpenDeployTasks:f1f365dfd0648c0fb5d9848c1db452cc] regionserver.HRegionServer(2190): Finished post open deploy task for tabletest,iii,1517713232695.f1f365dfd0648c0fb5d9848c1db452cc. 2018-02-04 11:00:34,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=39063] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=32, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=tabletest, region=20457e81e436062b31548a08737ebba4, target=ubuntu,46739,1517713227398; rit=OPENING, location=ubuntu,46739,1517713227398 2018-02-04 11:00:34,709 INFO [PEWorker-11] assignment.RegionStateStore(201): pid=32 updating hbase:meta row=tabletest,yyy,1517713232695.20457e81e436062b31548a08737ebba4., regionState=OPEN, openSeqNum=2, regionLocation=ubuntu,46739,1517713227398 2018-02-04 11:00:34,711 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] handler.OpenRegionHandler(128): Opened tabletest,iii,1517713232695.f1f365dfd0648c0fb5d9848c1db452cc. on ubuntu,39267,1517713227316 2018-02-04 11:00:34,712 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(6969): Opening region: {ENCODED => 68c9676b485038d63fd2cf8a435420e5, NAME => 'tabletest,ddd,1517713232695.68c9676b485038d63fd2cf8a435420e5.', STARTKEY => 'ddd', ENDKEY => 'eee'} 2018-02-04 11:00:34,712 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table tabletest 68c9676b485038d63fd2cf8a435420e5 2018-02-04 11:00:34,712 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(809): Instantiated tabletest,ddd,1517713232695.68c9676b485038d63fd2cf8a435420e5. 2018-02-04 11:00:34,713 DEBUG [PostOpenDeployTasks:20457e81e436062b31548a08737ebba4] regionserver.HRegionServer(2190): Finished post open deploy task for tabletest,yyy,1517713232695.20457e81e436062b31548a08737ebba4. 2018-02-04 11:00:34,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=39063] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=27, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=tabletest, region=31b3e09f2db65dd6057a91f45cb8b163, target=ubuntu,39267,1517713227316; rit=OPENING, location=ubuntu,39267,1517713227316 2018-02-04 11:00:34,719 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] handler.OpenRegionHandler(128): Opened tabletest,yyy,1517713232695.20457e81e436062b31548a08737ebba4. on ubuntu,46739,1517713227398 2018-02-04 11:00:34,719 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(6969): Opening region: {ENCODED => cd5c06da9ebf17554a264cbdd6fcd9ef, NAME => 'tabletest,ccc,1517713232695.cd5c06da9ebf17554a264cbdd6fcd9ef.', STARTKEY => 'ccc', ENDKEY => 'ddd'} 2018-02-04 11:00:34,719 INFO [PEWorker-12] assignment.RegionStateStore(201): pid=27 updating hbase:meta row=tabletest,ttt,1517713232695.31b3e09f2db65dd6057a91f45cb8b163., regionState=OPEN, openSeqNum=2, regionLocation=ubuntu,39267,1517713227316 2018-02-04 11:00:34,720 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table tabletest cd5c06da9ebf17554a264cbdd6fcd9ef 2018-02-04 11:00:34,720 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(809): Instantiated tabletest,ccc,1517713232695.cd5c06da9ebf17554a264cbdd6fcd9ef. 2018-02-04 11:00:34,720 DEBUG [PostOpenDeployTasks:31b3e09f2db65dd6057a91f45cb8b163] regionserver.HRegionServer(2190): Finished post open deploy task for tabletest,ttt,1517713232695.31b3e09f2db65dd6057a91f45cb8b163. 2018-02-04 11:00:34,722 INFO [PostOpenDeployTasks:a19e4bf3b69ad0ea43c87985b14ec9fe] regionserver.HRegionServer(2166): Post open deploy tasks for tabletest,mmm,1517713232695.a19e4bf3b69ad0ea43c87985b14ec9fe. 2018-02-04 11:00:34,722 DEBUG [StoreOpener-e4481a648e0b377eb5bf59bfb99dfd88-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/e4481a648e0b377eb5bf59bfb99dfd88/family 2018-02-04 11:00:34,723 DEBUG [StoreOpener-e4481a648e0b377eb5bf59bfb99dfd88-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/e4481a648e0b377eb5bf59bfb99dfd88/family 2018-02-04 11:00:34,723 INFO [StoreOpener-e4481a648e0b377eb5bf59bfb99dfd88-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-02-04 11:00:34,723 INFO [StoreOpener-e4481a648e0b377eb5bf59bfb99dfd88-1] regionserver.HStore(294): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-02-04 11:00:34,723 INFO [StoreOpener-e4481a648e0b377eb5bf59bfb99dfd88-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=819720, freeSize=1116962296, maxSize=1117782016, heapSize=819720, minSize=1061892928, minFactor=0.95, multiSize=530946464, multiFactor=0.5, singleSize=265473232, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-02-04 11:00:34,723 INFO [StoreOpener-e4481a648e0b377eb5bf59bfb99dfd88-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-02-04 11:00:34,726 DEBUG [StoreOpener-68c9676b485038d63fd2cf8a435420e5-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/68c9676b485038d63fd2cf8a435420e5/family 2018-02-04 11:00:34,726 DEBUG [StoreOpener-68c9676b485038d63fd2cf8a435420e5-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/68c9676b485038d63fd2cf8a435420e5/family 2018-02-04 11:00:34,727 INFO [StoreOpener-68c9676b485038d63fd2cf8a435420e5-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-02-04 11:00:34,727 INFO [StoreOpener-68c9676b485038d63fd2cf8a435420e5-1] regionserver.HStore(294): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-02-04 11:00:34,727 INFO [StoreOpener-68c9676b485038d63fd2cf8a435420e5-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=819720, freeSize=1116962296, maxSize=1117782016, heapSize=819720, minSize=1061892928, minFactor=0.95, multiSize=530946464, multiFactor=0.5, singleSize=265473232, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-02-04 11:00:34,727 INFO [StoreOpener-68c9676b485038d63fd2cf8a435420e5-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-02-04 11:00:34,728 INFO [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(1035): Setting FlushNonSloppyStoresFirstPolicy for the region=tabletest,ddd,1517713232695.68c9676b485038d63fd2cf8a435420e5. 2018-02-04 11:00:34,731 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] handler.OpenRegionHandler(128): Opened tabletest,ttt,1517713232695.31b3e09f2db65dd6057a91f45cb8b163. on ubuntu,39267,1517713227316 2018-02-04 11:00:34,731 INFO [PostOpenDeployTasks:9579a7c2f22113dfe574ba9569101416] regionserver.HRegionServer(2166): Post open deploy tasks for tabletest,uuu,1517713232695.9579a7c2f22113dfe574ba9569101416. 2018-02-04 11:00:34,731 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(6969): Opening region: {ENCODED => 7b96ff3277efd9751a2175b9c2b4d481, NAME => 'tabletest,fff,1517713232695.7b96ff3277efd9751a2175b9c2b4d481.', STARTKEY => 'fff', ENDKEY => 'ggg'} 2018-02-04 11:00:34,731 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/68c9676b485038d63fd2cf8a435420e5 2018-02-04 11:00:34,732 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table tabletest 7b96ff3277efd9751a2175b9c2b4d481 2018-02-04 11:00:34,732 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(809): Instantiated tabletest,fff,1517713232695.7b96ff3277efd9751a2175b9c2b4d481. 2018-02-04 11:00:34,732 DEBUG [StoreOpener-41f447d2065af74b219164f0a70d08b3-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/41f447d2065af74b219164f0a70d08b3/family 2018-02-04 11:00:34,732 DEBUG [StoreOpener-41f447d2065af74b219164f0a70d08b3-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/41f447d2065af74b219164f0a70d08b3/family 2018-02-04 11:00:34,733 INFO [StoreOpener-41f447d2065af74b219164f0a70d08b3-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-02-04 11:00:34,736 INFO [StoreOpener-41f447d2065af74b219164f0a70d08b3-1] regionserver.HStore(294): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-02-04 11:00:34,736 INFO [StoreOpener-41f447d2065af74b219164f0a70d08b3-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=819720, freeSize=1116962296, maxSize=1117782016, heapSize=819720, minSize=1061892928, minFactor=0.95, multiSize=530946464, multiFactor=0.5, singleSize=265473232, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-02-04 11:00:34,736 INFO [StoreOpener-41f447d2065af74b219164f0a70d08b3-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-02-04 11:00:34,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39063] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=28, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=tabletest, region=9579a7c2f22113dfe574ba9569101416, target=ubuntu,36383,1517713227466; rit=OPENING, location=ubuntu,36383,1517713227466 2018-02-04 11:00:34,740 INFO [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(1035): Setting FlushNonSloppyStoresFirstPolicy for the region=tabletest,jjj,1517713232695.e4481a648e0b377eb5bf59bfb99dfd88. 2018-02-04 11:00:34,740 INFO [PEWorker-9] assignment.RegionStateStore(201): pid=28 updating hbase:meta row=tabletest,uuu,1517713232695.9579a7c2f22113dfe574ba9569101416., regionState=OPEN, openSeqNum=2, regionLocation=ubuntu,36383,1517713227466 2018-02-04 11:00:34,745 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/e4481a648e0b377eb5bf59bfb99dfd88 2018-02-04 11:00:34,750 INFO [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(1035): Setting FlushNonSloppyStoresFirstPolicy for the region=tabletest,ooo,1517713232695.41f447d2065af74b219164f0a70d08b3. 2018-02-04 11:00:34,751 DEBUG [PostOpenDeployTasks:9579a7c2f22113dfe574ba9569101416] regionserver.HRegionServer(2190): Finished post open deploy task for tabletest,uuu,1517713232695.9579a7c2f22113dfe574ba9569101416. 2018-02-04 11:00:34,753 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table tabletest descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-02-04 11:00:34,754 DEBUG [StoreOpener-e063e6b2c2640eb6eafc8737a71fb5f5-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/e063e6b2c2640eb6eafc8737a71fb5f5/family 2018-02-04 11:00:34,754 DEBUG [StoreOpener-e063e6b2c2640eb6eafc8737a71fb5f5-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/e063e6b2c2640eb6eafc8737a71fb5f5/family 2018-02-04 11:00:34,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=39063] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=21, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=tabletest, region=7429e32b4c4e0aadde8a8154cbd1f071, target=ubuntu,36383,1517713227466; rit=OPENING, location=ubuntu,36383,1517713227466 2018-02-04 11:00:34,787 INFO [StoreOpener-e063e6b2c2640eb6eafc8737a71fb5f5-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-02-04 11:00:34,787 INFO [StoreOpener-e063e6b2c2640eb6eafc8737a71fb5f5-1] regionserver.HStore(294): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-02-04 11:00:34,787 INFO [StoreOpener-e063e6b2c2640eb6eafc8737a71fb5f5-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=819720, freeSize=1116962296, maxSize=1117782016, heapSize=819720, minSize=1061892928, minFactor=0.95, multiSize=530946464, multiFactor=0.5, singleSize=265473232, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-02-04 11:00:34,788 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] handler.OpenRegionHandler(128): Opened tabletest,uuu,1517713232695.9579a7c2f22113dfe574ba9569101416. on ubuntu,36383,1517713227466 2018-02-04 11:00:34,788 INFO [StoreOpener-e063e6b2c2640eb6eafc8737a71fb5f5-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-02-04 11:00:34,788 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(6969): Opening region: {ENCODED => 879a95042a6098ee953dbfe218826b40, NAME => 'tabletest,ggg,1517713232695.879a95042a6098ee953dbfe218826b40.', STARTKEY => 'ggg', ENDKEY => 'hhh'} 2018-02-04 11:00:34,788 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2018-02-04 11:00:34,789 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2018-02-04 11:00:34,789 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2018-02-04 11:00:34,790 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2018-02-04 11:00:34,790 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2018-02-04 11:00:34,791 DEBUG [PostOpenDeployTasks:7429e32b4c4e0aadde8a8154cbd1f071] regionserver.HRegionServer(2190): Finished post open deploy task for tabletest,nnn,1517713232695.7429e32b4c4e0aadde8a8154cbd1f071. 2018-02-04 11:00:34,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39063] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=20, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=tabletest, region=a19e4bf3b69ad0ea43c87985b14ec9fe, target=ubuntu,36383,1517713227466; rit=OPENING, location=ubuntu,36383,1517713227466 2018-02-04 11:00:34,792 INFO [PEWorker-4] assignment.RegionStateStore(201): pid=20 updating hbase:meta row=tabletest,mmm,1517713232695.a19e4bf3b69ad0ea43c87985b14ec9fe., regionState=OPEN, openSeqNum=2, regionLocation=ubuntu,36383,1517713227466 2018-02-04 11:00:34,788 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table tabletest 879a95042a6098ee953dbfe218826b40 2018-02-04 11:00:34,793 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(809): Instantiated tabletest,ggg,1517713232695.879a95042a6098ee953dbfe218826b40. 2018-02-04 11:00:34,794 DEBUG [PostOpenDeployTasks:a19e4bf3b69ad0ea43c87985b14ec9fe] regionserver.HRegionServer(2190): Finished post open deploy task for tabletest,mmm,1517713232695.a19e4bf3b69ad0ea43c87985b14ec9fe. 2018-02-04 11:00:34,787 INFO [PEWorker-10] assignment.RegionStateStore(201): pid=21 updating hbase:meta row=tabletest,nnn,1517713232695.7429e32b4c4e0aadde8a8154cbd1f071., regionState=OPEN, openSeqNum=2, regionLocation=ubuntu,36383,1517713227466 2018-02-04 11:00:34,799 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table tabletest descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-02-04 11:00:34,805 INFO [PEWorker-14] procedure2.ProcedureExecutor(1249): Finished pid=25, ppid=7, state=SUCCESS; AssignProcedure table=tabletest, region=1b8fb6f4917edac2a9ec29ba9ebed5ef, target=ubuntu,46739,1517713227398 in 825msec 2018-02-04 11:00:34,806 INFO [PEWorker-7] procedure2.ProcedureExecutor(1249): Finished pid=31, ppid=7, state=SUCCESS; AssignProcedure table=tabletest, region=66ce5e96488227599f1cd0e5e39a2982, target=ubuntu,39267,1517713227316 in 848msec 2018-02-04 11:00:34,806 INFO [PEWorker-11] procedure2.ProcedureExecutor(1249): Finished pid=32, ppid=7, state=SUCCESS; AssignProcedure table=tabletest, region=20457e81e436062b31548a08737ebba4, target=ubuntu,46739,1517713227398 in 848msec 2018-02-04 11:00:34,806 INFO [PEWorker-8] procedure2.ProcedureExecutor(1249): Finished pid=16, ppid=7, state=SUCCESS; AssignProcedure table=tabletest, region=f1f365dfd0648c0fb5d9848c1db452cc, target=ubuntu,39267,1517713227316 in 863msec 2018-02-04 11:00:34,806 INFO [PEWorker-12] procedure2.ProcedureExecutor(1249): Finished pid=27, ppid=7, state=SUCCESS; AssignProcedure table=tabletest, region=31b3e09f2db65dd6057a91f45cb8b163, target=ubuntu,39267,1517713227316 in 861msec 2018-02-04 11:00:34,806 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/41f447d2065af74b219164f0a70d08b3 2018-02-04 11:00:34,808 INFO [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(1035): Setting FlushNonSloppyStoresFirstPolicy for the region=tabletest,sss,1517713232695.e063e6b2c2640eb6eafc8737a71fb5f5. 2018-02-04 11:00:34,823 DEBUG [StoreOpener-7b96ff3277efd9751a2175b9c2b4d481-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/7b96ff3277efd9751a2175b9c2b4d481/family 2018-02-04 11:00:34,823 DEBUG [StoreOpener-7b96ff3277efd9751a2175b9c2b4d481-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/7b96ff3277efd9751a2175b9c2b4d481/family 2018-02-04 11:00:34,823 INFO [StoreOpener-7b96ff3277efd9751a2175b9c2b4d481-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-02-04 11:00:34,824 INFO [StoreOpener-7b96ff3277efd9751a2175b9c2b4d481-1] regionserver.HStore(294): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-02-04 11:00:34,824 INFO [StoreOpener-7b96ff3277efd9751a2175b9c2b4d481-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=819720, freeSize=1116962296, maxSize=1117782016, heapSize=819720, minSize=1061892928, minFactor=0.95, multiSize=530946464, multiFactor=0.5, singleSize=265473232, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-02-04 11:00:34,825 INFO [StoreOpener-7b96ff3277efd9751a2175b9c2b4d481-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-02-04 11:00:34,837 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] handler.OpenRegionHandler(128): Opened tabletest,mmm,1517713232695.a19e4bf3b69ad0ea43c87985b14ec9fe. on ubuntu,36383,1517713227466 2018-02-04 11:00:34,838 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(6969): Opening region: {ENCODED => 756819a3f903733cb342d3913aab52a6, NAME => 'tabletest,bbb,1517713232695.756819a3f903733cb342d3913aab52a6.', STARTKEY => 'bbb', ENDKEY => 'ccc'} 2018-02-04 11:00:34,838 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] handler.OpenRegionHandler(128): Opened tabletest,nnn,1517713232695.7429e32b4c4e0aadde8a8154cbd1f071. on ubuntu,36383,1517713227466 2018-02-04 11:00:34,838 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table tabletest 756819a3f903733cb342d3913aab52a6 2018-02-04 11:00:34,838 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(6969): Opening region: {ENCODED => 2a83d01250e1489bfa6bcdc2057bb1b5, NAME => 'tabletest,vvv,1517713232695.2a83d01250e1489bfa6bcdc2057bb1b5.', STARTKEY => 'vvv', ENDKEY => 'www'} 2018-02-04 11:00:34,838 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table tabletest 2a83d01250e1489bfa6bcdc2057bb1b5 2018-02-04 11:00:34,838 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(809): Instantiated tabletest,vvv,1517713232695.2a83d01250e1489bfa6bcdc2057bb1b5. 2018-02-04 11:00:34,838 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(809): Instantiated tabletest,bbb,1517713232695.756819a3f903733cb342d3913aab52a6. 2018-02-04 11:00:34,845 DEBUG [StoreOpener-cd5c06da9ebf17554a264cbdd6fcd9ef-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/cd5c06da9ebf17554a264cbdd6fcd9ef/family 2018-02-04 11:00:34,845 DEBUG [StoreOpener-cd5c06da9ebf17554a264cbdd6fcd9ef-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/cd5c06da9ebf17554a264cbdd6fcd9ef/family 2018-02-04 11:00:34,845 INFO [StoreOpener-cd5c06da9ebf17554a264cbdd6fcd9ef-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-02-04 11:00:34,845 INFO [StoreOpener-cd5c06da9ebf17554a264cbdd6fcd9ef-1] regionserver.HStore(294): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-02-04 11:00:34,845 INFO [StoreOpener-cd5c06da9ebf17554a264cbdd6fcd9ef-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=819720, freeSize=1116962296, maxSize=1117782016, heapSize=819720, minSize=1061892928, minFactor=0.95, multiSize=530946464, multiFactor=0.5, singleSize=265473232, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-02-04 11:00:34,845 INFO [StoreOpener-cd5c06da9ebf17554a264cbdd6fcd9ef-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-02-04 11:00:34,847 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/e063e6b2c2640eb6eafc8737a71fb5f5 2018-02-04 11:00:34,851 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table tabletest descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-02-04 11:00:34,852 INFO [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(1035): Setting FlushNonSloppyStoresFirstPolicy for the region=tabletest,ccc,1517713232695.cd5c06da9ebf17554a264cbdd6fcd9ef. 2018-02-04 11:00:34,854 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table tabletest descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-02-04 11:00:34,869 INFO [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(1035): Setting FlushNonSloppyStoresFirstPolicy for the region=tabletest,fff,1517713232695.7b96ff3277efd9751a2175b9c2b4d481. 2018-02-04 11:00:34,873 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] wal.WALSplitter(675): Wrote file=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/41f447d2065af74b219164f0a70d08b3/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-02-04 11:00:34,874 INFO [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(962): Onlined 41f447d2065af74b219164f0a70d08b3; next sequenceid=2 2018-02-04 11:00:34,875 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/7b96ff3277efd9751a2175b9c2b4d481 2018-02-04 11:00:34,878 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table tabletest descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-02-04 11:00:34,881 INFO [PostOpenDeployTasks:41f447d2065af74b219164f0a70d08b3] regionserver.HRegionServer(2166): Post open deploy tasks for tabletest,ooo,1517713232695.41f447d2065af74b219164f0a70d08b3. 2018-02-04 11:00:34,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39063] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=22, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=tabletest, region=41f447d2065af74b219164f0a70d08b3, target=ubuntu,46739,1517713227398; rit=OPENING, location=ubuntu,46739,1517713227398 2018-02-04 11:00:34,887 INFO [PEWorker-13] assignment.RegionStateStore(201): pid=22 updating hbase:meta row=tabletest,ooo,1517713232695.41f447d2065af74b219164f0a70d08b3., regionState=OPEN, openSeqNum=2, regionLocation=ubuntu,46739,1517713227398 2018-02-04 11:00:34,888 DEBUG [PostOpenDeployTasks:41f447d2065af74b219164f0a70d08b3] regionserver.HRegionServer(2190): Finished post open deploy task for tabletest,ooo,1517713232695.41f447d2065af74b219164f0a70d08b3. 2018-02-04 11:00:34,888 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] handler.OpenRegionHandler(128): Opened tabletest,ooo,1517713232695.41f447d2065af74b219164f0a70d08b3. on ubuntu,46739,1517713227398 2018-02-04 11:00:34,889 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(6969): Opening region: {ENCODED => 7514f60148f3a526baae5c7ce2c78c1b, NAME => 'tabletest,www,1517713232695.7514f60148f3a526baae5c7ce2c78c1b.', STARTKEY => 'www', ENDKEY => 'xxx'} 2018-02-04 11:00:34,889 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table tabletest 7514f60148f3a526baae5c7ce2c78c1b 2018-02-04 11:00:34,889 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(809): Instantiated tabletest,www,1517713232695.7514f60148f3a526baae5c7ce2c78c1b. 2018-02-04 11:00:34,899 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/e4481a648e0b377eb5bf59bfb99dfd88/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-02-04 11:00:34,899 INFO [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(962): Onlined e4481a648e0b377eb5bf59bfb99dfd88; next sequenceid=2 2018-02-04 11:00:34,906 INFO [PEWorker-9] procedure2.ProcedureExecutor(1249): Finished pid=28, ppid=7, state=SUCCESS; AssignProcedure table=tabletest, region=9579a7c2f22113dfe574ba9569101416, target=ubuntu,36383,1517713227466 in 903msec 2018-02-04 11:00:34,906 INFO [PEWorker-4] procedure2.ProcedureExecutor(1249): Finished pid=20, ppid=7, state=SUCCESS; AssignProcedure table=tabletest, region=a19e4bf3b69ad0ea43c87985b14ec9fe, target=ubuntu,36383,1517713227466 in 935msec 2018-02-04 11:00:34,907 INFO [PEWorker-10] procedure2.ProcedureExecutor(1249): Finished pid=21, ppid=7, state=SUCCESS; AssignProcedure table=tabletest, region=7429e32b4c4e0aadde8a8154cbd1f071, target=ubuntu,36383,1517713227466 in 1.0040sec 2018-02-04 11:00:34,911 INFO [PEWorker-13] procedure2.ProcedureExecutor(1249): Finished pid=22, ppid=7, state=SUCCESS; AssignProcedure table=tabletest, region=41f447d2065af74b219164f0a70d08b3, target=ubuntu,46739,1517713227398 in 1.0100sec 2018-02-04 11:00:34,914 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] wal.WALSplitter(675): Wrote file=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/68c9676b485038d63fd2cf8a435420e5/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-02-04 11:00:34,914 INFO [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(962): Onlined 68c9676b485038d63fd2cf8a435420e5; next sequenceid=2 2018-02-04 11:00:34,915 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/cd5c06da9ebf17554a264cbdd6fcd9ef 2018-02-04 11:00:34,917 DEBUG [StoreOpener-879a95042a6098ee953dbfe218826b40-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/879a95042a6098ee953dbfe218826b40/family 2018-02-04 11:00:34,917 DEBUG [StoreOpener-879a95042a6098ee953dbfe218826b40-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/879a95042a6098ee953dbfe218826b40/family 2018-02-04 11:00:34,918 INFO [StoreOpener-879a95042a6098ee953dbfe218826b40-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-02-04 11:00:34,918 INFO [StoreOpener-879a95042a6098ee953dbfe218826b40-1] regionserver.HStore(294): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-02-04 11:00:34,919 INFO [StoreOpener-879a95042a6098ee953dbfe218826b40-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=819720, freeSize=1116962296, maxSize=1117782016, heapSize=819720, minSize=1061892928, minFactor=0.95, multiSize=530946464, multiFactor=0.5, singleSize=265473232, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-02-04 11:00:34,919 INFO [StoreOpener-879a95042a6098ee953dbfe218826b40-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-02-04 11:00:34,921 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] wal.WALSplitter(675): Wrote file=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/7b96ff3277efd9751a2175b9c2b4d481/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-02-04 11:00:34,921 INFO [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(962): Onlined 7b96ff3277efd9751a2175b9c2b4d481; next sequenceid=2 2018-02-04 11:00:34,922 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table tabletest descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-02-04 11:00:34,924 INFO [PostOpenDeployTasks:e4481a648e0b377eb5bf59bfb99dfd88] regionserver.HRegionServer(2166): Post open deploy tasks for tabletest,jjj,1517713232695.e4481a648e0b377eb5bf59bfb99dfd88. 2018-02-04 11:00:34,929 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/e063e6b2c2640eb6eafc8737a71fb5f5/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-02-04 11:00:34,929 INFO [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(962): Onlined e063e6b2c2640eb6eafc8737a71fb5f5; next sequenceid=2 2018-02-04 11:00:34,930 DEBUG [StoreOpener-2a83d01250e1489bfa6bcdc2057bb1b5-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/2a83d01250e1489bfa6bcdc2057bb1b5/family 2018-02-04 11:00:34,930 DEBUG [StoreOpener-2a83d01250e1489bfa6bcdc2057bb1b5-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/2a83d01250e1489bfa6bcdc2057bb1b5/family 2018-02-04 11:00:34,931 INFO [StoreOpener-2a83d01250e1489bfa6bcdc2057bb1b5-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-02-04 11:00:34,931 INFO [StoreOpener-2a83d01250e1489bfa6bcdc2057bb1b5-1] regionserver.HStore(294): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-02-04 11:00:34,931 INFO [StoreOpener-2a83d01250e1489bfa6bcdc2057bb1b5-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=819720, freeSize=1116962296, maxSize=1117782016, heapSize=819720, minSize=1061892928, minFactor=0.95, multiSize=530946464, multiFactor=0.5, singleSize=265473232, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-02-04 11:00:34,932 INFO [StoreOpener-2a83d01250e1489bfa6bcdc2057bb1b5-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-02-04 11:00:34,933 INFO [PostOpenDeployTasks:68c9676b485038d63fd2cf8a435420e5] regionserver.HRegionServer(2166): Post open deploy tasks for tabletest,ddd,1517713232695.68c9676b485038d63fd2cf8a435420e5. 2018-02-04 11:00:34,936 INFO [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(1035): Setting FlushNonSloppyStoresFirstPolicy for the region=tabletest,ggg,1517713232695.879a95042a6098ee953dbfe218826b40. 2018-02-04 11:00:34,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39063] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=17, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=tabletest, region=e4481a648e0b377eb5bf59bfb99dfd88, target=ubuntu,46739,1517713227398; rit=OPENING, location=ubuntu,46739,1517713227398 2018-02-04 11:00:34,941 INFO [PEWorker-1] assignment.RegionStateStore(201): pid=17 updating hbase:meta row=tabletest,jjj,1517713232695.e4481a648e0b377eb5bf59bfb99dfd88., regionState=OPEN, openSeqNum=2, regionLocation=ubuntu,46739,1517713227398 2018-02-04 11:00:34,941 DEBUG [PostOpenDeployTasks:e4481a648e0b377eb5bf59bfb99dfd88] regionserver.HRegionServer(2190): Finished post open deploy task for tabletest,jjj,1517713232695.e4481a648e0b377eb5bf59bfb99dfd88. 2018-02-04 11:00:34,945 INFO [PostOpenDeployTasks:7b96ff3277efd9751a2175b9c2b4d481] regionserver.HRegionServer(2166): Post open deploy tasks for tabletest,fff,1517713232695.7b96ff3277efd9751a2175b9c2b4d481. 2018-02-04 11:00:34,946 INFO [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(1035): Setting FlushNonSloppyStoresFirstPolicy for the region=tabletest,vvv,1517713232695.2a83d01250e1489bfa6bcdc2057bb1b5. 2018-02-04 11:00:34,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=39063] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=11, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=tabletest, region=68c9676b485038d63fd2cf8a435420e5, target=ubuntu,39267,1517713227316; rit=OPENING, location=ubuntu,39267,1517713227316 2018-02-04 11:00:34,951 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/879a95042a6098ee953dbfe218826b40 2018-02-04 11:00:34,951 INFO [PEWorker-5] assignment.RegionStateStore(201): pid=11 updating hbase:meta row=tabletest,ddd,1517713232695.68c9676b485038d63fd2cf8a435420e5., regionState=OPEN, openSeqNum=2, regionLocation=ubuntu,39267,1517713227316 2018-02-04 11:00:34,949 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] handler.OpenRegionHandler(128): Opened tabletest,jjj,1517713232695.e4481a648e0b377eb5bf59bfb99dfd88. on ubuntu,46739,1517713227398 2018-02-04 11:00:34,952 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(6969): Opening region: {ENCODED => 1da7d397be67579013b1f635bfbcb064, NAME => 'tabletest,zzz,1517713232695.1da7d397be67579013b1f635bfbcb064.', STARTKEY => 'zzz', ENDKEY => ''} 2018-02-04 11:00:34,952 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table tabletest 1da7d397be67579013b1f635bfbcb064 2018-02-04 11:00:34,952 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(809): Instantiated tabletest,zzz,1517713232695.1da7d397be67579013b1f635bfbcb064. 2018-02-04 11:00:34,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39063] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=13, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=tabletest, region=7b96ff3277efd9751a2175b9c2b4d481, target=ubuntu,39267,1517713227316; rit=OPENING, location=ubuntu,39267,1517713227316 2018-02-04 11:00:34,951 INFO [PEWorker-1] procedure2.ProcedureExecutor(1249): Finished pid=17, ppid=7, state=SUCCESS; AssignProcedure table=tabletest, region=e4481a648e0b377eb5bf59bfb99dfd88, target=ubuntu,46739,1517713227398 in 1.0560sec 2018-02-04 11:00:34,954 INFO [PEWorker-1] assignment.RegionStateStore(201): pid=13 updating hbase:meta row=tabletest,fff,1517713232695.7b96ff3277efd9751a2175b9c2b4d481., regionState=OPEN, openSeqNum=2, regionLocation=ubuntu,39267,1517713227316 2018-02-04 11:00:34,955 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/2a83d01250e1489bfa6bcdc2057bb1b5 2018-02-04 11:00:34,953 DEBUG [StoreOpener-756819a3f903733cb342d3913aab52a6-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/756819a3f903733cb342d3913aab52a6/family 2018-02-04 11:00:34,957 DEBUG [StoreOpener-756819a3f903733cb342d3913aab52a6-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/756819a3f903733cb342d3913aab52a6/family 2018-02-04 11:00:34,957 INFO [StoreOpener-756819a3f903733cb342d3913aab52a6-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-02-04 11:00:34,957 INFO [StoreOpener-756819a3f903733cb342d3913aab52a6-1] regionserver.HStore(294): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-02-04 11:00:34,957 INFO [StoreOpener-756819a3f903733cb342d3913aab52a6-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=819720, freeSize=1116962296, maxSize=1117782016, heapSize=819720, minSize=1061892928, minFactor=0.95, multiSize=530946464, multiFactor=0.5, singleSize=265473232, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-02-04 11:00:34,957 INFO [StoreOpener-756819a3f903733cb342d3913aab52a6-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-02-04 11:00:34,958 DEBUG [PostOpenDeployTasks:68c9676b485038d63fd2cf8a435420e5] regionserver.HRegionServer(2190): Finished post open deploy task for tabletest,ddd,1517713232695.68c9676b485038d63fd2cf8a435420e5. 2018-02-04 11:00:34,958 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] handler.OpenRegionHandler(128): Opened tabletest,ddd,1517713232695.68c9676b485038d63fd2cf8a435420e5. on ubuntu,39267,1517713227316 2018-02-04 11:00:34,958 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(6969): Opening region: {ENCODED => 4f8ea7a76697ee6c4cd92147cc852db7, NAME => 'tabletest,,1517713232695.4f8ea7a76697ee6c4cd92147cc852db7.', STARTKEY => '', ENDKEY => 'bbb'} 2018-02-04 11:00:34,958 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table tabletest descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-02-04 11:00:34,959 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table tabletest 4f8ea7a76697ee6c4cd92147cc852db7 2018-02-04 11:00:34,959 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(809): Instantiated tabletest,,1517713232695.4f8ea7a76697ee6c4cd92147cc852db7. 2018-02-04 11:00:34,959 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] wal.WALSplitter(675): Wrote file=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/cd5c06da9ebf17554a264cbdd6fcd9ef/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-02-04 11:00:34,960 INFO [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(962): Onlined cd5c06da9ebf17554a264cbdd6fcd9ef; next sequenceid=2 2018-02-04 11:00:34,961 DEBUG [PostOpenDeployTasks:7b96ff3277efd9751a2175b9c2b4d481] regionserver.HRegionServer(2190): Finished post open deploy task for tabletest,fff,1517713232695.7b96ff3277efd9751a2175b9c2b4d481. 2018-02-04 11:00:34,961 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] handler.OpenRegionHandler(128): Opened tabletest,fff,1517713232695.7b96ff3277efd9751a2175b9c2b4d481. on ubuntu,39267,1517713227316 2018-02-04 11:00:34,962 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(6969): Opening region: {ENCODED => 14343105008c3cdd39a78990296fbe94, NAME => 'tabletest,eee,1517713232695.14343105008c3cdd39a78990296fbe94.', STARTKEY => 'eee', ENDKEY => 'fff'} 2018-02-04 11:00:34,962 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table tabletest 14343105008c3cdd39a78990296fbe94 2018-02-04 11:00:34,962 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(809): Instantiated tabletest,eee,1517713232695.14343105008c3cdd39a78990296fbe94. 2018-02-04 11:00:34,968 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table tabletest descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-02-04 11:00:34,968 INFO [PostOpenDeployTasks:e063e6b2c2640eb6eafc8737a71fb5f5] regionserver.HRegionServer(2166): Post open deploy tasks for tabletest,sss,1517713232695.e063e6b2c2640eb6eafc8737a71fb5f5. 2018-02-04 11:00:34,972 INFO [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(1035): Setting FlushNonSloppyStoresFirstPolicy for the region=tabletest,bbb,1517713232695.756819a3f903733cb342d3913aab52a6. 2018-02-04 11:00:34,972 DEBUG [StoreOpener-7514f60148f3a526baae5c7ce2c78c1b-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/7514f60148f3a526baae5c7ce2c78c1b/family 2018-02-04 11:00:34,972 DEBUG [StoreOpener-7514f60148f3a526baae5c7ce2c78c1b-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/7514f60148f3a526baae5c7ce2c78c1b/family 2018-02-04 11:00:34,973 INFO [StoreOpener-7514f60148f3a526baae5c7ce2c78c1b-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-02-04 11:00:34,973 INFO [StoreOpener-7514f60148f3a526baae5c7ce2c78c1b-1] regionserver.HStore(294): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-02-04 11:00:34,973 INFO [StoreOpener-7514f60148f3a526baae5c7ce2c78c1b-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=819720, freeSize=1116962296, maxSize=1117782016, heapSize=819720, minSize=1061892928, minFactor=0.95, multiSize=530946464, multiFactor=0.5, singleSize=265473232, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-02-04 11:00:34,974 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/756819a3f903733cb342d3913aab52a6 2018-02-04 11:00:34,974 INFO [StoreOpener-7514f60148f3a526baae5c7ce2c78c1b-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-02-04 11:00:34,977 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table tabletest descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-02-04 11:00:34,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39063] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=26, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=tabletest, region=e063e6b2c2640eb6eafc8737a71fb5f5, target=ubuntu,39267,1517713227316; rit=OPENING, location=ubuntu,39267,1517713227316 2018-02-04 11:00:34,995 INFO [PEWorker-15] assignment.RegionStateStore(201): pid=26 updating hbase:meta row=tabletest,sss,1517713232695.e063e6b2c2640eb6eafc8737a71fb5f5., regionState=OPEN, openSeqNum=2, regionLocation=ubuntu,39267,1517713227316 2018-02-04 11:00:34,995 DEBUG [PostOpenDeployTasks:e063e6b2c2640eb6eafc8737a71fb5f5] regionserver.HRegionServer(2190): Finished post open deploy task for tabletest,sss,1517713232695.e063e6b2c2640eb6eafc8737a71fb5f5. 2018-02-04 11:00:34,995 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] handler.OpenRegionHandler(128): Opened tabletest,sss,1517713232695.e063e6b2c2640eb6eafc8737a71fb5f5. on ubuntu,39267,1517713227316 2018-02-04 11:00:34,995 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(6969): Opening region: {ENCODED => ad4f7bcab0a803ddae20b2866466d222, NAME => 'tabletest,hhh,1517713232695.ad4f7bcab0a803ddae20b2866466d222.', STARTKEY => 'hhh', ENDKEY => 'iii'} 2018-02-04 11:00:34,996 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table tabletest ad4f7bcab0a803ddae20b2866466d222 2018-02-04 11:00:34,996 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(809): Instantiated tabletest,hhh,1517713232695.ad4f7bcab0a803ddae20b2866466d222. 2018-02-04 11:00:35,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39063] master.MasterRpcServices(1157): Checking to see if procedure is done pid=7 2018-02-04 11:00:35,007 INFO [PostOpenDeployTasks:cd5c06da9ebf17554a264cbdd6fcd9ef] regionserver.HRegionServer(2166): Post open deploy tasks for tabletest,ccc,1517713232695.cd5c06da9ebf17554a264cbdd6fcd9ef. 2018-02-04 11:00:35,015 DEBUG [StoreOpener-14343105008c3cdd39a78990296fbe94-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/14343105008c3cdd39a78990296fbe94/family 2018-02-04 11:00:35,015 DEBUG [StoreOpener-14343105008c3cdd39a78990296fbe94-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/14343105008c3cdd39a78990296fbe94/family 2018-02-04 11:00:35,016 INFO [StoreOpener-14343105008c3cdd39a78990296fbe94-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-02-04 11:00:35,017 INFO [StoreOpener-14343105008c3cdd39a78990296fbe94-1] regionserver.HStore(294): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-02-04 11:00:35,017 INFO [StoreOpener-14343105008c3cdd39a78990296fbe94-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=819720, freeSize=1116962296, maxSize=1117782016, heapSize=819720, minSize=1061892928, minFactor=0.95, multiSize=530946464, multiFactor=0.5, singleSize=265473232, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-02-04 11:00:35,020 INFO [StoreOpener-14343105008c3cdd39a78990296fbe94-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-02-04 11:00:35,020 DEBUG [StoreOpener-4f8ea7a76697ee6c4cd92147cc852db7-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/4f8ea7a76697ee6c4cd92147cc852db7/family 2018-02-04 11:00:35,021 DEBUG [StoreOpener-4f8ea7a76697ee6c4cd92147cc852db7-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/4f8ea7a76697ee6c4cd92147cc852db7/family 2018-02-04 11:00:35,022 INFO [StoreOpener-4f8ea7a76697ee6c4cd92147cc852db7-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-02-04 11:00:35,022 INFO [StoreOpener-4f8ea7a76697ee6c4cd92147cc852db7-1] regionserver.HStore(294): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-02-04 11:00:35,022 INFO [StoreOpener-4f8ea7a76697ee6c4cd92147cc852db7-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=819720, freeSize=1116962296, maxSize=1117782016, heapSize=819720, minSize=1061892928, minFactor=0.95, multiSize=530946464, multiFactor=0.5, singleSize=265473232, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-02-04 11:00:35,022 INFO [StoreOpener-4f8ea7a76697ee6c4cd92147cc852db7-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-02-04 11:00:35,025 INFO [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(1035): Setting FlushNonSloppyStoresFirstPolicy for the region=tabletest,eee,1517713232695.14343105008c3cdd39a78990296fbe94. 2018-02-04 11:00:35,025 INFO [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(1035): Setting FlushNonSloppyStoresFirstPolicy for the region=tabletest,,1517713232695.4f8ea7a76697ee6c4cd92147cc852db7. 2018-02-04 11:00:35,031 INFO [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(1035): Setting FlushNonSloppyStoresFirstPolicy for the region=tabletest,www,1517713232695.7514f60148f3a526baae5c7ce2c78c1b. 2018-02-04 11:00:35,034 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/879a95042a6098ee953dbfe218826b40/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-02-04 11:00:35,034 INFO [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(962): Onlined 879a95042a6098ee953dbfe218826b40; next sequenceid=2 2018-02-04 11:00:35,036 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/7514f60148f3a526baae5c7ce2c78c1b 2018-02-04 11:00:35,043 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] wal.WALSplitter(675): Wrote file=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/2a83d01250e1489bfa6bcdc2057bb1b5/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-02-04 11:00:35,065 INFO [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(962): Onlined 2a83d01250e1489bfa6bcdc2057bb1b5; next sequenceid=2 2018-02-04 11:00:35,066 INFO [PEWorker-5] procedure2.ProcedureExecutor(1249): Finished pid=11, ppid=7, state=SUCCESS; AssignProcedure table=tabletest, region=68c9676b485038d63fd2cf8a435420e5, target=ubuntu,39267,1517713227316 in 1.0830sec 2018-02-04 11:00:35,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39063] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=10, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=tabletest, region=cd5c06da9ebf17554a264cbdd6fcd9ef, target=ubuntu,46739,1517713227398; rit=OPENING, location=ubuntu,46739,1517713227398 2018-02-04 11:00:35,068 INFO [PEWorker-16] assignment.RegionStateStore(201): pid=10 updating hbase:meta row=tabletest,ccc,1517713232695.cd5c06da9ebf17554a264cbdd6fcd9ef., regionState=OPEN, openSeqNum=2, regionLocation=ubuntu,46739,1517713227398 2018-02-04 11:00:35,070 DEBUG [PostOpenDeployTasks:cd5c06da9ebf17554a264cbdd6fcd9ef] regionserver.HRegionServer(2190): Finished post open deploy task for tabletest,ccc,1517713232695.cd5c06da9ebf17554a264cbdd6fcd9ef. 2018-02-04 11:00:35,072 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] wal.WALSplitter(675): Wrote file=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/756819a3f903733cb342d3913aab52a6/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-02-04 11:00:35,072 INFO [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(962): Onlined 756819a3f903733cb342d3913aab52a6; next sequenceid=2 2018-02-04 11:00:35,073 INFO [PostOpenDeployTasks:879a95042a6098ee953dbfe218826b40] regionserver.HRegionServer(2166): Post open deploy tasks for tabletest,ggg,1517713232695.879a95042a6098ee953dbfe218826b40. 2018-02-04 11:00:35,074 DEBUG [StoreOpener-ad4f7bcab0a803ddae20b2866466d222-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/ad4f7bcab0a803ddae20b2866466d222/family 2018-02-04 11:00:35,074 DEBUG [StoreOpener-ad4f7bcab0a803ddae20b2866466d222-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/ad4f7bcab0a803ddae20b2866466d222/family 2018-02-04 11:00:35,075 INFO [StoreOpener-ad4f7bcab0a803ddae20b2866466d222-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-02-04 11:00:35,075 INFO [StoreOpener-ad4f7bcab0a803ddae20b2866466d222-1] regionserver.HStore(294): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-02-04 11:00:35,075 INFO [StoreOpener-ad4f7bcab0a803ddae20b2866466d222-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=819720, freeSize=1116962296, maxSize=1117782016, heapSize=819720, minSize=1061892928, minFactor=0.95, multiSize=530946464, multiFactor=0.5, singleSize=265473232, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-02-04 11:00:35,076 INFO [StoreOpener-ad4f7bcab0a803ddae20b2866466d222-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-02-04 11:00:35,077 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/14343105008c3cdd39a78990296fbe94 2018-02-04 11:00:35,079 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/4f8ea7a76697ee6c4cd92147cc852db7 2018-02-04 11:00:35,080 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table tabletest descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-02-04 11:00:35,063 DEBUG [StoreOpener-1da7d397be67579013b1f635bfbcb064-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/1da7d397be67579013b1f635bfbcb064/family 2018-02-04 11:00:35,083 DEBUG [StoreOpener-1da7d397be67579013b1f635bfbcb064-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/1da7d397be67579013b1f635bfbcb064/family 2018-02-04 11:00:35,084 INFO [StoreOpener-1da7d397be67579013b1f635bfbcb064-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-02-04 11:00:35,084 INFO [StoreOpener-1da7d397be67579013b1f635bfbcb064-1] regionserver.HStore(294): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-02-04 11:00:35,085 INFO [StoreOpener-1da7d397be67579013b1f635bfbcb064-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=819720, freeSize=1116962296, maxSize=1117782016, heapSize=819720, minSize=1061892928, minFactor=0.95, multiSize=530946464, multiFactor=0.5, singleSize=265473232, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-02-04 11:00:35,066 INFO [PEWorker-1] procedure2.ProcedureExecutor(1249): Finished pid=13, ppid=7, state=SUCCESS; AssignProcedure table=tabletest, region=7b96ff3277efd9751a2175b9c2b4d481, target=ubuntu,39267,1517713227316 in 1.1510sec 2018-02-04 11:00:35,090 INFO [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(1035): Setting FlushNonSloppyStoresFirstPolicy for the region=tabletest,hhh,1517713232695.ad4f7bcab0a803ddae20b2866466d222. 2018-02-04 11:00:35,093 INFO [StoreOpener-1da7d397be67579013b1f635bfbcb064-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-02-04 11:00:35,099 INFO [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(1035): Setting FlushNonSloppyStoresFirstPolicy for the region=tabletest,zzz,1517713232695.1da7d397be67579013b1f635bfbcb064. 2018-02-04 11:00:35,103 INFO [PostOpenDeployTasks:2a83d01250e1489bfa6bcdc2057bb1b5] regionserver.HRegionServer(2166): Post open deploy tasks for tabletest,vvv,1517713232695.2a83d01250e1489bfa6bcdc2057bb1b5. 2018-02-04 11:00:35,105 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/1da7d397be67579013b1f635bfbcb064 2018-02-04 11:00:35,116 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] handler.OpenRegionHandler(128): Opened tabletest,ccc,1517713232695.cd5c06da9ebf17554a264cbdd6fcd9ef. on ubuntu,46739,1517713227398 2018-02-04 11:00:35,117 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(6969): Opening region: {ENCODED => 85d5a900ae52e80b00b89bb74335621c, NAME => 'tabletest,lll,1517713232695.85d5a900ae52e80b00b89bb74335621c.', STARTKEY => 'lll', ENDKEY => 'mmm'} 2018-02-04 11:00:35,117 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table tabletest 85d5a900ae52e80b00b89bb74335621c 2018-02-04 11:00:35,118 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(809): Instantiated tabletest,lll,1517713232695.85d5a900ae52e80b00b89bb74335621c. 2018-02-04 11:00:35,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39063] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=14, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=tabletest, region=879a95042a6098ee953dbfe218826b40, target=ubuntu,36383,1517713227466; rit=OPENING, location=ubuntu,36383,1517713227466 2018-02-04 11:00:35,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=39063] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=29, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=tabletest, region=2a83d01250e1489bfa6bcdc2057bb1b5, target=ubuntu,36383,1517713227466; rit=OPENING, location=ubuntu,36383,1517713227466 2018-02-04 11:00:35,124 INFO [PEWorker-6] assignment.RegionStateStore(201): pid=14 updating hbase:meta row=tabletest,ggg,1517713232695.879a95042a6098ee953dbfe218826b40., regionState=OPEN, openSeqNum=2, regionLocation=ubuntu,36383,1517713227466 2018-02-04 11:00:35,125 DEBUG [PostOpenDeployTasks:879a95042a6098ee953dbfe218826b40] regionserver.HRegionServer(2190): Finished post open deploy task for tabletest,ggg,1517713232695.879a95042a6098ee953dbfe218826b40. 2018-02-04 11:00:35,126 INFO [PEWorker-3] assignment.RegionStateStore(201): pid=29 updating hbase:meta row=tabletest,vvv,1517713232695.2a83d01250e1489bfa6bcdc2057bb1b5., regionState=OPEN, openSeqNum=2, regionLocation=ubuntu,36383,1517713227466 2018-02-04 11:00:35,131 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table tabletest descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-02-04 11:00:35,132 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/ad4f7bcab0a803ddae20b2866466d222 2018-02-04 11:00:35,134 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table tabletest descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-02-04 11:00:35,139 INFO [PostOpenDeployTasks:756819a3f903733cb342d3913aab52a6] regionserver.HRegionServer(2166): Post open deploy tasks for tabletest,bbb,1517713232695.756819a3f903733cb342d3913aab52a6. 2018-02-04 11:00:35,140 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] handler.OpenRegionHandler(128): Opened tabletest,ggg,1517713232695.879a95042a6098ee953dbfe218826b40. on ubuntu,36383,1517713227466 2018-02-04 11:00:35,140 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(6969): Opening region: {ENCODED => 6a7b42fe3d02ea7aca1d05fa666e87ac, NAME => 'tabletest,qqq,1517713232695.6a7b42fe3d02ea7aca1d05fa666e87ac.', STARTKEY => 'qqq', ENDKEY => 'rrr'} 2018-02-04 11:00:35,140 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table tabletest 6a7b42fe3d02ea7aca1d05fa666e87ac 2018-02-04 11:00:35,140 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(809): Instantiated tabletest,qqq,1517713232695.6a7b42fe3d02ea7aca1d05fa666e87ac. 2018-02-04 11:00:35,142 DEBUG [PostOpenDeployTasks:2a83d01250e1489bfa6bcdc2057bb1b5] regionserver.HRegionServer(2190): Finished post open deploy task for tabletest,vvv,1517713232695.2a83d01250e1489bfa6bcdc2057bb1b5. 2018-02-04 11:00:35,143 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] handler.OpenRegionHandler(128): Opened tabletest,vvv,1517713232695.2a83d01250e1489bfa6bcdc2057bb1b5. on ubuntu,36383,1517713227466 2018-02-04 11:00:35,143 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(6969): Opening region: {ENCODED => 12bc4be2348528b0bd4575ae03200fc3, NAME => 'tabletest,ppp,1517713232695.12bc4be2348528b0bd4575ae03200fc3.', STARTKEY => 'ppp', ENDKEY => 'qqq'} 2018-02-04 11:00:35,156 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table tabletest 12bc4be2348528b0bd4575ae03200fc3 2018-02-04 11:00:35,156 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(809): Instantiated tabletest,ppp,1517713232695.12bc4be2348528b0bd4575ae03200fc3. 2018-02-04 11:00:35,157 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] wal.WALSplitter(675): Wrote file=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/7514f60148f3a526baae5c7ce2c78c1b/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-02-04 11:00:35,157 INFO [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(962): Onlined 7514f60148f3a526baae5c7ce2c78c1b; next sequenceid=2 2018-02-04 11:00:35,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=39063] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=9, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=tabletest, region=756819a3f903733cb342d3913aab52a6, target=ubuntu,36383,1517713227466; rit=OPENING, location=ubuntu,36383,1517713227466 2018-02-04 11:00:35,159 INFO [PEWorker-3] procedure2.ProcedureExecutor(1249): Finished pid=29, ppid=7, state=SUCCESS; AssignProcedure table=tabletest, region=2a83d01250e1489bfa6bcdc2057bb1b5, target=ubuntu,36383,1517713227466 in 1.2610sec 2018-02-04 11:00:35,159 INFO [PEWorker-16] procedure2.ProcedureExecutor(1249): Finished pid=10, ppid=7, state=SUCCESS; AssignProcedure table=tabletest, region=cd5c06da9ebf17554a264cbdd6fcd9ef, target=ubuntu,46739,1517713227398 in 1.2070sec 2018-02-04 11:00:35,158 INFO [PEWorker-15] procedure2.ProcedureExecutor(1249): Finished pid=26, ppid=7, state=SUCCESS; AssignProcedure table=tabletest, region=e063e6b2c2640eb6eafc8737a71fb5f5, target=ubuntu,39267,1517713227316 in 1.1840sec 2018-02-04 11:00:35,174 DEBUG [PostOpenDeployTasks:756819a3f903733cb342d3913aab52a6] regionserver.HRegionServer(2190): Finished post open deploy task for tabletest,bbb,1517713232695.756819a3f903733cb342d3913aab52a6. 2018-02-04 11:00:35,178 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table tabletest descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-02-04 11:00:35,198 INFO [PEWorker-3] assignment.RegionStateStore(201): pid=9 updating hbase:meta row=tabletest,bbb,1517713232695.756819a3f903733cb342d3913aab52a6., regionState=OPEN, openSeqNum=2, regionLocation=ubuntu,36383,1517713227466 2018-02-04 11:00:35,199 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] handler.OpenRegionHandler(128): Opened tabletest,bbb,1517713232695.756819a3f903733cb342d3913aab52a6. on ubuntu,36383,1517713227466 2018-02-04 11:00:35,203 INFO [PostOpenDeployTasks:7514f60148f3a526baae5c7ce2c78c1b] regionserver.HRegionServer(2166): Post open deploy tasks for tabletest,www,1517713232695.7514f60148f3a526baae5c7ce2c78c1b. 2018-02-04 11:00:35,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=39063] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=30, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=tabletest, region=7514f60148f3a526baae5c7ce2c78c1b, target=ubuntu,46739,1517713227398; rit=OPENING, location=ubuntu,46739,1517713227398 2018-02-04 11:00:35,208 DEBUG [PostOpenDeployTasks:7514f60148f3a526baae5c7ce2c78c1b] regionserver.HRegionServer(2190): Finished post open deploy task for tabletest,www,1517713232695.7514f60148f3a526baae5c7ce2c78c1b. 2018-02-04 11:00:35,208 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] handler.OpenRegionHandler(128): Opened tabletest,www,1517713232695.7514f60148f3a526baae5c7ce2c78c1b. on ubuntu,46739,1517713227398 2018-02-04 11:00:35,216 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table tabletest descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-02-04 11:00:35,217 DEBUG [StoreOpener-85d5a900ae52e80b00b89bb74335621c-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/85d5a900ae52e80b00b89bb74335621c/family 2018-02-04 11:00:35,217 DEBUG [StoreOpener-85d5a900ae52e80b00b89bb74335621c-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/85d5a900ae52e80b00b89bb74335621c/family 2018-02-04 11:00:35,217 INFO [StoreOpener-85d5a900ae52e80b00b89bb74335621c-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-02-04 11:00:35,217 INFO [StoreOpener-85d5a900ae52e80b00b89bb74335621c-1] regionserver.HStore(294): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-02-04 11:00:35,218 INFO [StoreOpener-85d5a900ae52e80b00b89bb74335621c-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=819720, freeSize=1116962296, maxSize=1117782016, heapSize=819720, minSize=1061892928, minFactor=0.95, multiSize=530946464, multiFactor=0.5, singleSize=265473232, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-02-04 11:00:35,218 INFO [StoreOpener-85d5a900ae52e80b00b89bb74335621c-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-02-04 11:00:35,219 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] wal.WALSplitter(675): Wrote file=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/4f8ea7a76697ee6c4cd92147cc852db7/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-02-04 11:00:35,219 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/ad4f7bcab0a803ddae20b2866466d222/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-02-04 11:00:35,220 INFO [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(962): Onlined ad4f7bcab0a803ddae20b2866466d222; next sequenceid=2 2018-02-04 11:00:35,222 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] wal.WALSplitter(675): Wrote file=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/14343105008c3cdd39a78990296fbe94/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-02-04 11:00:35,222 INFO [RS_OPEN_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(962): Onlined 14343105008c3cdd39a78990296fbe94; next sequenceid=2 2018-02-04 11:00:35,219 INFO [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(962): Onlined 4f8ea7a76697ee6c4cd92147cc852db7; next sequenceid=2 2018-02-04 11:00:35,230 INFO [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(1035): Setting FlushNonSloppyStoresFirstPolicy for the region=tabletest,lll,1517713232695.85d5a900ae52e80b00b89bb74335621c. 2018-02-04 11:00:35,230 DEBUG [StoreOpener-6a7b42fe3d02ea7aca1d05fa666e87ac-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/6a7b42fe3d02ea7aca1d05fa666e87ac/family 2018-02-04 11:00:35,231 DEBUG [StoreOpener-6a7b42fe3d02ea7aca1d05fa666e87ac-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/6a7b42fe3d02ea7aca1d05fa666e87ac/family 2018-02-04 11:00:35,231 INFO [StoreOpener-6a7b42fe3d02ea7aca1d05fa666e87ac-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-02-04 11:00:35,231 INFO [StoreOpener-6a7b42fe3d02ea7aca1d05fa666e87ac-1] regionserver.HStore(294): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-02-04 11:00:35,231 INFO [StoreOpener-6a7b42fe3d02ea7aca1d05fa666e87ac-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=819720, freeSize=1116962296, maxSize=1117782016, heapSize=819720, minSize=1061892928, minFactor=0.95, multiSize=530946464, multiFactor=0.5, singleSize=265473232, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-02-04 11:00:35,231 INFO [StoreOpener-6a7b42fe3d02ea7aca1d05fa666e87ac-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-02-04 11:00:35,236 DEBUG [StoreOpener-12bc4be2348528b0bd4575ae03200fc3-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/12bc4be2348528b0bd4575ae03200fc3/family 2018-02-04 11:00:35,237 DEBUG [StoreOpener-12bc4be2348528b0bd4575ae03200fc3-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/12bc4be2348528b0bd4575ae03200fc3/family 2018-02-04 11:00:35,237 INFO [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(1035): Setting FlushNonSloppyStoresFirstPolicy for the region=tabletest,qqq,1517713232695.6a7b42fe3d02ea7aca1d05fa666e87ac. 2018-02-04 11:00:35,237 INFO [StoreOpener-12bc4be2348528b0bd4575ae03200fc3-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-02-04 11:00:35,237 INFO [StoreOpener-12bc4be2348528b0bd4575ae03200fc3-1] regionserver.HStore(294): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-02-04 11:00:35,237 INFO [StoreOpener-12bc4be2348528b0bd4575ae03200fc3-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=819720, freeSize=1116962296, maxSize=1117782016, heapSize=819720, minSize=1061892928, minFactor=0.95, multiSize=530946464, multiFactor=0.5, singleSize=265473232, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-02-04 11:00:35,238 INFO [StoreOpener-12bc4be2348528b0bd4575ae03200fc3-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-02-04 11:00:35,239 INFO [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(1035): Setting FlushNonSloppyStoresFirstPolicy for the region=tabletest,ppp,1517713232695.12bc4be2348528b0bd4575ae03200fc3. 2018-02-04 11:00:35,244 INFO [PEWorker-7] assignment.RegionStateStore(201): pid=30 updating hbase:meta row=tabletest,www,1517713232695.7514f60148f3a526baae5c7ce2c78c1b., regionState=OPEN, openSeqNum=2, regionLocation=ubuntu,46739,1517713227398 2018-02-04 11:00:35,248 INFO [PostOpenDeployTasks:ad4f7bcab0a803ddae20b2866466d222] regionserver.HRegionServer(2166): Post open deploy tasks for tabletest,hhh,1517713232695.ad4f7bcab0a803ddae20b2866466d222. 2018-02-04 11:00:35,248 INFO [PostOpenDeployTasks:14343105008c3cdd39a78990296fbe94] regionserver.HRegionServer(2166): Post open deploy tasks for tabletest,eee,1517713232695.14343105008c3cdd39a78990296fbe94. 2018-02-04 11:00:35,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=39063] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=15, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=tabletest, region=ad4f7bcab0a803ddae20b2866466d222, target=ubuntu,39267,1517713227316; rit=OPENING, location=ubuntu,39267,1517713227316 2018-02-04 11:00:35,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39063] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=12, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=tabletest, region=14343105008c3cdd39a78990296fbe94, target=ubuntu,39267,1517713227316; rit=OPENING, location=ubuntu,39267,1517713227316 2018-02-04 11:00:35,251 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/12bc4be2348528b0bd4575ae03200fc3 2018-02-04 11:00:35,252 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/6a7b42fe3d02ea7aca1d05fa666e87ac 2018-02-04 11:00:35,252 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/85d5a900ae52e80b00b89bb74335621c 2018-02-04 11:00:35,258 DEBUG [PostOpenDeployTasks:ad4f7bcab0a803ddae20b2866466d222] regionserver.HRegionServer(2190): Finished post open deploy task for tabletest,hhh,1517713232695.ad4f7bcab0a803ddae20b2866466d222. 2018-02-04 11:00:35,258 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] handler.OpenRegionHandler(128): Opened tabletest,hhh,1517713232695.ad4f7bcab0a803ddae20b2866466d222. on ubuntu,39267,1517713227316 2018-02-04 11:00:35,258 DEBUG [PostOpenDeployTasks:14343105008c3cdd39a78990296fbe94] regionserver.HRegionServer(2190): Finished post open deploy task for tabletest,eee,1517713232695.14343105008c3cdd39a78990296fbe94. 2018-02-04 11:00:35,258 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-2] handler.OpenRegionHandler(128): Opened tabletest,eee,1517713232695.14343105008c3cdd39a78990296fbe94. on ubuntu,39267,1517713227316 2018-02-04 11:00:35,259 INFO [PostOpenDeployTasks:4f8ea7a76697ee6c4cd92147cc852db7] regionserver.HRegionServer(2166): Post open deploy tasks for tabletest,,1517713232695.4f8ea7a76697ee6c4cd92147cc852db7. 2018-02-04 11:00:35,261 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table tabletest descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-02-04 11:00:35,262 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table tabletest descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-02-04 11:00:35,265 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table tabletest descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-02-04 11:00:35,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39063] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=tabletest, region=4f8ea7a76697ee6c4cd92147cc852db7, target=ubuntu,39267,1517713227316; rit=OPENING, location=ubuntu,39267,1517713227316 2018-02-04 11:00:35,266 INFO [PEWorker-3] procedure2.ProcedureExecutor(1249): Finished pid=9, ppid=7, state=SUCCESS; AssignProcedure table=tabletest, region=756819a3f903733cb342d3913aab52a6, target=ubuntu,36383,1517713227466 in 1.3290sec 2018-02-04 11:00:35,266 INFO [PEWorker-7] procedure2.ProcedureExecutor(1249): Finished pid=30, ppid=7, state=SUCCESS; AssignProcedure table=tabletest, region=7514f60148f3a526baae5c7ce2c78c1b, target=ubuntu,46739,1517713227398 in 1.3650sec 2018-02-04 11:00:35,266 INFO [PEWorker-6] procedure2.ProcedureExecutor(1249): Finished pid=14, ppid=7, state=SUCCESS; AssignProcedure table=tabletest, region=879a95042a6098ee953dbfe218826b40, target=ubuntu,36383,1517713227466 in 1.2670sec 2018-02-04 11:00:35,281 INFO [PEWorker-11] assignment.RegionStateStore(201): pid=12 updating hbase:meta row=tabletest,eee,1517713232695.14343105008c3cdd39a78990296fbe94., regionState=OPEN, openSeqNum=2, regionLocation=ubuntu,39267,1517713227316 2018-02-04 11:00:35,281 INFO [PEWorker-8] assignment.RegionStateStore(201): pid=15 updating hbase:meta row=tabletest,hhh,1517713232695.ad4f7bcab0a803ddae20b2866466d222., regionState=OPEN, openSeqNum=2, regionLocation=ubuntu,39267,1517713227316 2018-02-04 11:00:35,282 INFO [PEWorker-12] assignment.RegionStateStore(201): pid=8 updating hbase:meta row=tabletest,,1517713232695.4f8ea7a76697ee6c4cd92147cc852db7., regionState=OPEN, openSeqNum=2, regionLocation=ubuntu,39267,1517713227316 2018-02-04 11:00:35,282 DEBUG [PostOpenDeployTasks:4f8ea7a76697ee6c4cd92147cc852db7] regionserver.HRegionServer(2190): Finished post open deploy task for tabletest,,1517713232695.4f8ea7a76697ee6c4cd92147cc852db7. 2018-02-04 11:00:35,295 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/1da7d397be67579013b1f635bfbcb064/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-02-04 11:00:35,295 INFO [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(962): Onlined 1da7d397be67579013b1f635bfbcb064; next sequenceid=2 2018-02-04 11:00:35,300 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] handler.OpenRegionHandler(128): Opened tabletest,,1517713232695.4f8ea7a76697ee6c4cd92147cc852db7. on ubuntu,39267,1517713227316 2018-02-04 11:00:35,301 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] wal.WALSplitter(675): Wrote file=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/85d5a900ae52e80b00b89bb74335621c/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-02-04 11:00:35,301 INFO [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(962): Onlined 85d5a900ae52e80b00b89bb74335621c; next sequenceid=2 2018-02-04 11:00:35,313 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] wal.WALSplitter(675): Wrote file=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/12bc4be2348528b0bd4575ae03200fc3/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-02-04 11:00:35,313 INFO [RS_OPEN_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(962): Onlined 12bc4be2348528b0bd4575ae03200fc3; next sequenceid=2 2018-02-04 11:00:35,316 INFO [PEWorker-11] procedure2.ProcedureExecutor(1249): Finished pid=12, ppid=7, state=SUCCESS; AssignProcedure table=tabletest, region=14343105008c3cdd39a78990296fbe94, target=ubuntu,39267,1517713227316 in 1.4010sec 2018-02-04 11:00:35,316 INFO [PostOpenDeployTasks:1da7d397be67579013b1f635bfbcb064] regionserver.HRegionServer(2166): Post open deploy tasks for tabletest,zzz,1517713232695.1da7d397be67579013b1f635bfbcb064. 2018-02-04 11:00:35,316 INFO [PEWorker-12] procedure2.ProcedureExecutor(1249): Finished pid=8, ppid=7, state=SUCCESS; AssignProcedure table=tabletest, region=4f8ea7a76697ee6c4cd92147cc852db7, target=ubuntu,39267,1517713227316 in 1.4170sec 2018-02-04 11:00:35,317 INFO [PEWorker-8] procedure2.ProcedureExecutor(1249): Finished pid=15, ppid=7, state=SUCCESS; AssignProcedure table=tabletest, region=ad4f7bcab0a803ddae20b2866466d222, target=ubuntu,39267,1517713227316 in 1.4200sec 2018-02-04 11:00:35,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39063] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=33, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=tabletest, region=1da7d397be67579013b1f635bfbcb064, target=ubuntu,46739,1517713227398; rit=OPENING, location=ubuntu,46739,1517713227398 2018-02-04 11:00:35,320 DEBUG [PostOpenDeployTasks:1da7d397be67579013b1f635bfbcb064] regionserver.HRegionServer(2190): Finished post open deploy task for tabletest,zzz,1517713232695.1da7d397be67579013b1f635bfbcb064. 2018-02-04 11:00:35,322 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] handler.OpenRegionHandler(128): Opened tabletest,zzz,1517713232695.1da7d397be67579013b1f635bfbcb064. on ubuntu,46739,1517713227398 2018-02-04 11:00:35,326 INFO [PEWorker-9] assignment.RegionStateStore(201): pid=33 updating hbase:meta row=tabletest,zzz,1517713232695.1da7d397be67579013b1f635bfbcb064., regionState=OPEN, openSeqNum=2, regionLocation=ubuntu,46739,1517713227398 2018-02-04 11:00:35,326 INFO [PostOpenDeployTasks:12bc4be2348528b0bd4575ae03200fc3] regionserver.HRegionServer(2166): Post open deploy tasks for tabletest,ppp,1517713232695.12bc4be2348528b0bd4575ae03200fc3. 2018-02-04 11:00:35,328 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/6a7b42fe3d02ea7aca1d05fa666e87ac/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-02-04 11:00:35,328 INFO [RS_OPEN_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(962): Onlined 6a7b42fe3d02ea7aca1d05fa666e87ac; next sequenceid=2 2018-02-04 11:00:35,329 INFO [PostOpenDeployTasks:85d5a900ae52e80b00b89bb74335621c] regionserver.HRegionServer(2166): Post open deploy tasks for tabletest,lll,1517713232695.85d5a900ae52e80b00b89bb74335621c. 2018-02-04 11:00:35,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=39063] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=19, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=tabletest, region=85d5a900ae52e80b00b89bb74335621c, target=ubuntu,46739,1517713227398; rit=OPENING, location=ubuntu,46739,1517713227398 2018-02-04 11:00:35,332 DEBUG [PostOpenDeployTasks:85d5a900ae52e80b00b89bb74335621c] regionserver.HRegionServer(2190): Finished post open deploy task for tabletest,lll,1517713232695.85d5a900ae52e80b00b89bb74335621c. 2018-02-04 11:00:35,332 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] handler.OpenRegionHandler(128): Opened tabletest,lll,1517713232695.85d5a900ae52e80b00b89bb74335621c. on ubuntu,46739,1517713227398 2018-02-04 11:00:35,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39063] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=23, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=tabletest, region=12bc4be2348528b0bd4575ae03200fc3, target=ubuntu,36383,1517713227466; rit=OPENING, location=ubuntu,36383,1517713227466 2018-02-04 11:00:35,337 INFO [PostOpenDeployTasks:6a7b42fe3d02ea7aca1d05fa666e87ac] regionserver.HRegionServer(2166): Post open deploy tasks for tabletest,qqq,1517713232695.6a7b42fe3d02ea7aca1d05fa666e87ac. 2018-02-04 11:00:35,337 DEBUG [PostOpenDeployTasks:12bc4be2348528b0bd4575ae03200fc3] regionserver.HRegionServer(2190): Finished post open deploy task for tabletest,ppp,1517713232695.12bc4be2348528b0bd4575ae03200fc3. 2018-02-04 11:00:35,337 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-1] handler.OpenRegionHandler(128): Opened tabletest,ppp,1517713232695.12bc4be2348528b0bd4575ae03200fc3. on ubuntu,36383,1517713227466 2018-02-04 11:00:35,340 INFO [PEWorker-4] assignment.RegionStateStore(201): pid=19 updating hbase:meta row=tabletest,lll,1517713232695.85d5a900ae52e80b00b89bb74335621c., regionState=OPEN, openSeqNum=2, regionLocation=ubuntu,46739,1517713227398 2018-02-04 11:00:35,340 INFO [PEWorker-10] assignment.RegionStateStore(201): pid=23 updating hbase:meta row=tabletest,ppp,1517713232695.12bc4be2348528b0bd4575ae03200fc3., regionState=OPEN, openSeqNum=2, regionLocation=ubuntu,36383,1517713227466 2018-02-04 11:00:35,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39063] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=24, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=tabletest, region=6a7b42fe3d02ea7aca1d05fa666e87ac, target=ubuntu,36383,1517713227466; rit=OPENING, location=ubuntu,36383,1517713227466 2018-02-04 11:00:35,343 DEBUG [PostOpenDeployTasks:6a7b42fe3d02ea7aca1d05fa666e87ac] regionserver.HRegionServer(2190): Finished post open deploy task for tabletest,qqq,1517713232695.6a7b42fe3d02ea7aca1d05fa666e87ac. 2018-02-04 11:00:35,350 INFO [PEWorker-13] assignment.RegionStateStore(201): pid=24 updating hbase:meta row=tabletest,qqq,1517713232695.6a7b42fe3d02ea7aca1d05fa666e87ac., regionState=OPEN, openSeqNum=2, regionLocation=ubuntu,36383,1517713227466 2018-02-04 11:00:35,350 DEBUG [RS_OPEN_REGION-regionserver/ubuntu:0-0] handler.OpenRegionHandler(128): Opened tabletest,qqq,1517713232695.6a7b42fe3d02ea7aca1d05fa666e87ac. on ubuntu,36383,1517713227466 2018-02-04 11:00:35,371 INFO [PEWorker-4] procedure2.ProcedureExecutor(1249): Finished pid=19, ppid=7, state=SUCCESS; AssignProcedure table=tabletest, region=85d5a900ae52e80b00b89bb74335621c, target=ubuntu,46739,1517713227398 in 1.4710sec 2018-02-04 11:00:35,371 INFO [PEWorker-9] procedure2.ProcedureExecutor(1249): Finished pid=33, ppid=7, state=SUCCESS; AssignProcedure table=tabletest, region=1da7d397be67579013b1f635bfbcb064, target=ubuntu,46739,1517713227398 in 1.4520sec 2018-02-04 11:00:35,371 INFO [PEWorker-10] procedure2.ProcedureExecutor(1249): Finished pid=23, ppid=7, state=SUCCESS; AssignProcedure table=tabletest, region=12bc4be2348528b0bd4575ae03200fc3, target=ubuntu,36383,1517713227466 in 1.4710sec 2018-02-04 11:00:35,374 INFO [PEWorker-13] procedure2.ProcedureExecutor(1617): Finished subprocedure(s) of pid=7, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=tabletest; resume parent processing. 2018-02-04 11:00:35,375 INFO [PEWorker-13] procedure2.ProcedureExecutor(1249): Finished pid=24, ppid=7, state=SUCCESS; AssignProcedure table=tabletest, region=6a7b42fe3d02ea7aca1d05fa666e87ac, target=ubuntu,36383,1517713227466 in 1.4750sec 2018-02-04 11:00:35,375 DEBUG [PEWorker-13] hbase.MetaTableAccessor(2146): Put {"totalColumns":1,"row":"tabletest","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1517713235375}]},"ts":1517713235375} 2018-02-04 11:00:35,384 INFO [PEWorker-13] hbase.MetaTableAccessor(1801): Updated table tabletest state to ENABLED in META 2018-02-04 11:00:35,394 INFO [PEWorker-13] procedure2.ProcedureExecutor(1249): Finished pid=7, state=SUCCESS; CreateTableProcedure table=tabletest in 2.6820sec 2018-02-04 11:00:36,213 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2018-02-04 11:00:36,411 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2018-02-04 11:00:36,412 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(138): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2018-02-04 11:00:37,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=39063] master.MasterRpcServices(1157): Checking to see if procedure is done pid=7 2018-02-04 11:00:37,016 INFO [Time-limited test] client.HBaseAdmin$TableFuture(3645): Operation: CREATE, Table Name: default:tabletest completed 2018-02-04 11:00:37,016 DEBUG [Time-limited test] hbase.HBaseTestingUtility(3274): Waiting until all regions of table tabletest get assigned. Timeout = 60000ms 2018-02-04 11:00:37,021 INFO [Time-limited test] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2018-02-04 11:00:37,064 INFO [Time-limited test] hbase.HBaseTestingUtility(3326): All regions for table tabletest assigned to meta. Checking AM states. 2018-02-04 11:00:37,065 INFO [Time-limited test] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2018-02-04 11:00:37,065 INFO [Time-limited test] hbase.HBaseTestingUtility(3346): All regions for table tabletest assigned. 2018-02-04 11:00:37,279 DEBUG [hconnection-0x78b0f3e2-shared-pool13-t3] ipc.NettyRpcConnection(254): Connecting to ubuntu/127.0.1.1:36383 2018-02-04 11:00:37,288 DEBUG [hconnection-0x78b0f3e2-shared-pool13-t5] ipc.NettyRpcConnection(254): Connecting to ubuntu/127.0.1.1:39267 2018-02-04 11:00:37,321 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=46739] regionserver.RSRpcServices$1(1127): Large batch operation detected (greater than 5000) (HBASE-18023). Requested Number of Rows: 5625 Client: zhangduo//127.0.0.1 first region in multi=tabletest,ccc,1517713232695.cd5c06da9ebf17554a264cbdd6fcd9ef. 2018-02-04 11:00:37,337 INFO [RS-EventLoopGroup-3-4] ipc.ServerRpcConnection(528): Auth successful for zhangduo (auth:SIMPLE) 2018-02-04 11:00:37,337 INFO [RS-EventLoopGroup-3-4] ipc.ServerRpcConnection(558): Connection from 127.0.0.1 port: 35746 with version info: version: "3.0.0-SNAPSHOT" url: "git://ubuntu/home/zhangduo/hbase/code" revision: "7fd9ea800d3ec8bfbba59879d0c98e0268d0ccde" user: "zhangduo" date: "Sat Feb 3 18:15:39 CST 2018" src_checksum: "e1f78921fcd876d508017ada55edc99a" version_major: 3 version_minor: 0 2018-02-04 11:00:37,345 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(528): Auth successful for zhangduo (auth:SIMPLE) 2018-02-04 11:00:37,346 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(558): Connection from 127.0.0.1 port: 49798 with version info: version: "3.0.0-SNAPSHOT" url: "git://ubuntu/home/zhangduo/hbase/code" revision: "7fd9ea800d3ec8bfbba59879d0c98e0268d0ccde" user: "zhangduo" date: "Sat Feb 3 18:15:39 CST 2018" src_checksum: "e1f78921fcd876d508017ada55edc99a" version_major: 3 version_minor: 0 2018-02-04 11:00:37,354 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=36383] regionserver.RSRpcServices$1(1127): Large batch operation detected (greater than 5000) (HBASE-18023). Requested Number of Rows: 5624 Client: zhangduo//127.0.0.1 first region in multi=tabletest,bbb,1517713232695.756819a3f903733cb342d3913aab52a6. 2018-02-04 11:00:37,362 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39267] regionserver.RSRpcServices$1(1127): Large batch operation detected (greater than 5000) (HBASE-18023). Requested Number of Rows: 6327 Client: zhangduo//127.0.0.1 first region in multi=tabletest,,1517713232695.4f8ea7a76697ee6c4cd92147cc852db7. 2018-02-04 11:00:37,618 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at 'bbb', inclusive 2018-02-04 11:00:37,636 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at 'ccc', inclusive 2018-02-04 11:00:37,655 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at 'ddd', inclusive 2018-02-04 11:00:37,668 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at 'eee', inclusive 2018-02-04 11:00:37,680 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at 'fff', inclusive 2018-02-04 11:00:37,694 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at 'ggg', inclusive 2018-02-04 11:00:37,700 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at 'hhh', inclusive 2018-02-04 11:00:37,707 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at 'iii', inclusive 2018-02-04 11:00:37,715 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at 'jjj', inclusive 2018-02-04 11:00:37,722 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at 'kkk', inclusive 2018-02-04 11:00:37,727 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at 'lll', inclusive 2018-02-04 11:00:37,734 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at 'mmm', inclusive 2018-02-04 11:00:37,743 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at 'nnn', inclusive 2018-02-04 11:00:37,749 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at 'ooo', inclusive 2018-02-04 11:00:37,754 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at 'ppp', inclusive 2018-02-04 11:00:37,765 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at 'qqq', inclusive 2018-02-04 11:00:37,774 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at 'rrr', inclusive 2018-02-04 11:00:37,781 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at 'sss', inclusive 2018-02-04 11:00:37,791 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at 'ttt', inclusive 2018-02-04 11:00:37,797 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at 'uuu', inclusive 2018-02-04 11:00:37,807 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at 'vvv', inclusive 2018-02-04 11:00:37,814 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at 'www', inclusive 2018-02-04 11:00:37,827 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at 'xxx', inclusive 2018-02-04 11:00:37,833 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at 'yyy', inclusive 2018-02-04 11:00:37,841 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at 'zzz', inclusive 2018-02-04 11:00:37,896 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=36383] regionserver.RSRpcServices$1(1127): Large batch operation detected (greater than 5000) (HBASE-18023). Requested Number of Rows: 5624 Client: zhangduo//127.0.0.1 first region in multi=tabletest,bbb,1517713232695.756819a3f903733cb342d3913aab52a6. 2018-02-04 11:00:37,908 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=46739] regionserver.RSRpcServices$1(1127): Large batch operation detected (greater than 5000) (HBASE-18023). Requested Number of Rows: 5625 Client: zhangduo//127.0.0.1 first region in multi=tabletest,ccc,1517713232695.cd5c06da9ebf17554a264cbdd6fcd9ef. 2018-02-04 11:00:37,916 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39267] regionserver.RSRpcServices$1(1127): Large batch operation detected (greater than 5000) (HBASE-18023). Requested Number of Rows: 6327 Client: zhangduo//127.0.0.1 first region in multi=tabletest,,1517713232695.4f8ea7a76697ee6c4cd92147cc852db7. 2018-02-04 11:00:38,034 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=36383] regionserver.RSRpcServices$1(1127): Large batch operation detected (greater than 5000) (HBASE-18023). Requested Number of Rows: 5624 Client: zhangduo//127.0.0.1 first region in multi=tabletest,bbb,1517713232695.756819a3f903733cb342d3913aab52a6. 2018-02-04 11:00:38,042 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=46739] regionserver.RSRpcServices$1(1127): Large batch operation detected (greater than 5000) (HBASE-18023). Requested Number of Rows: 5625 Client: zhangduo//127.0.0.1 first region in multi=tabletest,ccc,1517713232695.cd5c06da9ebf17554a264cbdd6fcd9ef. 2018-02-04 11:00:38,063 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39267] regionserver.RSRpcServices$1(1127): Large batch operation detected (greater than 5000) (HBASE-18023). Requested Number of Rows: 6327 Client: zhangduo//127.0.0.1 first region in multi=tabletest,,1517713232695.4f8ea7a76697ee6c4cd92147cc852db7. 2018-02-04 11:00:38,190 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=36383] regionserver.RSRpcServices$1(1127): Large batch operation detected (greater than 5000) (HBASE-18023). Requested Number of Rows: 5624 Client: zhangduo//127.0.0.1 first region in multi=tabletest,bbb,1517713232695.756819a3f903733cb342d3913aab52a6. 2018-02-04 11:00:38,199 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=46739] regionserver.RSRpcServices$1(1127): Large batch operation detected (greater than 5000) (HBASE-18023). Requested Number of Rows: 5625 Client: zhangduo//127.0.0.1 first region in multi=tabletest,ccc,1517713232695.cd5c06da9ebf17554a264cbdd6fcd9ef. 2018-02-04 11:00:38,218 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39267] regionserver.RSRpcServices$1(1127): Large batch operation detected (greater than 5000) (HBASE-18023). Requested Number of Rows: 6327 Client: zhangduo//127.0.0.1 first region in multi=tabletest,,1517713232695.4f8ea7a76697ee6c4cd92147cc852db7. 2018-02-04 11:00:38,348 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=46739] regionserver.RSRpcServices$1(1127): Large batch operation detected (greater than 5000) (HBASE-18023). Requested Number of Rows: 5625 Client: zhangduo//127.0.0.1 first region in multi=tabletest,ccc,1517713232695.cd5c06da9ebf17554a264cbdd6fcd9ef. 2018-02-04 11:00:38,348 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=36383] regionserver.RSRpcServices$1(1127): Large batch operation detected (greater than 5000) (HBASE-18023). Requested Number of Rows: 5624 Client: zhangduo//127.0.0.1 first region in multi=tabletest,bbb,1517713232695.756819a3f903733cb342d3913aab52a6. 2018-02-04 11:00:38,351 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39267] regionserver.RSRpcServices$1(1127): Large batch operation detected (greater than 5000) (HBASE-18023). Requested Number of Rows: 6327 Client: zhangduo//127.0.0.1 first region in multi=tabletest,,1517713232695.4f8ea7a76697ee6c4cd92147cc852db7. 2018-02-04 11:00:38,456 INFO [Time-limited test-EventThread] hbase.HBaseTestingUtility$2(2699): Monitor ZKW received event=WatchedEvent state:SyncConnected type:None path:null 2018-02-04 11:00:38,557 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(443): regionserver:39267-0x1615ec1f6c10001, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2018-02-04 11:00:38,557 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(511): regionserver:39267-0x1615ec1f6c10001, quorum=localhost:56841, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring 2018-02-04 11:00:38,557 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(443): regionserver:39267-0x1615ec1f6c10001, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2018-02-04 11:00:38,557 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(511): regionserver:39267-0x1615ec1f6c10001, quorum=localhost:56841, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring 2018-02-04 11:00:38,558 INFO [Time-limited test] hbase.HBaseTestingUtility(2715): ZK Closed Session 0x1615ec1f6c10001 2018-02-04 11:00:38,560 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(443): regionserver:46739-0x1615ec1f6c10002, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/ubuntu,39267,1517713227316 2018-02-04 11:00:38,561 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(443): master:39063-0x1615ec1f6c10000, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/ubuntu,39267,1517713227316 2018-02-04 11:00:38,560 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(443): regionserver:36383-0x1615ec1f6c10003, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/ubuntu,39267,1517713227316 2018-02-04 11:00:38,561 INFO [Time-limited test-EventThread] master.RegionServerTracker(122): RegionServer ephemeral node deleted, processing expiration [ubuntu,39267,1517713227316] 2018-02-04 11:00:38,561 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:46739-0x1615ec1f6c10002, quorum=localhost:56841, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ubuntu,36383,1517713227466 2018-02-04 11:00:38,561 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:36383-0x1615ec1f6c10003, quorum=localhost:56841, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ubuntu,36383,1517713227466 2018-02-04 11:00:38,562 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:46739-0x1615ec1f6c10002, quorum=localhost:56841, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ubuntu,46739,1517713227398 2018-02-04 11:00:38,562 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:36383-0x1615ec1f6c10003, quorum=localhost:56841, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ubuntu,46739,1517713227398 2018-02-04 11:00:38,565 INFO [Time-limited test-EventThread] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(122): /hbase/rs/ubuntu,39267,1517713227316 znode expired, triggering replicatorRemoved event 2018-02-04 11:00:38,565 INFO [Time-limited test-EventThread] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(122): /hbase/rs/ubuntu,39267,1517713227316 znode expired, triggering replicatorRemoved event 2018-02-04 11:00:38,566 INFO [Time-limited test-EventThread] master.ServerManager(595): Processing expiration of ubuntu,39267,1517713227316 on ubuntu,39063,1517713226282 2018-02-04 11:00:38,568 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(443): regionserver:46739-0x1615ec1f6c10002, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-02-04 11:00:38,568 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(443): regionserver:36383-0x1615ec1f6c10003, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-02-04 11:00:38,569 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:46739-0x1615ec1f6c10002, quorum=localhost:56841, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ubuntu,36383,1517713227466 2018-02-04 11:00:38,569 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:46739-0x1615ec1f6c10002, quorum=localhost:56841, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ubuntu,46739,1517713227398 2018-02-04 11:00:38,571 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:36383-0x1615ec1f6c10003, quorum=localhost:56841, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ubuntu,36383,1517713227466 2018-02-04 11:00:38,573 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:36383-0x1615ec1f6c10003, quorum=localhost:56841, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ubuntu,46739,1517713227398 2018-02-04 11:00:38,578 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at 'bbb', inclusive 2018-02-04 11:00:38,586 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at 'ccc', inclusive 2018-02-04 11:00:38,601 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at 'ddd', inclusive 2018-02-04 11:00:38,612 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at 'eee', inclusive 2018-02-04 11:00:38,621 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at 'fff', inclusive 2018-02-04 11:00:38,631 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at 'ggg', inclusive 2018-02-04 11:00:38,641 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at 'hhh', inclusive 2018-02-04 11:00:38,652 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at 'iii', inclusive 2018-02-04 11:00:38,666 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at 'jjj', inclusive 2018-02-04 11:00:38,683 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at 'kkk', inclusive 2018-02-04 11:00:38,685 DEBUG [Time-limited test-EventThread] procedure2.ProcedureExecutor(868): Stored pid=34, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure server=ubuntu,39267,1517713227316, splitWal=true, meta=false 2018-02-04 11:00:38,685 DEBUG [Time-limited test-EventThread] assignment.AssignmentManager(1377): Added=ubuntu,39267,1517713227316 to dead servers, submitted shutdown handler to be executed meta=false 2018-02-04 11:00:38,685 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(443): master:39063-0x1615ec1f6c10000, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-02-04 11:00:38,687 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:39063-0x1615ec1f6c10000, quorum=localhost:56841, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ubuntu,36383,1517713227466 2018-02-04 11:00:38,687 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:39063-0x1615ec1f6c10000, quorum=localhost:56841, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ubuntu,46739,1517713227398 2018-02-04 11:00:38,697 INFO [PEWorker-5] procedure.ServerCrashProcedure(118): Start pid=34, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure server=ubuntu,39267,1517713227316, splitWal=true, meta=false 2018-02-04 11:00:38,705 DEBUG [PEWorker-14] procedure.ServerCrashProcedure(218): Splitting WALs pid=34, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS; ServerCrashProcedure server=ubuntu,39267,1517713227316, splitWal=true, meta=false 2018-02-04 11:00:38,705 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at 'lll', inclusive 2018-02-04 11:00:38,711 DEBUG [PEWorker-14] master.MasterWalManager(257): Renamed region directory: hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/WALs/ubuntu,39267,1517713227316-splitting 2018-02-04 11:00:38,711 INFO [PEWorker-14] master.SplitLogManager(461): dead splitlog workers [ubuntu,39267,1517713227316] 2018-02-04 11:00:38,717 INFO [PEWorker-14] master.SplitLogManager(241): Started splitting 1 logs in [hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/WALs/ubuntu,39267,1517713227316-splitting] for [ubuntu,39267,1517713227316] 2018-02-04 11:00:38,722 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at 'mmm', inclusive 2018-02-04 11:00:38,733 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(443): regionserver:36383-0x1615ec1f6c10003, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2018-02-04 11:00:38,734 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(443): regionserver:46739-0x1615ec1f6c10002, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2018-02-04 11:00:38,737 DEBUG [Time-limited test-EventThread] coordination.ZKSplitLogManagerCoordination(331): Put up splitlog task at znode /hbase/splitWAL/WALs%2Fubuntu%2C39267%2C1517713227316-splitting%2Fubuntu%252C39267%252C1517713227316.1517713230044 2018-02-04 11:00:38,749 DEBUG [Time-limited test-EventThread] coordination.ZKSplitLogManagerCoordination(363): Task not yet acquired /hbase/splitWAL/WALs%2Fubuntu%2C39267%2C1517713227316-splitting%2Fubuntu%252C39267%252C1517713227316.1517713230044, ver=0 2018-02-04 11:00:38,750 DEBUG [SplitLogWorker-ubuntu:46739] coordination.ZkSplitLogWorkerCoordination(436): Current region server ubuntu,46739,1517713227398 is ready to take more tasks, will get task list and try grab tasks again. 2018-02-04 11:00:38,750 DEBUG [SplitLogWorker-ubuntu:36383] coordination.ZkSplitLogWorkerCoordination(436): Current region server ubuntu,36383,1517713227466 is ready to take more tasks, will get task list and try grab tasks again. 2018-02-04 11:00:38,754 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at 'nnn', inclusive 2018-02-04 11:00:38,765 INFO [SplitLogWorker-ubuntu:36383] coordination.ZkSplitLogWorkerCoordination(258): worker ubuntu,36383,1517713227466 acquired task /hbase/splitWAL/WALs%2Fubuntu%2C39267%2C1517713227316-splitting%2Fubuntu%252C39267%252C1517713227316.1517713230044 2018-02-04 11:00:38,766 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(443): master:39063-0x1615ec1f6c10000, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fubuntu%2C39267%2C1517713227316-splitting%2Fubuntu%252C39267%252C1517713227316.1517713230044 2018-02-04 11:00:38,784 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at 'ooo', inclusive 2018-02-04 11:00:38,784 INFO [Time-limited test-EventThread] coordination.ZKSplitLogManagerCoordination(453): Task /hbase/splitWAL/WALs%2Fubuntu%2C39267%2C1517713227316-splitting%2Fubuntu%252C39267%252C1517713227316.1517713230044 acquired by ubuntu,36383,1517713227466 2018-02-04 11:00:38,800 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at 'ppp', inclusive 2018-02-04 11:00:38,814 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at 'qqq', inclusive 2018-02-04 11:00:38,824 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at 'rrr', inclusive 2018-02-04 11:00:38,826 INFO [RS_LOG_REPLAY_OPS-regionserver/ubuntu:0-0] wal.WALSplitter(251): Splitting WAL=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/WALs/ubuntu,39267,1517713227316-splitting/ubuntu%2C39267%2C1517713227316.1517713230044, length=0 2018-02-04 11:00:38,827 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(443): master:39063-0x1615ec1f6c10000, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fubuntu%2C39267%2C1517713227316-splitting%2Fubuntu%252C39267%252C1517713227316.1517713230044 2018-02-04 11:00:38,828 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(443): regionserver:36383-0x1615ec1f6c10003, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fubuntu%2C39267%2C1517713227316-splitting%2Fubuntu%252C39267%252C1517713227316.1517713230044 2018-02-04 11:00:38,829 WARN [RS_LOG_REPLAY_OPS-regionserver/ubuntu:0-0] wal.WALSplitter(710): File hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/WALs/ubuntu,39267,1517713227316-splitting/ubuntu%2C39267%2C1517713227316.1517713230044 might be still open, length is 0 2018-02-04 11:00:38,830 INFO [RS_LOG_REPLAY_OPS-regionserver/ubuntu:0-0] util.FSHDFSUtils(195): Recover lease on dfs file hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/WALs/ubuntu,39267,1517713227316-splitting/ubuntu%2C39267%2C1517713227316.1517713230044 2018-02-04 11:00:38,833 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at 'sss', inclusive 2018-02-04 11:00:38,836 INFO [IPC Server handler 7 on 44491] blockmanagement.BlockInfoContiguousUnderConstruction(331): BLOCK* blk_1073741832_1008{UCState=UNDER_RECOVERY, truncateBlock=null, primaryNodeIndex=2, replicas=[ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619:NORMAL:127.0.0.1:45015|RBW]]} recovery started, primary=ReplicaUC[[DISK]DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619:NORMAL:127.0.0.1:45015|RBW] 2018-02-04 11:00:38,836 WARN [IPC Server handler 7 on 44491] namenode.FSNamesystem(4102): DIR* NameSystem.internalReleaseLease: File /user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/WALs/ubuntu,39267,1517713227316-splitting/ubuntu%2C39267%2C1517713227316.1517713230044 has not been closed. Lease recovery is in progress. RecoveryId = 1039 for block blk_1073741832_1008{UCState=UNDER_RECOVERY, truncateBlock=null, primaryNodeIndex=2, replicas=[ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619:NORMAL:127.0.0.1:45015|RBW]]} 2018-02-04 11:00:38,841 INFO [RS_LOG_REPLAY_OPS-regionserver/ubuntu:0-0] util.FSHDFSUtils(284): Failed to recover lease, attempt=0 on file=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/WALs/ubuntu,39267,1517713227316-splitting/ubuntu%2C39267%2C1517713227316.1517713230044 after 11ms 2018-02-04 11:00:38,844 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at 'ttt', inclusive 2018-02-04 11:00:38,856 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at 'uuu', inclusive 2018-02-04 11:00:38,880 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at 'vvv', inclusive 2018-02-04 11:00:38,893 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at 'www', inclusive 2018-02-04 11:00:38,902 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at 'xxx', inclusive 2018-02-04 11:00:38,914 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at 'yyy', inclusive 2018-02-04 11:00:38,921 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at 'zzz', inclusive 2018-02-04 11:00:38,930 INFO [Time-limited test] hbase.HBaseTestingUtility(1100): Shutting down minicluster 2018-02-04 11:00:38,931 INFO [Time-limited test] client.ConnectionImplementation(1764): Closing master protocol: MasterService 2018-02-04 11:00:38,937 INFO [Time-limited test] zookeeper.ReadOnlyZKClient(344): Close zookeeper connection 0x4107e73b to localhost:56841 2018-02-04 11:00:38,938 DEBUG [Time-limited test] ipc.AbstractRpcClient(483): Stopping rpc client 2018-02-04 11:00:38,947 DEBUG [Time-limited test] util.JVMClusterUtil(238): Shutting down HBase Cluster 2018-02-04 11:00:38,947 INFO [Time-limited test] master.ServerManager(957): Cluster shutdown requested of master=ubuntu,39063,1517713226282 2018-02-04 11:00:38,960 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(443): master:39063-0x1615ec1f6c10000, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2018-02-04 11:00:38,960 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(443): regionserver:46739-0x1615ec1f6c10002, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2018-02-04 11:00:38,961 INFO [Time-limited test] procedure2.ProcedureExecutor(571): Stopping 2018-02-04 11:00:38,962 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(443): regionserver:36383-0x1615ec1f6c10003, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2018-02-04 11:00:38,964 INFO [Time-limited test] zookeeper.ReadOnlyZKClient(344): Close zookeeper connection 0x7d1e2e0a to localhost:56841 2018-02-04 11:00:38,965 DEBUG [Time-limited test] ipc.AbstractRpcClient(483): Stopping rpc client 2018-02-04 11:00:38,968 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(357): regionserver:36383-0x1615ec1f6c10003, quorum=localhost:56841, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-02-04 11:00:38,968 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(357): master:39063-0x1615ec1f6c10000, quorum=localhost:56841, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-02-04 11:00:38,968 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(357): regionserver:46739-0x1615ec1f6c10002, quorum=localhost:56841, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-02-04 11:00:38,969 INFO [Time-limited test] regionserver.HRegionServer(2127): ***** STOPPING region server 'ubuntu,39267,1517713227316' ***** 2018-02-04 11:00:38,969 INFO [Time-limited test] regionserver.HRegionServer(2141): STOPPED: Shutdown requested 2018-02-04 11:00:38,969 INFO [Time-limited test] regionserver.HRegionServer(2127): ***** STOPPING region server 'ubuntu,46739,1517713227398' ***** 2018-02-04 11:00:38,969 INFO [Time-limited test] regionserver.HRegionServer(2141): STOPPED: Shutdown requested 2018-02-04 11:00:38,969 INFO [RS:0;ubuntu:39267] regionserver.SplitLogWorker(160): Sending interrupt to stop the worker thread 2018-02-04 11:00:38,970 INFO [RS:0;ubuntu:39267] regionserver.HeapMemoryManager(221): Stopping HeapMemoryTuner chore. 2018-02-04 11:00:38,970 INFO [RS:0;ubuntu:39267] flush.RegionServerFlushTableProcedureManager(116): Stopping region server flush procedure manager gracefully. 2018-02-04 11:00:38,969 INFO [Time-limited test] regionserver.HRegionServer(2127): ***** STOPPING region server 'ubuntu,36383,1517713227466' ***** 2018-02-04 11:00:38,969 INFO [RS:1;ubuntu:46739] regionserver.SplitLogWorker(160): Sending interrupt to stop the worker thread 2018-02-04 11:00:38,971 INFO [RS:1;ubuntu:46739] regionserver.HeapMemoryManager(221): Stopping HeapMemoryTuner chore. 2018-02-04 11:00:38,972 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(273): MemStoreFlusher.0 exiting 2018-02-04 11:00:38,972 INFO [Time-limited test] regionserver.HRegionServer(2141): STOPPED: Shutdown requested 2018-02-04 11:00:38,971 INFO [SplitLogWorker-ubuntu:39267] regionserver.SplitLogWorker(142): SplitLogWorker interrupted. Exiting. 2018-02-04 11:00:38,970 INFO [RS:0;ubuntu:39267] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager gracefully. 2018-02-04 11:00:38,970 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(273): MemStoreFlusher.0 exiting 2018-02-04 11:00:38,973 INFO [SplitLogWorker-ubuntu:39267] regionserver.SplitLogWorker(151): SplitLogWorker ubuntu,39267,1517713227316 exiting 2018-02-04 11:00:38,973 INFO [RS:2;ubuntu:36383] regionserver.SplitLogWorker(160): Sending interrupt to stop the worker thread 2018-02-04 11:00:38,972 INFO [SplitLogWorker-ubuntu:46739] regionserver.SplitLogWorker(142): SplitLogWorker interrupted. Exiting. 2018-02-04 11:00:38,974 INFO [SplitLogWorker-ubuntu:46739] regionserver.SplitLogWorker(151): SplitLogWorker ubuntu,46739,1517713227398 exiting 2018-02-04 11:00:38,974 WARN [SplitLogWorker-ubuntu:36383] coordination.ZkSplitLogWorkerCoordination(269): Interrupted while yielding for other region servers java.lang.InterruptedException: sleep interrupted at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.hbase.coordination.ZkSplitLogWorkerCoordination.grabTask(ZkSplitLogWorkerCoordination.java:267) at org.apache.hadoop.hbase.coordination.ZkSplitLogWorkerCoordination.taskLoop(ZkSplitLogWorkerCoordination.java:441) at org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:138) at java.lang.Thread.run(Thread.java:748) 2018-02-04 11:00:38,972 INFO [RS:1;ubuntu:46739] flush.RegionServerFlushTableProcedureManager(116): Stopping region server flush procedure manager gracefully. 2018-02-04 11:00:38,975 INFO [SplitLogWorker-ubuntu:36383] regionserver.SplitLogWorker(151): SplitLogWorker ubuntu,36383,1517713227466 exiting 2018-02-04 11:00:38,974 INFO [RS:2;ubuntu:36383] regionserver.HeapMemoryManager(221): Stopping HeapMemoryTuner chore. 2018-02-04 11:00:38,975 INFO [RS:2;ubuntu:36383] flush.RegionServerFlushTableProcedureManager(116): Stopping region server flush procedure manager gracefully. 2018-02-04 11:00:38,975 INFO [RS:2;ubuntu:36383] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager gracefully. 2018-02-04 11:00:38,975 INFO [RS:1;ubuntu:46739] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager gracefully. 2018-02-04 11:00:38,975 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] handler.CloseRegionHandler(92): Processing close of tabletest,hhh,1517713232695.ad4f7bcab0a803ddae20b2866466d222. 2018-02-04 11:00:38,975 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(273): MemStoreFlusher.0 exiting 2018-02-04 11:00:38,975 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] handler.CloseRegionHandler(92): Processing close of tabletest,xxx,1517713232695.66ce5e96488227599f1cd0e5e39a2982. 2018-02-04 11:00:38,976 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] handler.CloseRegionHandler(92): Processing close of tabletest,ggg,1517713232695.879a95042a6098ee953dbfe218826b40. 2018-02-04 11:00:38,977 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] handler.CloseRegionHandler(92): Processing close of tabletest,mmm,1517713232695.a19e4bf3b69ad0ea43c87985b14ec9fe. 2018-02-04 11:00:38,977 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] handler.CloseRegionHandler(92): Processing close of tabletest,vvv,1517713232695.2a83d01250e1489bfa6bcdc2057bb1b5. 2018-02-04 11:00:38,977 INFO [RS:2;ubuntu:36383] regionserver.HRegionServer(1076): stopping server ubuntu,36383,1517713227466 2018-02-04 11:00:38,977 DEBUG [RS:2;ubuntu:36383] zookeeper.MetaTableLocator(641): Stopping MetaTableLocator 2018-02-04 11:00:38,977 INFO [RS:2;ubuntu:36383] zookeeper.ReadOnlyZKClient(344): Close zookeeper connection 0x1c128423 to localhost:56841 2018-02-04 11:00:38,978 DEBUG [RS:2;ubuntu:36383] ipc.AbstractRpcClient(483): Stopping rpc client 2018-02-04 11:00:38,978 INFO [RS:2;ubuntu:36383] regionserver.HRegionServer(1383): Waiting on 8 regions to close 2018-02-04 11:00:38,980 DEBUG [RS:2;ubuntu:36383] regionserver.HRegionServer(1387): Online Regions={879a95042a6098ee953dbfe218826b40=tabletest,ggg,1517713232695.879a95042a6098ee953dbfe218826b40., a19e4bf3b69ad0ea43c87985b14ec9fe=tabletest,mmm,1517713232695.a19e4bf3b69ad0ea43c87985b14ec9fe., 2a83d01250e1489bfa6bcdc2057bb1b5=tabletest,vvv,1517713232695.2a83d01250e1489bfa6bcdc2057bb1b5., 9579a7c2f22113dfe574ba9569101416=tabletest,uuu,1517713232695.9579a7c2f22113dfe574ba9569101416., 7429e32b4c4e0aadde8a8154cbd1f071=tabletest,nnn,1517713232695.7429e32b4c4e0aadde8a8154cbd1f071., 6a7b42fe3d02ea7aca1d05fa666e87ac=tabletest,qqq,1517713232695.6a7b42fe3d02ea7aca1d05fa666e87ac., 756819a3f903733cb342d3913aab52a6=tabletest,bbb,1517713232695.756819a3f903733cb342d3913aab52a6., 12bc4be2348528b0bd4575ae03200fc3=tabletest,ppp,1517713232695.12bc4be2348528b0bd4575ae03200fc3.} 2018-02-04 11:00:38,981 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(273): MemStoreFlusher.1 exiting 2018-02-04 11:00:38,982 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(273): MemStoreFlusher.1 exiting 2018-02-04 11:00:38,982 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(273): MemStoreFlusher.1 exiting 2018-02-04 11:00:38,982 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(1483): Closing tabletest,hhh,1517713232695.ad4f7bcab0a803ddae20b2866466d222.: disabling compactions & flushes 2018-02-04 11:00:38,982 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(1483): Closing tabletest,ggg,1517713232695.879a95042a6098ee953dbfe218826b40.: disabling compactions & flushes 2018-02-04 11:00:38,983 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(1522): Updates disabled for region tabletest,ggg,1517713232695.879a95042a6098ee953dbfe218826b40. 2018-02-04 11:00:38,983 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(2518): Flushing 1/1 column families, memstore=130.44 KB 2018-02-04 11:00:38,982 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(1483): Closing tabletest,xxx,1517713232695.66ce5e96488227599f1cd0e5e39a2982.: disabling compactions & flushes 2018-02-04 11:00:38,985 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(1522): Updates disabled for region tabletest,xxx,1517713232695.66ce5e96488227599f1cd0e5e39a2982. 2018-02-04 11:00:38,985 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(2518): Flushing 1/1 column families, memstore=130.44 KB 2018-02-04 11:00:38,987 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(1522): Updates disabled for region tabletest,hhh,1517713232695.ad4f7bcab0a803ddae20b2866466d222. 2018-02-04 11:00:38,987 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] handler.CloseRegionHandler(92): Processing close of tabletest,ooo,1517713232695.41f447d2065af74b219164f0a70d08b3. 2018-02-04 11:00:38,987 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(1483): Closing tabletest,mmm,1517713232695.a19e4bf3b69ad0ea43c87985b14ec9fe.: disabling compactions & flushes 2018-02-04 11:00:38,988 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] handler.CloseRegionHandler(92): Processing close of tabletest,kkk,1517713232695.a127c3f989561b0b55d12be9bd499407. 2018-02-04 11:00:38,988 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(2518): Flushing 1/1 column families, memstore=130.44 KB 2018-02-04 11:00:38,991 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(1483): Closing tabletest,kkk,1517713232695.a127c3f989561b0b55d12be9bd499407.: disabling compactions & flushes 2018-02-04 11:00:38,990 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(1483): Closing tabletest,ooo,1517713232695.41f447d2065af74b219164f0a70d08b3.: disabling compactions & flushes 2018-02-04 11:00:38,992 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(1522): Updates disabled for region tabletest,ooo,1517713232695.41f447d2065af74b219164f0a70d08b3. 2018-02-04 11:00:38,993 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(2518): Flushing 1/1 column families, memstore=130.44 KB 2018-02-04 11:00:38,990 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(1483): Closing tabletest,vvv,1517713232695.2a83d01250e1489bfa6bcdc2057bb1b5.: disabling compactions & flushes 2018-02-04 11:00:38,993 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(1522): Updates disabled for region tabletest,vvv,1517713232695.2a83d01250e1489bfa6bcdc2057bb1b5. 2018-02-04 11:00:38,994 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(2518): Flushing 1/1 column families, memstore=130.44 KB 2018-02-04 11:00:38,988 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(1522): Updates disabled for region tabletest,mmm,1517713232695.a19e4bf3b69ad0ea43c87985b14ec9fe. 2018-02-04 11:00:38,988 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] handler.CloseRegionHandler(92): Processing close of tabletest,rrr,1517713232695.1b8fb6f4917edac2a9ec29ba9ebed5ef. 2018-02-04 11:00:38,992 INFO [RS:1;ubuntu:46739] regionserver.HRegionServer(1076): stopping server ubuntu,46739,1517713227398 2018-02-04 11:00:38,995 DEBUG [RS:1;ubuntu:46739] zookeeper.MetaTableLocator(641): Stopping MetaTableLocator 2018-02-04 11:00:38,995 INFO [RS:1;ubuntu:46739] zookeeper.ReadOnlyZKClient(344): Close zookeeper connection 0x702c28fc to localhost:56841 2018-02-04 11:00:38,995 DEBUG [RS:1;ubuntu:46739] ipc.AbstractRpcClient(483): Stopping rpc client 2018-02-04 11:00:38,995 INFO [RS:1;ubuntu:46739] regionserver.CompactSplit(394): Waiting for Split Thread to finish... 2018-02-04 11:00:38,996 INFO [RS:1;ubuntu:46739] regionserver.CompactSplit(394): Waiting for Large Compaction Thread to finish... 2018-02-04 11:00:38,996 INFO [RS:1;ubuntu:46739] regionserver.CompactSplit(394): Waiting for Small Compaction Thread to finish... 2018-02-04 11:00:38,992 INFO [RS:0;ubuntu:39267] regionserver.HRegionServer(1076): stopping server ubuntu,39267,1517713227316 2018-02-04 11:00:38,997 DEBUG [RS:0;ubuntu:39267] zookeeper.MetaTableLocator(641): Stopping MetaTableLocator 2018-02-04 11:00:38,997 INFO [RS:0;ubuntu:39267] zookeeper.ReadOnlyZKClient(344): Close zookeeper connection 0x7f64e850 to localhost:56841 2018-02-04 11:00:38,997 DEBUG [RS:0;ubuntu:39267] ipc.AbstractRpcClient(483): Stopping rpc client 2018-02-04 11:00:38,997 INFO [RS:0;ubuntu:39267] regionserver.HRegionServer(1383): Waiting on 10 regions to close 2018-02-04 11:00:38,992 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(1522): Updates disabled for region tabletest,kkk,1517713232695.a127c3f989561b0b55d12be9bd499407. 2018-02-04 11:00:38,998 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(2518): Flushing 1/1 column families, memstore=130.44 KB 2018-02-04 11:00:38,991 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] handler.CloseRegionHandler(92): Processing close of tabletest,,1517713232695.4f8ea7a76697ee6c4cd92147cc852db7. 2018-02-04 11:00:38,995 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(2518): Flushing 1/1 column families, memstore=130.44 KB 2018-02-04 11:00:39,005 INFO [RS:1;ubuntu:46739] regionserver.HRegionServer(1383): Waiting on 10 regions to close 2018-02-04 11:00:39,006 DEBUG [RS_CLOSE_META-regionserver/ubuntu:0-0] handler.CloseRegionHandler(92): Processing close of hbase:meta,,1.1588230740 2018-02-04 11:00:39,010 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.CompactingMemStore(205): FLUSHING TO DISK: region tabletest,kkk,1517713232695.a127c3f989561b0b55d12be9bd499407.store: family 2018-02-04 11:00:39,012 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(1483): Closing tabletest,rrr,1517713232695.1b8fb6f4917edac2a9ec29ba9ebed5ef.: disabling compactions & flushes 2018-02-04 11:00:39,012 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(1483): Closing tabletest,,1517713232695.4f8ea7a76697ee6c4cd92147cc852db7.: disabling compactions & flushes 2018-02-04 11:00:39,012 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(1522): Updates disabled for region tabletest,rrr,1517713232695.1b8fb6f4917edac2a9ec29ba9ebed5ef. 2018-02-04 11:00:39,012 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(2518): Flushing 1/1 column families, memstore=130.44 KB 2018-02-04 11:00:39,012 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.CompactingMemStore(205): FLUSHING TO DISK: region tabletest,ggg,1517713232695.879a95042a6098ee953dbfe218826b40.store: family 2018-02-04 11:00:39,013 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.CompactionPipeline(128): Swapping pipeline suffix; before=1, new segement=null 2018-02-04 11:00:39,012 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.CompactionPipeline(128): Swapping pipeline suffix; before=1, new segement=null 2018-02-04 11:00:39,012 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(1522): Updates disabled for region tabletest,,1517713232695.4f8ea7a76697ee6c4cd92147cc852db7. 2018-02-04 11:00:39,012 DEBUG [RS_CLOSE_META-regionserver/ubuntu:0-0] regionserver.HRegion(1483): Closing hbase:meta,,1.1588230740: disabling compactions & flushes 2018-02-04 11:00:39,014 DEBUG [RS_CLOSE_META-regionserver/ubuntu:0-0] regionserver.HRegion(1522): Updates disabled for region hbase:meta,,1.1588230740 2018-02-04 11:00:39,014 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.CompactingMemStore(205): FLUSHING TO DISK: region tabletest,mmm,1517713232695.a19e4bf3b69ad0ea43c87985b14ec9fe.store: family 2018-02-04 11:00:39,014 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.CompactionPipeline(128): Swapping pipeline suffix; before=1, new segement=null 2018-02-04 11:00:39,015 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.CompactingMemStore(205): FLUSHING TO DISK: region tabletest,hhh,1517713232695.ad4f7bcab0a803ddae20b2866466d222.store: family 2018-02-04 11:00:39,015 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.CompactionPipeline(128): Swapping pipeline suffix; before=1, new segement=null 2018-02-04 11:00:39,016 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.CompactingMemStore(205): FLUSHING TO DISK: region tabletest,xxx,1517713232695.66ce5e96488227599f1cd0e5e39a2982.store: family 2018-02-04 11:00:39,016 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.CompactionPipeline(128): Swapping pipeline suffix; before=1, new segement=null 2018-02-04 11:00:39,017 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.CompactingMemStore(205): FLUSHING TO DISK: region tabletest,vvv,1517713232695.2a83d01250e1489bfa6bcdc2057bb1b5.store: family 2018-02-04 11:00:39,017 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.CompactionPipeline(128): Swapping pipeline suffix; before=1, new segement=null 2018-02-04 11:00:39,014 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(2518): Flushing 1/1 column families, memstore=130.44 KB 2018-02-04 11:00:39,012 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.CompactingMemStore(205): FLUSHING TO DISK: region tabletest,rrr,1517713232695.1b8fb6f4917edac2a9ec29ba9ebed5ef.store: family 2018-02-04 11:00:39,018 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.CompactionPipeline(128): Swapping pipeline suffix; before=1, new segement=null 2018-02-04 11:00:39,014 INFO [RS_CLOSE_META-regionserver/ubuntu:0-0] regionserver.HRegion(2518): Flushing 5/5 column families, memstore=30.51 KB 2018-02-04 11:00:39,014 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.CompactingMemStore(205): FLUSHING TO DISK: region tabletest,ooo,1517713232695.41f447d2065af74b219164f0a70d08b3.store: family 2018-02-04 11:00:39,021 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.CompactionPipeline(128): Swapping pipeline suffix; before=1, new segement=null 2018-02-04 11:00:39,020 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.CompactingMemStore(205): FLUSHING TO DISK: region tabletest,,1517713232695.4f8ea7a76697ee6c4cd92147cc852db7.store: family 2018-02-04 11:00:39,022 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.CompactionPipeline(128): Swapping pipeline suffix; before=1, new segement=null 2018-02-04 11:00:39,043 INFO [master/ubuntu:0.splitLogManager..Chore.1] master.SplitLogManager$TimeoutMonitor(591): total=1, unassigned=0, tasks={/hbase/splitWAL/WALs%2Fubuntu%2C39267%2C1517713227316-splitting%2Fubuntu%252C39267%252C1517713227316.1517713230044=last_update = 1517713238832 last_version = 2 cur_worker_name = ubuntu,36383,1517713227466 status = in_progress incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0} 2018-02-04 11:00:39,184 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741865_1042{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW]]} size 0 2018-02-04 11:00:39,185 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741865_1042{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW]]} size 0 2018-02-04 11:00:39,185 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741865_1042 size 33261 2018-02-04 11:00:39,195 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741863_1040{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|RBW]]} size 0 2018-02-04 11:00:39,195 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741863_1040{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|RBW]]} size 0 2018-02-04 11:00:39,196 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741863_1040 size 36503 2018-02-04 11:00:39,198 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.DefaultStoreFlusher(85): Flushed, sequenceid=9, memsize=130.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/2a83d01250e1489bfa6bcdc2057bb1b5/.tmp/family/017438bae20041688d32eba34cfd9046 2018-02-04 11:00:39,211 INFO [RS_CLOSE_META-regionserver/ubuntu:0-0] regionserver.DefaultStoreFlusher(85): Flushed, sequenceid=65, memsize=30.3 K, hasBloomFilter=false, into tmp file hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/hbase/meta/1588230740/.tmp/info/64da2d30fce241ed894130d19efa9ae1 2018-02-04 11:00:39,214 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741864_1041{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW]]} size 0 2018-02-04 11:00:39,215 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741864_1041{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW]]} size 0 2018-02-04 11:00:39,219 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741869_1046{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619:NORMAL:127.0.0.1:45015|RBW]]} size 0 2018-02-04 11:00:39,221 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741869_1046 size 33261 2018-02-04 11:00:39,223 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741864_1041 size 33261 2018-02-04 11:00:39,223 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741868_1045{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|RBW]]} size 0 2018-02-04 11:00:39,223 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741869_1046 size 33261 2018-02-04 11:00:39,224 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741868_1045{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|RBW]]} size 0 2018-02-04 11:00:39,225 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741868_1045{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|RBW]]} size 0 2018-02-04 11:00:39,240 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741867_1044{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619:NORMAL:127.0.0.1:45015|RBW]]} size 0 2018-02-04 11:00:39,242 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741867_1044 size 33261 2018-02-04 11:00:39,242 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741867_1044 size 33261 2018-02-04 11:00:39,245 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.DefaultStoreFlusher(85): Flushed, sequenceid=9, memsize=130.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/4f8ea7a76697ee6c4cd92147cc852db7/.tmp/family/5ca3abc02ddc474896adc1ca8d2e3d74 2018-02-04 11:00:39,248 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.DefaultStoreFlusher(85): Flushed, sequenceid=9, memsize=130.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/66ce5e96488227599f1cd0e5e39a2982/.tmp/family/1704972cd57c491bb878684aba389c21 2018-02-04 11:00:39,248 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.DefaultStoreFlusher(85): Flushed, sequenceid=9, memsize=130.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/1b8fb6f4917edac2a9ec29ba9ebed5ef/.tmp/family/79c72172b8364a03868d21f1405b1f69 2018-02-04 11:00:39,253 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.DefaultStoreFlusher(85): Flushed, sequenceid=9, memsize=130.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/a19e4bf3b69ad0ea43c87985b14ec9fe/.tmp/family/1be78502449b4a879bf22e402281cdcb 2018-02-04 11:00:39,288 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741866_1043{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|RBW]]} size 33261 2018-02-04 11:00:39,288 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741871_1048{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|RBW]]} size 33261 2018-02-04 11:00:39,288 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741866_1043 size 33261 2018-02-04 11:00:39,288 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741866_1043 size 33261 2018-02-04 11:00:39,290 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741870_1047{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW]]} size 33261 2018-02-04 11:00:39,290 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741870_1047 size 33261 2018-02-04 11:00:39,290 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741871_1048 size 33261 2018-02-04 11:00:39,290 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741870_1047 size 33261 2018-02-04 11:00:39,290 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741871_1048 size 33261 2018-02-04 11:00:39,322 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741872_1049{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW]]} size 0 2018-02-04 11:00:39,324 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741872_1049{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW]]} size 0 2018-02-04 11:00:39,327 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741872_1049{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW]]} size 0 2018-02-04 11:00:39,330 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.DefaultStoreFlusher(85): Flushed, sequenceid=9, memsize=130.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/879a95042a6098ee953dbfe218826b40/.tmp/family/a14e36f60e0e4be984cfd6618da84fb0 2018-02-04 11:00:39,347 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegionFileSystem(463): Committing store file hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/66ce5e96488227599f1cd0e5e39a2982/.tmp/family/1704972cd57c491bb878684aba389c21 as hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/66ce5e96488227599f1cd0e5e39a2982/family/1704972cd57c491bb878684aba389c21 2018-02-04 11:00:39,351 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegionFileSystem(463): Committing store file hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/1b8fb6f4917edac2a9ec29ba9ebed5ef/.tmp/family/79c72172b8364a03868d21f1405b1f69 as hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/1b8fb6f4917edac2a9ec29ba9ebed5ef/family/79c72172b8364a03868d21f1405b1f69 2018-02-04 11:00:39,351 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegionFileSystem(463): Committing store file hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/a19e4bf3b69ad0ea43c87985b14ec9fe/.tmp/family/1be78502449b4a879bf22e402281cdcb as hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/a19e4bf3b69ad0ea43c87985b14ec9fe/family/1be78502449b4a879bf22e402281cdcb 2018-02-04 11:00:39,356 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegionFileSystem(463): Committing store file hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/4f8ea7a76697ee6c4cd92147cc852db7/.tmp/family/5ca3abc02ddc474896adc1ca8d2e3d74 as hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/4f8ea7a76697ee6c4cd92147cc852db7/family/5ca3abc02ddc474896adc1ca8d2e3d74 2018-02-04 11:00:39,358 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegionFileSystem(463): Committing store file hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/879a95042a6098ee953dbfe218826b40/.tmp/family/a14e36f60e0e4be984cfd6618da84fb0 as hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/879a95042a6098ee953dbfe218826b40/family/a14e36f60e0e4be984cfd6618da84fb0 2018-02-04 11:00:39,366 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegionFileSystem(463): Committing store file hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/2a83d01250e1489bfa6bcdc2057bb1b5/.tmp/family/017438bae20041688d32eba34cfd9046 as hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/2a83d01250e1489bfa6bcdc2057bb1b5/family/017438bae20041688d32eba34cfd9046 2018-02-04 11:00:39,368 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HStore(1030): Added hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/a19e4bf3b69ad0ea43c87985b14ec9fe/family/1be78502449b4a879bf22e402281cdcb, entries=703, sequenceid=9, filesize=32.5 K 2018-02-04 11:00:39,382 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HStore(1030): Added hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/66ce5e96488227599f1cd0e5e39a2982/family/1704972cd57c491bb878684aba389c21, entries=703, sequenceid=9, filesize=32.5 K 2018-02-04 11:00:39,396 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(2714): Finished memstore flush of ~130.44 KB/133570, currentsize=0 B/0 for region tabletest,xxx,1517713232695.66ce5e96488227599f1cd0e5e39a2982. in 411ms, sequenceid=9, compaction requested=false 2018-02-04 11:00:39,397 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HStore(1030): Added hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/1b8fb6f4917edac2a9ec29ba9ebed5ef/family/79c72172b8364a03868d21f1405b1f69, entries=703, sequenceid=9, filesize=32.5 K 2018-02-04 11:00:39,401 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(2714): Finished memstore flush of ~130.44 KB/133570, currentsize=0 B/0 for region tabletest,mmm,1517713232695.a19e4bf3b69ad0ea43c87985b14ec9fe. in 405ms, sequenceid=9, compaction requested=false 2018-02-04 11:00:39,401 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(2714): Finished memstore flush of ~130.44 KB/133570, currentsize=0 B/0 for region tabletest,rrr,1517713232695.1b8fb6f4917edac2a9ec29ba9ebed5ef. in 389ms, sequenceid=9, compaction requested=false 2018-02-04 11:00:39,410 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HStore(1030): Added hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/4f8ea7a76697ee6c4cd92147cc852db7/family/5ca3abc02ddc474896adc1ca8d2e3d74, entries=703, sequenceid=9, filesize=32.5 K 2018-02-04 11:00:39,414 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HStore(1030): Added hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/879a95042a6098ee953dbfe218826b40/family/a14e36f60e0e4be984cfd6618da84fb0, entries=703, sequenceid=9, filesize=32.5 K 2018-02-04 11:00:39,420 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(2714): Finished memstore flush of ~130.44 KB/133570, currentsize=0 B/0 for region tabletest,,1517713232695.4f8ea7a76697ee6c4cd92147cc852db7. in 406ms, sequenceid=9, compaction requested=false 2018-02-04 11:00:39,420 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(2714): Finished memstore flush of ~130.44 KB/133570, currentsize=0 B/0 for region tabletest,ggg,1517713232695.879a95042a6098ee953dbfe218826b40. in 437ms, sequenceid=9, compaction requested=false 2018-02-04 11:00:39,424 INFO [StoreCloserThread-tabletest,xxx,1517713232695.66ce5e96488227599f1cd0e5e39a2982.-1] regionserver.HStore(932): Closed family 2018-02-04 11:00:39,426 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HStore(1030): Added hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/2a83d01250e1489bfa6bcdc2057bb1b5/family/017438bae20041688d32eba34cfd9046, entries=703, sequenceid=9, filesize=32.5 K 2018-02-04 11:00:39,426 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741873_1050{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|FINALIZED]]} size 0 2018-02-04 11:00:39,429 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741873_1050{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|FINALIZED], ReplicaUC[[DISK]DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619:NORMAL:127.0.0.1:45015|FINALIZED]]} size 0 2018-02-04 11:00:39,429 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(2714): Finished memstore flush of ~130.44 KB/133570, currentsize=0 B/0 for region tabletest,vvv,1517713232695.2a83d01250e1489bfa6bcdc2057bb1b5. in 436ms, sequenceid=9, compaction requested=false 2018-02-04 11:00:39,430 INFO [RS_CLOSE_META-regionserver/ubuntu:0-0] regionserver.DefaultStoreFlusher(85): Flushed, sequenceid=65, memsize=176, hasBloomFilter=false, into tmp file hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/hbase/meta/1588230740/.tmp/table/65f24369b9ed4fe5b30662d00e4ca5ad 2018-02-04 11:00:39,430 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741873_1050 size 4830 2018-02-04 11:00:39,441 DEBUG [RS_CLOSE_META-regionserver/ubuntu:0-0] regionserver.HRegionFileSystem(463): Committing store file hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/hbase/meta/1588230740/.tmp/info/64da2d30fce241ed894130d19efa9ae1 as hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/hbase/meta/1588230740/info/64da2d30fce241ed894130d19efa9ae1 2018-02-04 11:00:39,451 INFO [StoreCloserThread-tabletest,rrr,1517713232695.1b8fb6f4917edac2a9ec29ba9ebed5ef.-1] regionserver.HStore(932): Closed family 2018-02-04 11:00:39,452 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/66ce5e96488227599f1cd0e5e39a2982/recovered.edits/12.seqid, newSeqId=12, maxSeqId=2 2018-02-04 11:00:39,458 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] wal.WALSplitter(675): Wrote file=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/1b8fb6f4917edac2a9ec29ba9ebed5ef/recovered.edits/12.seqid, newSeqId=12, maxSeqId=2 2018-02-04 11:00:39,463 INFO [StoreCloserThread-tabletest,vvv,1517713232695.2a83d01250e1489bfa6bcdc2057bb1b5.-1] regionserver.HStore(932): Closed family 2018-02-04 11:00:39,464 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(1635): Closed tabletest,xxx,1517713232695.66ce5e96488227599f1cd0e5e39a2982. 2018-02-04 11:00:39,464 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] handler.CloseRegionHandler(125): Closed tabletest,xxx,1517713232695.66ce5e96488227599f1cd0e5e39a2982. 2018-02-04 11:00:39,464 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] handler.CloseRegionHandler(92): Processing close of tabletest,ttt,1517713232695.31b3e09f2db65dd6057a91f45cb8b163. 2018-02-04 11:00:39,465 INFO [StoreCloserThread-tabletest,,1517713232695.4f8ea7a76697ee6c4cd92147cc852db7.-1] regionserver.HStore(932): Closed family 2018-02-04 11:00:39,466 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(1635): Closed tabletest,rrr,1517713232695.1b8fb6f4917edac2a9ec29ba9ebed5ef. 2018-02-04 11:00:39,466 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] handler.CloseRegionHandler(125): Closed tabletest,rrr,1517713232695.1b8fb6f4917edac2a9ec29ba9ebed5ef. 2018-02-04 11:00:39,466 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] handler.CloseRegionHandler(92): Processing close of tabletest,yyy,1517713232695.20457e81e436062b31548a08737ebba4. 2018-02-04 11:00:39,467 INFO [RS_CLOSE_META-regionserver/ubuntu:0-0] regionserver.HStore(1030): Added hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/hbase/meta/1588230740/info/64da2d30fce241ed894130d19efa9ae1, entries=270, sequenceid=65, filesize=35.6 K 2018-02-04 11:00:39,467 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(1483): Closing tabletest,ttt,1517713232695.31b3e09f2db65dd6057a91f45cb8b163.: disabling compactions & flushes 2018-02-04 11:00:39,467 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(1522): Updates disabled for region tabletest,ttt,1517713232695.31b3e09f2db65dd6057a91f45cb8b163. 2018-02-04 11:00:39,467 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(2518): Flushing 1/1 column families, memstore=130.44 KB 2018-02-04 11:00:39,467 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.CompactingMemStore(205): FLUSHING TO DISK: region tabletest,ttt,1517713232695.31b3e09f2db65dd6057a91f45cb8b163.store: family 2018-02-04 11:00:39,467 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.CompactionPipeline(128): Swapping pipeline suffix; before=1, new segement=null 2018-02-04 11:00:39,467 INFO [StoreCloserThread-tabletest,mmm,1517713232695.a19e4bf3b69ad0ea43c87985b14ec9fe.-1] regionserver.HStore(932): Closed family 2018-02-04 11:00:39,470 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(1483): Closing tabletest,yyy,1517713232695.20457e81e436062b31548a08737ebba4.: disabling compactions & flushes 2018-02-04 11:00:39,470 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(1522): Updates disabled for region tabletest,yyy,1517713232695.20457e81e436062b31548a08737ebba4. 2018-02-04 11:00:39,470 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(2518): Flushing 1/1 column families, memstore=130.44 KB 2018-02-04 11:00:39,470 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.CompactingMemStore(205): FLUSHING TO DISK: region tabletest,yyy,1517713232695.20457e81e436062b31548a08737ebba4.store: family 2018-02-04 11:00:39,470 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.CompactionPipeline(128): Swapping pipeline suffix; before=1, new segement=null 2018-02-04 11:00:39,473 INFO [StoreCloserThread-tabletest,ggg,1517713232695.879a95042a6098ee953dbfe218826b40.-1] regionserver.HStore(932): Closed family 2018-02-04 11:00:39,474 DEBUG [RS_CLOSE_META-regionserver/ubuntu:0-0] regionserver.HRegionFileSystem(463): Committing store file hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/hbase/meta/1588230740/.tmp/table/65f24369b9ed4fe5b30662d00e4ca5ad as hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/hbase/meta/1588230740/table/65f24369b9ed4fe5b30662d00e4ca5ad 2018-02-04 11:00:39,491 INFO [RS_CLOSE_META-regionserver/ubuntu:0-0] regionserver.HStore(1030): Added hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/hbase/meta/1588230740/table/65f24369b9ed4fe5b30662d00e4ca5ad, entries=4, sequenceid=65, filesize=4.7 K 2018-02-04 11:00:39,497 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/879a95042a6098ee953dbfe218826b40/recovered.edits/12.seqid, newSeqId=12, maxSeqId=2 2018-02-04 11:00:39,521 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] wal.WALSplitter(675): Wrote file=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/a19e4bf3b69ad0ea43c87985b14ec9fe/recovered.edits/12.seqid, newSeqId=12, maxSeqId=2 2018-02-04 11:00:39,523 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(1635): Closed tabletest,ggg,1517713232695.879a95042a6098ee953dbfe218826b40. 2018-02-04 11:00:39,523 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] handler.CloseRegionHandler(125): Closed tabletest,ggg,1517713232695.879a95042a6098ee953dbfe218826b40. 2018-02-04 11:00:39,524 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] handler.CloseRegionHandler(92): Processing close of tabletest,uuu,1517713232695.9579a7c2f22113dfe574ba9569101416. 2018-02-04 11:00:39,530 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(1483): Closing tabletest,uuu,1517713232695.9579a7c2f22113dfe574ba9569101416.: disabling compactions & flushes 2018-02-04 11:00:39,530 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(1522): Updates disabled for region tabletest,uuu,1517713232695.9579a7c2f22113dfe574ba9569101416. 2018-02-04 11:00:39,530 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(2518): Flushing 1/1 column families, memstore=130.44 KB 2018-02-04 11:00:39,530 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.CompactingMemStore(205): FLUSHING TO DISK: region tabletest,uuu,1517713232695.9579a7c2f22113dfe574ba9569101416.store: family 2018-02-04 11:00:39,532 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] wal.WALSplitter(675): Wrote file=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/2a83d01250e1489bfa6bcdc2057bb1b5/recovered.edits/12.seqid, newSeqId=12, maxSeqId=2 2018-02-04 11:00:39,536 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(1635): Closed tabletest,mmm,1517713232695.a19e4bf3b69ad0ea43c87985b14ec9fe. 2018-02-04 11:00:39,536 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] wal.WALSplitter(675): Wrote file=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/4f8ea7a76697ee6c4cd92147cc852db7/recovered.edits/12.seqid, newSeqId=12, maxSeqId=2 2018-02-04 11:00:39,536 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] handler.CloseRegionHandler(125): Closed tabletest,mmm,1517713232695.a19e4bf3b69ad0ea43c87985b14ec9fe. 2018-02-04 11:00:39,536 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] handler.CloseRegionHandler(92): Processing close of tabletest,nnn,1517713232695.7429e32b4c4e0aadde8a8154cbd1f071. 2018-02-04 11:00:39,539 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(1483): Closing tabletest,nnn,1517713232695.7429e32b4c4e0aadde8a8154cbd1f071.: disabling compactions & flushes 2018-02-04 11:00:39,542 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(1522): Updates disabled for region tabletest,nnn,1517713232695.7429e32b4c4e0aadde8a8154cbd1f071. 2018-02-04 11:00:39,543 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(2518): Flushing 1/1 column families, memstore=130.44 KB 2018-02-04 11:00:39,543 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.CompactingMemStore(205): FLUSHING TO DISK: region tabletest,nnn,1517713232695.7429e32b4c4e0aadde8a8154cbd1f071.store: family 2018-02-04 11:00:39,544 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.CompactionPipeline(128): Swapping pipeline suffix; before=1, new segement=null 2018-02-04 11:00:39,547 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.CompactionPipeline(128): Swapping pipeline suffix; before=1, new segement=null 2018-02-04 11:00:39,553 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(1635): Closed tabletest,vvv,1517713232695.2a83d01250e1489bfa6bcdc2057bb1b5. 2018-02-04 11:00:39,553 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] handler.CloseRegionHandler(125): Closed tabletest,vvv,1517713232695.2a83d01250e1489bfa6bcdc2057bb1b5. 2018-02-04 11:00:39,553 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] handler.CloseRegionHandler(92): Processing close of tabletest,qqq,1517713232695.6a7b42fe3d02ea7aca1d05fa666e87ac. 2018-02-04 11:00:39,554 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(1483): Closing tabletest,qqq,1517713232695.6a7b42fe3d02ea7aca1d05fa666e87ac.: disabling compactions & flushes 2018-02-04 11:00:39,554 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(1522): Updates disabled for region tabletest,qqq,1517713232695.6a7b42fe3d02ea7aca1d05fa666e87ac. 2018-02-04 11:00:39,554 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(2518): Flushing 1/1 column families, memstore=130.44 KB 2018-02-04 11:00:39,555 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.CompactingMemStore(205): FLUSHING TO DISK: region tabletest,qqq,1517713232695.6a7b42fe3d02ea7aca1d05fa666e87ac.store: family 2018-02-04 11:00:39,555 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.CompactionPipeline(128): Swapping pipeline suffix; before=1, new segement=null 2018-02-04 11:00:39,561 INFO [RS_CLOSE_META-regionserver/ubuntu:0-0] regionserver.HRegion(2714): Finished memstore flush of ~30.51 KB/31238, currentsize=0 B/0 for region hbase:meta,,1.1588230740 in 547ms, sequenceid=65, compaction requested=false 2018-02-04 11:00:39,571 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(1635): Closed tabletest,,1517713232695.4f8ea7a76697ee6c4cd92147cc852db7. 2018-02-04 11:00:39,571 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] handler.CloseRegionHandler(125): Closed tabletest,,1517713232695.4f8ea7a76697ee6c4cd92147cc852db7. 2018-02-04 11:00:39,571 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] handler.CloseRegionHandler(92): Processing close of tabletest,ddd,1517713232695.68c9676b485038d63fd2cf8a435420e5. 2018-02-04 11:00:39,613 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741874_1051{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW]]} size 33261 2018-02-04 11:00:39,613 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741874_1051 size 33261 2018-02-04 11:00:39,614 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741874_1051 size 33261 2018-02-04 11:00:39,628 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(1483): Closing tabletest,ddd,1517713232695.68c9676b485038d63fd2cf8a435420e5.: disabling compactions & flushes 2018-02-04 11:00:39,628 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(1522): Updates disabled for region tabletest,ddd,1517713232695.68c9676b485038d63fd2cf8a435420e5. 2018-02-04 11:00:39,629 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(2518): Flushing 1/1 column families, memstore=130.44 KB 2018-02-04 11:00:39,629 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.CompactingMemStore(205): FLUSHING TO DISK: region tabletest,ddd,1517713232695.68c9676b485038d63fd2cf8a435420e5.store: family 2018-02-04 11:00:39,630 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.CompactionPipeline(128): Swapping pipeline suffix; before=1, new segement=null 2018-02-04 11:00:39,632 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(932): Closed info 2018-02-04 11:00:39,632 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(932): Closed rep_barrier 2018-02-04 11:00:39,632 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(932): Closed rep_meta 2018-02-04 11:00:39,632 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(932): Closed rep_position 2018-02-04 11:00:39,665 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741876_1053{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW]]} size 0 2018-02-04 11:00:39,665 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741876_1053{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619:NORMAL:127.0.0.1:45015|FINALIZED]]} size 0 2018-02-04 11:00:39,674 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(932): Closed table 2018-02-04 11:00:39,695 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.DefaultStoreFlusher(85): Flushed, sequenceid=9, memsize=130.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/ad4f7bcab0a803ddae20b2866466d222/.tmp/family/3dc6cc18a9f94778b998d171f8b3d60a 2018-02-04 11:00:39,695 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.DefaultStoreFlusher(85): Flushed, sequenceid=9, memsize=130.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/a127c3f989561b0b55d12be9bd499407/.tmp/family/8c0e6a8bdba443749e38b34d717c1b62 2018-02-04 11:00:39,696 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.DefaultStoreFlusher(85): Flushed, sequenceid=9, memsize=130.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/41f447d2065af74b219164f0a70d08b3/.tmp/family/3b7e8219c4394dc1afc72aa7a7e2bc33 2018-02-04 11:00:39,703 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_426488853_13 at /127.0.0.1:38956 [Receiving block BP-35161775-127.0.1.1-1517713219102:blk_1073741832_1008]] datanode.DataXceiver(280): 127.0.0.1:46483:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:38956 dst: /127.0.0.1:46483 java.io.InterruptedIOException: Interrupted while waiting for IO on channel java.nio.channels.SocketChannel[connected local=/127.0.0.1:46483 remote=/127.0.0.1:38956]. 5000 millis timeout left. at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:342) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at java.io.DataInputStream.read(DataInputStream.java:149) at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:200) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:503) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:903) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:805) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:253) at java.lang.Thread.run(Thread.java:748) 2018-02-04 11:00:39,711 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_426488853_13 at /127.0.0.1:33704 [Receiving block BP-35161775-127.0.1.1-1517713219102:blk_1073741832_1008]] datanode.DataXceiver(280): 127.0.0.1:45015:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:33704 dst: /127.0.0.1:45015 java.io.IOException: Premature EOF from inputStream at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:202) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:503) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:903) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:805) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:253) at java.lang.Thread.run(Thread.java:748) 2018-02-04 11:00:39,711 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_426488853_13 at /127.0.0.1:42294 [Receiving block BP-35161775-127.0.1.1-1517713219102:blk_1073741832_1008]] datanode.DataXceiver(280): 127.0.0.1:33785:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:42294 dst: /127.0.0.1:33785 java.io.IOException: Premature EOF from inputStream at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:202) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:503) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:903) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:805) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:253) at java.lang.Thread.run(Thread.java:748) 2018-02-04 11:00:39,708 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741878_1055{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|FINALIZED]]} size 0 2018-02-04 11:00:39,713 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741878_1055{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|FINALIZED]]} size 0 2018-02-04 11:00:39,735 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741875_1052{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW]]} size 0 2018-02-04 11:00:39,741 DEBUG [RS_CLOSE_META-regionserver/ubuntu:0-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/hbase/meta/1588230740/recovered.edits/68.seqid, newSeqId=68, maxSeqId=3 2018-02-04 11:00:39,746 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741875_1052{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW]]} size 0 2018-02-04 11:00:39,747 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegionFileSystem(463): Committing store file hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/a127c3f989561b0b55d12be9bd499407/.tmp/family/8c0e6a8bdba443749e38b34d717c1b62 as hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/a127c3f989561b0b55d12be9bd499407/family/8c0e6a8bdba443749e38b34d717c1b62 2018-02-04 11:00:39,748 DEBUG [RS_CLOSE_META-regionserver/ubuntu:0-0] coprocessor.CoprocessorHost(288): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2018-02-04 11:00:39,748 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegionFileSystem(463): Committing store file hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/41f447d2065af74b219164f0a70d08b3/.tmp/family/3b7e8219c4394dc1afc72aa7a7e2bc33 as hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/41f447d2065af74b219164f0a70d08b3/family/3b7e8219c4394dc1afc72aa7a7e2bc33 2018-02-04 11:00:39,748 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741878_1055 size 33261 2018-02-04 11:00:39,749 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741875_1052 size 33261 2018-02-04 11:00:39,749 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741876_1053 size 33261 2018-02-04 11:00:39,749 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741832_1008{UCState=UNDER_RECOVERY, truncateBlock=null, primaryNodeIndex=2, replicas=[ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619:NORMAL:127.0.0.1:45015|RBW]]} size 0 2018-02-04 11:00:39,749 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741832_1008{UCState=UNDER_RECOVERY, truncateBlock=null, primaryNodeIndex=2, replicas=[ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619:NORMAL:127.0.0.1:45015|RBW]]} size 0 2018-02-04 11:00:39,750 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741832_1008{UCState=UNDER_RECOVERY, truncateBlock=null, primaryNodeIndex=2, replicas=[ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619:NORMAL:127.0.0.1:45015|RBW]]} size 0 2018-02-04 11:00:39,751 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegionFileSystem(463): Committing store file hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/ad4f7bcab0a803ddae20b2866466d222/.tmp/family/3dc6cc18a9f94778b998d171f8b3d60a as hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/ad4f7bcab0a803ddae20b2866466d222/family/3dc6cc18a9f94778b998d171f8b3d60a 2018-02-04 11:00:39,775 INFO [RS_CLOSE_META-regionserver/ubuntu:0-0] regionserver.HRegion(1635): Closed hbase:meta,,1.1588230740 2018-02-04 11:00:39,775 DEBUG [RS_CLOSE_META-regionserver/ubuntu:0-0] handler.CloseRegionHandler(125): Closed hbase:meta,,1.1588230740 2018-02-04 11:00:39,775 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.DefaultStoreFlusher(85): Flushed, sequenceid=9, memsize=130.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/7429e32b4c4e0aadde8a8154cbd1f071/.tmp/family/c448f8a76b1449aba18ed1f0a5ac209d 2018-02-04 11:00:39,775 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.DefaultStoreFlusher(85): Flushed, sequenceid=9, memsize=130.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/6a7b42fe3d02ea7aca1d05fa666e87ac/.tmp/family/4a40ff3513884f8ca965cde522e81245 2018-02-04 11:00:39,780 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.DefaultStoreFlusher(85): Flushed, sequenceid=9, memsize=130.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/31b3e09f2db65dd6057a91f45cb8b163/.tmp/family/2406cb6cde66499cbb4a4b7d4fdd3cb0 2018-02-04 11:00:39,780 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HStore(1030): Added hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/ad4f7bcab0a803ddae20b2866466d222/family/3dc6cc18a9f94778b998d171f8b3d60a, entries=703, sequenceid=9, filesize=32.5 K 2018-02-04 11:00:39,781 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HStore(1030): Added hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/a127c3f989561b0b55d12be9bd499407/family/8c0e6a8bdba443749e38b34d717c1b62, entries=703, sequenceid=9, filesize=32.5 K 2018-02-04 11:00:39,784 WARN [AsyncFSWAL-0] wal.AsyncFSWAL(290): sync failed java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:424) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:513) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.sync(AsyncProtobufLogWriter.java:134) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:350) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.consume(AsyncFSWAL.java:533) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-02-04 11:00:39,785 DEBUG [regionserver/ubuntu:0.logRoller] regionserver.LogRoller(160): WAL roll requested 2018-02-04 11:00:39,785 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(2714): Finished memstore flush of ~130.44 KB/133570, currentsize=0 B/0 for region tabletest,kkk,1517713232695.a127c3f989561b0b55d12be9bd499407. in 787ms, sequenceid=9, compaction requested=false 2018-02-04 11:00:39,787 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741877_1054{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW]]} size 33261 2018-02-04 11:00:39,788 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741877_1054 size 33261 2018-02-04 11:00:39,788 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741879_1056{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW]]} size 33261 2018-02-04 11:00:39,788 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741877_1054 size 33261 2018-02-04 11:00:39,788 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741879_1056 size 33261 2018-02-04 11:00:39,788 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741879_1056 size 33261 2018-02-04 11:00:39,791 INFO [StoreCloserThread-tabletest,kkk,1517713232695.a127c3f989561b0b55d12be9bd499407.-1] regionserver.HStore(932): Closed family 2018-02-04 11:00:39,806 DEBUG [regionserver/ubuntu:0.logRoller] wal.AsyncFSWALProvider(109): Error instantiating log writer. org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): Parent directory doesn't exist: /user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/WALs/ubuntu,39267,1517713227316 at org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyParentDir(FSDirectory.java:1776) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2533) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2465) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2349) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:624) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:398) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2213) at org.apache.hadoop.ipc.Client.call(Client.java:1476) at org.apache.hadoop.ipc.Client.call(Client.java:1413) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy28.create(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296) at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy29.create(Unknown Source) at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:372) at com.sun.proxy.$Proxy30.create(Unknown Source) at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.lambda$1(FanOutOneBlockAsyncDFSOutputHelper.java:519) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$FileCreator.create(FanOutOneBlockAsyncDFSOutputHelper.java:210) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createOutput(FanOutOneBlockAsyncDFSOutputHelper.java:757) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$4(FanOutOneBlockAsyncDFSOutputHelper.java:743) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$16.doCall(FanOutOneBlockAsyncDFSOutputHelper.java:851) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$16.doCall(FanOutOneBlockAsyncDFSOutputHelper.java:1) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createOutput(FanOutOneBlockAsyncDFSOutputHelper.java:859) at org.apache.hadoop.hbase.io.asyncfs.AsyncFSOutputHelper.createOutput(AsyncFSOutputHelper.java:51) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.initOutput(AsyncProtobufLogWriter.java:159) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:167) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:99) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:612) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:123) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:766) at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:169) at java.lang.Thread.run(Thread.java:748) 2018-02-04 11:00:39,822 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegionFileSystem(463): Committing store file hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/31b3e09f2db65dd6057a91f45cb8b163/.tmp/family/2406cb6cde66499cbb4a4b7d4fdd3cb0 as hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/31b3e09f2db65dd6057a91f45cb8b163/family/2406cb6cde66499cbb4a4b7d4fdd3cb0 2018-02-04 11:00:39,823 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HStore(1030): Added hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/41f447d2065af74b219164f0a70d08b3/family/3b7e8219c4394dc1afc72aa7a7e2bc33, entries=703, sequenceid=9, filesize=32.5 K 2018-02-04 11:00:39,823 ERROR [regionserver/ubuntu:0.logRoller] helpers.MarkerIgnoringBase(159): ***** ABORTING region server ubuntu,39267,1517713227316: IOE in log roller ***** java.io.FileNotFoundException: Parent directory doesn't exist: /user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/WALs/ubuntu,39267,1517713227316 at org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyParentDir(FSDirectory.java:1776) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2533) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2465) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2349) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:624) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:398) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2213) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:183) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): Parent directory doesn't exist: /user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/WALs/ubuntu,39267,1517713227316 at org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyParentDir(FSDirectory.java:1776) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2533) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2465) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2349) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:624) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:398) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2213) at org.apache.hadoop.ipc.Client.call(Client.java:1476) at org.apache.hadoop.ipc.Client.call(Client.java:1413) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy28.create(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296) at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy29.create(Unknown Source) at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:372) at com.sun.proxy.$Proxy30.create(Unknown Source) at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.lambda$1(FanOutOneBlockAsyncDFSOutputHelper.java:519) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$FileCreator.create(FanOutOneBlockAsyncDFSOutputHelper.java:210) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createOutput(FanOutOneBlockAsyncDFSOutputHelper.java:757) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$4(FanOutOneBlockAsyncDFSOutputHelper.java:743) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$16.doCall(FanOutOneBlockAsyncDFSOutputHelper.java:851) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$16.doCall(FanOutOneBlockAsyncDFSOutputHelper.java:1) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createOutput(FanOutOneBlockAsyncDFSOutputHelper.java:859) at org.apache.hadoop.hbase.io.asyncfs.AsyncFSOutputHelper.createOutput(AsyncFSOutputHelper.java:51) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.initOutput(AsyncProtobufLogWriter.java:159) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:167) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:99) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:612) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:123) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:766) at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:169) ... 1 more 2018-02-04 11:00:39,823 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-1] helpers.MarkerIgnoringBase(159): ***** ABORTING region server ubuntu,39267,1517713227316: Replay of WAL required. Forcing server shutdown ***** org.apache.hadoop.hbase.DroppedSnapshotException: region: tabletest,hhh,1517713232695.ad4f7bcab0a803ddae20b2866466d222. at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2667) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2356) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): Parent directory doesn't exist: /user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/WALs/ubuntu,39267,1517713227316 at org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyParentDir(FSDirectory.java:1776) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2533) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2465) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2349) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:624) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:398) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2213) at org.apache.hadoop.ipc.Client.call(Client.java:1476) at org.apache.hadoop.ipc.Client.call(Client.java:1413) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy28.create(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296) at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy29.create(Unknown Source) at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:372) at com.sun.proxy.$Proxy30.create(Unknown Source) at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.lambda$1(FanOutOneBlockAsyncDFSOutputHelper.java:519) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$FileCreator.create(FanOutOneBlockAsyncDFSOutputHelper.java:210) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createOutput(FanOutOneBlockAsyncDFSOutputHelper.java:757) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$4(FanOutOneBlockAsyncDFSOutputHelper.java:743) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$16.doCall(FanOutOneBlockAsyncDFSOutputHelper.java:851) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$16.doCall(FanOutOneBlockAsyncDFSOutputHelper.java:1) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createOutput(FanOutOneBlockAsyncDFSOutputHelper.java:859) at org.apache.hadoop.hbase.io.asyncfs.AsyncFSOutputHelper.createOutput(AsyncFSOutputHelper.java:51) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.initOutput(AsyncProtobufLogWriter.java:159) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:167) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:99) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:612) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:123) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:766) at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:169) ... 1 more 2018-02-04 11:00:39,843 ERROR [regionserver/ubuntu:0.logRoller] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-02-04 11:00:39,843 INFO [regionserver/ubuntu:0.leaseChecker] regionserver.Leases(147): regionserver/ubuntu:0.leaseChecker closing leases 2018-02-04 11:00:39,848 INFO [regionserver/ubuntu:0.leaseChecker] regionserver.Leases(150): regionserver/ubuntu:0.leaseChecker closed leases 2018-02-04 11:00:39,848 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(2714): Finished memstore flush of ~130.44 KB/133570, currentsize=0 B/0 for region tabletest,ooo,1517713232695.41f447d2065af74b219164f0a70d08b3. in 856ms, sequenceid=9, compaction requested=false 2018-02-04 11:00:39,843 INFO [regionserver/ubuntu:0.leaseChecker] regionserver.Leases(147): regionserver/ubuntu:0.leaseChecker closing leases 2018-02-04 11:00:39,850 INFO [regionserver/ubuntu:0.leaseChecker] regionserver.Leases(150): regionserver/ubuntu:0.leaseChecker closed leases 2018-02-04 11:00:39,851 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-1] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-02-04 11:00:39,852 INFO [regionserver/ubuntu:0.Chore.1] hbase.ScheduledChore(180): Chore: MemstoreFlusherChore was stopped 2018-02-04 11:00:39,859 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegionFileSystem(463): Committing store file hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/7429e32b4c4e0aadde8a8154cbd1f071/.tmp/family/c448f8a76b1449aba18ed1f0a5ac209d as hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/7429e32b4c4e0aadde8a8154cbd1f071/family/c448f8a76b1449aba18ed1f0a5ac209d 2018-02-04 11:00:39,872 INFO [regionserver/ubuntu:0.leaseChecker] regionserver.Leases(147): regionserver/ubuntu:0.leaseChecker closing leases 2018-02-04 11:00:39,873 INFO [regionserver/ubuntu:0.leaseChecker] regionserver.Leases(150): regionserver/ubuntu:0.leaseChecker closed leases 2018-02-04 11:00:39,868 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] wal.WALSplitter(675): Wrote file=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/a127c3f989561b0b55d12be9bd499407/recovered.edits/12.seqid, newSeqId=12, maxSeqId=2 2018-02-04 11:00:39,865 INFO [regionserver/ubuntu:0.Chore.1] hbase.ScheduledChore(180): Chore: MemstoreFlusherChore was stopped 2018-02-04 11:00:39,877 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegionFileSystem(463): Committing store file hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/6a7b42fe3d02ea7aca1d05fa666e87ac/.tmp/family/4a40ff3513884f8ca965cde522e81245 as hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/6a7b42fe3d02ea7aca1d05fa666e87ac/family/4a40ff3513884f8ca965cde522e81245 2018-02-04 11:00:39,891 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(1635): Closed tabletest,kkk,1517713232695.a127c3f989561b0b55d12be9bd499407. 2018-02-04 11:00:39,892 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] handler.CloseRegionHandler(125): Closed tabletest,kkk,1517713232695.a127c3f989561b0b55d12be9bd499407. 2018-02-04 11:00:39,892 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] handler.CloseRegionHandler(92): Processing close of tabletest,ccc,1517713232695.cd5c06da9ebf17554a264cbdd6fcd9ef. 2018-02-04 11:00:39,898 INFO [regionserver/ubuntu:0.logRoller] regionserver.HRegionServer(2350): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 855638016, "init" : 197132288, "max" : 2794455040, "used" : 619489952 }, "NonHeapMemoryUsage" : { "committed" : 118317056, "init" : 2555904, "max" : -1, "used" : 115799672 }, "Verbose" : false, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : 1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 50164, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 113, "TotalCallTime_min" : 0, "TotalCallTime_max" : 120, "TotalCallTime_mean" : 12, "TotalCallTime_25th_percentile" : 1, "TotalCallTime_median" : 4, "TotalCallTime_75th_percentile" : 12, "TotalCallTime_90th_percentile" : 38, "TotalCallTime_95th_percentile" : 45, "TotalCallTime_98th_percentile" : 52, "TotalCallTime_99th_percentile" : 53, "TotalCallTime_99.9th_percentile" : 120, "TotalCallTime_TimeRangeCount_0-1" : 29, "TotalCallTime_TimeRangeCount_1-3" : 17, "TotalCallTime_TimeRangeCount_3-10" : 36, "TotalCallTime_TimeRangeCount_10-30" : 12, "TotalCallTime_TimeRangeCount_30-100" : 18, "TotalCallTime_TimeRangeCount_100-300" : 1, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 113, "ResponseSize_min" : 0, "ResponseSize_max" : 787, "ResponseSize_mean" : 37, "ResponseSize_25th_percentile" : 10, "ResponseSize_median" : 10, "ResponseSize_75th_percentile" : 51, "ResponseSize_90th_percentile" : 51, "ResponseSize_95th_percentile" : 57, "ResponseSize_98th_percentile" : 161, "ResponseSize_99th_percentile" : 486, "ResponseSize_99.9th_percentile" : 756, "ResponseSize_SizeRangeCount_0-10" : 73, "ResponseSize_SizeRangeCount_10-100" : 36, "ResponseSize_SizeRangeCount_100-1000" : 4, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 113, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 120, "ProcessCallTime_mean" : 9, "ProcessCallTime_25th_percentile" : 1, "ProcessCallTime_median" : 4, "ProcessCallTime_75th_percentile" : 11, "ProcessCallTime_90th_percentile" : 28, "ProcessCallTime_95th_percentile" : 38, "ProcessCallTime_98th_percentile" : 51, "ProcessCallTime_99th_percentile" : 53, "ProcessCallTime_99.9th_percentile" : 120, "ProcessCallTime_TimeRangeCount_0-1" : 36, "ProcessCallTime_TimeRangeCount_1-3" : 15, "ProcessCallTime_TimeRangeCount_3-10" : 33, "ProcessCallTime_TimeRangeCount_10-30" : 19, "ProcessCallTime_TimeRangeCount_30-100" : 9, "ProcessCallTime_TimeRangeCount_100-300" : 1, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 7, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 113, "RequestSize_min" : 70, "RequestSize_max" : 8691, "RequestSize_mean" : 424, "RequestSize_25th_percentile" : 81, "RequestSize_median" : 613, "RequestSize_75th_percentile" : 4652, "RequestSize_90th_percentile" : 7075, "RequestSize_95th_percentile" : 7883, "RequestSize_98th_percentile" : 8367, "RequestSize_99th_percentile" : 8529, "RequestSize_99.9th_percentile" : 8674, "RequestSize_SizeRangeCount_10-100" : 42, "RequestSize_SizeRangeCount_100-1000" : 71, "sentBytes" : 5263, "QueueCallTime_num_ops" : 113, "QueueCallTime_min" : 0, "QueueCallTime_max" : 31, "QueueCallTime_mean" : 2, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 1, "QueueCallTime_90th_percentile" : 9, "QueueCallTime_95th_percentile" : 19, "QueueCallTime_98th_percentile" : 29, "QueueCallTime_99th_percentile" : 29, "QueueCallTime_99.9th_percentile" : 31, "QueueCallTime_TimeRangeCount_0-1" : 92, "QueueCallTime_TimeRangeCount_1-3" : 6, "QueueCallTime_TimeRangeCount_3-10" : 4, "QueueCallTime_TimeRangeCount_10-30" : 10, "QueueCallTime_TimeRangeCount_30-100" : 1, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:56841", "tag.serverName" : "ubuntu,39267,1517713227316", "tag.clusterId" : "990c040f-15f4-4bef-a30c-08f39bc273d8", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "regionCount" : 10, "storeCount" : 1, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 0, "memStoreSize" : 78, "storeFileSize" : 0, "maxStoreFileAge" : 0, "minStoreFileAge" : 0, "avgStoreFileAge" : 0, "numReferenceFiles" : 0, "regionServerStartTime" : 1517713227316, "averageRegionSize" : 78, "totalRequestCount" : 9, "totalRowActionRequestCount" : 4, "readRequestCount" : 2, "filteredReadRequestCount" : 0, "writeRequestCount" : 2, "rpcGetRequestCount" : 4, "rpcScanRequestCount" : 1, "rpcMultiRequestCount" : 0, "rpcMutateRequestCount" : 2, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 0, "staticIndexSize" : 0, "staticBloomSize" : 0, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1116962296, "blockCacheCount" : 0, "blockCacheSize" : 819720, "blockCacheHitCount" : 0, "blockCacheHitCountPrimary" : 0, "blockCacheMissCount" : 0, "blockCacheMissCountPrimary" : 0, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 0.0, "blockCacheExpressHitPercent" : 0.0, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 0, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 0, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 0, "l1CacheMissCount" : 0, "l1CacheHitRatio" : NaN, "l1CacheMissRatio" : NaN, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 0, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 0, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 1, "ScanTime_min" : 1, "ScanTime_max" : 1, "ScanTime_mean" : 1, "ScanTime_25th_percentile" : 1, "ScanTime_median" : 1, "ScanTime_75th_percentile" : 1, "ScanTime_90th_percentile" : 1, "ScanTime_95th_percentile" : 1, "ScanTime_98th_percentile" : 1, "ScanTime_99th_percentile" : 1, "ScanTime_99.9th_percentile" : 1, "ScanTime_TimeRangeCount_0-1" : 1, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 2, "Put_min" : 2, "Put_max" : 9, "Put_mean" : 5, "Put_25th_percentile" : 2, "Put_median" : 3, "Put_75th_percentile" : 9, "Put_90th_percentile" : 9, "Put_95th_percentile" : 9, "Put_98th_percentile" : 9, "Put_99th_percentile" : 9, "Put_99.9th_percentile" : 9, "Put_TimeRangeCount_1-3" : 1, "Put_TimeRangeCount_3-10" : 1, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 0, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 0, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 4, "Get_min" : 0, "Get_max" : 1, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 1, "Get_75th_percentile" : 1, "Get_90th_percentile" : 1, "Get_95th_percentile" : 1, "Get_98th_percentile" : 1, "Get_99th_percentile" : 1, "Get_99.9th_percentile" : 1, "Get_TimeRangeCount_0-1" : 4, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 0, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 0, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 1, "ScanSize_min" : 86, "ScanSize_max" : 86, "ScanSize_mean" : 86, "ScanSize_25th_percentile" : 86, "ScanSize_median" : 86, "ScanSize_75th_percentile" : 86, "ScanSize_90th_percentile" : 86, "ScanSize_95th_percentile" : 86, "ScanSize_98th_percentile" : 86, "ScanSize_99th_percentile" : 86, "ScanSize_99.9th_percentile" : 86, "ScanSize_SizeRangeCount_10-100" : 1, "slowGetCount" : 0, "flushedMemstoreBytes" : 0, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 0, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-02-04 11:00:39,903 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(1483): Closing tabletest,ccc,1517713232695.cd5c06da9ebf17554a264cbdd6fcd9ef.: disabling compactions & flushes 2018-02-04 11:00:39,905 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(1522): Updates disabled for region tabletest,ccc,1517713232695.cd5c06da9ebf17554a264cbdd6fcd9ef. 2018-02-04 11:00:39,906 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(2518): Flushing 1/1 column families, memstore=130.44 KB 2018-02-04 11:00:39,906 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.CompactingMemStore(205): FLUSHING TO DISK: region tabletest,ccc,1517713232695.cd5c06da9ebf17554a264cbdd6fcd9ef.store: family 2018-02-04 11:00:39,907 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.CompactionPipeline(128): Swapping pipeline suffix; before=1, new segement=null 2018-02-04 11:00:39,895 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HStore(1030): Added hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/31b3e09f2db65dd6057a91f45cb8b163/family/2406cb6cde66499cbb4a4b7d4fdd3cb0, entries=703, sequenceid=9, filesize=32.5 K 2018-02-04 11:00:39,896 INFO [regionserver/ubuntu:0.Chore.1] hbase.ScheduledChore(180): Chore: MemstoreFlusherChore was stopped 2018-02-04 11:00:39,932 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HStore(1030): Added hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/7429e32b4c4e0aadde8a8154cbd1f071/family/c448f8a76b1449aba18ed1f0a5ac209d, entries=703, sequenceid=9, filesize=32.5 K 2018-02-04 11:00:39,930 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegionServer(2350): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 855638016, "init" : 197132288, "max" : 2794455040, "used" : 619402744 }, "NonHeapMemoryUsage" : { "committed" : 118317056, "init" : 2555904, "max" : -1, "used" : 115797072 }, "Verbose" : false, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : 1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 50164, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 113, "TotalCallTime_min" : 0, "TotalCallTime_max" : 120, "TotalCallTime_mean" : 12, "TotalCallTime_25th_percentile" : 1, "TotalCallTime_median" : 4, "TotalCallTime_75th_percentile" : 12, "TotalCallTime_90th_percentile" : 38, "TotalCallTime_95th_percentile" : 45, "TotalCallTime_98th_percentile" : 52, "TotalCallTime_99th_percentile" : 53, "TotalCallTime_99.9th_percentile" : 120, "TotalCallTime_TimeRangeCount_0-1" : 29, "TotalCallTime_TimeRangeCount_1-3" : 17, "TotalCallTime_TimeRangeCount_3-10" : 36, "TotalCallTime_TimeRangeCount_10-30" : 12, "TotalCallTime_TimeRangeCount_30-100" : 18, "TotalCallTime_TimeRangeCount_100-300" : 1, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 113, "ResponseSize_min" : 0, "ResponseSize_max" : 787, "ResponseSize_mean" : 37, "ResponseSize_25th_percentile" : 10, "ResponseSize_median" : 10, "ResponseSize_75th_percentile" : 51, "ResponseSize_90th_percentile" : 51, "ResponseSize_95th_percentile" : 57, "ResponseSize_98th_percentile" : 161, "ResponseSize_99th_percentile" : 486, "ResponseSize_99.9th_percentile" : 756, "ResponseSize_SizeRangeCount_0-10" : 73, "ResponseSize_SizeRangeCount_10-100" : 36, "ResponseSize_SizeRangeCount_100-1000" : 4, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 113, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 120, "ProcessCallTime_mean" : 9, "ProcessCallTime_25th_percentile" : 1, "ProcessCallTime_median" : 4, "ProcessCallTime_75th_percentile" : 11, "ProcessCallTime_90th_percentile" : 28, "ProcessCallTime_95th_percentile" : 38, "ProcessCallTime_98th_percentile" : 51, "ProcessCallTime_99th_percentile" : 53, "ProcessCallTime_99.9th_percentile" : 120, "ProcessCallTime_TimeRangeCount_0-1" : 36, "ProcessCallTime_TimeRangeCount_1-3" : 15, "ProcessCallTime_TimeRangeCount_3-10" : 33, "ProcessCallTime_TimeRangeCount_10-30" : 19, "ProcessCallTime_TimeRangeCount_30-100" : 9, "ProcessCallTime_TimeRangeCount_100-300" : 1, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 7, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 113, "RequestSize_min" : 70, "RequestSize_max" : 8691, "RequestSize_mean" : 424, "RequestSize_25th_percentile" : 81, "RequestSize_median" : 613, "RequestSize_75th_percentile" : 4652, "RequestSize_90th_percentile" : 7075, "RequestSize_95th_percentile" : 7883, "RequestSize_98th_percentile" : 8367, "RequestSize_99th_percentile" : 8529, "RequestSize_99.9th_percentile" : 8674, "RequestSize_SizeRangeCount_10-100" : 42, "RequestSize_SizeRangeCount_100-1000" : 71, "sentBytes" : 5263, "QueueCallTime_num_ops" : 113, "QueueCallTime_min" : 0, "QueueCallTime_max" : 31, "QueueCallTime_mean" : 2, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 1, "QueueCallTime_90th_percentile" : 9, "QueueCallTime_95th_percentile" : 19, "QueueCallTime_98th_percentile" : 29, "QueueCallTime_99th_percentile" : 29, "QueueCallTime_99.9th_percentile" : 31, "QueueCallTime_TimeRangeCount_0-1" : 92, "QueueCallTime_TimeRangeCount_1-3" : 6, "QueueCallTime_TimeRangeCount_3-10" : 4, "QueueCallTime_TimeRangeCount_10-30" : 10, "QueueCallTime_TimeRangeCount_30-100" : 1, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:56841", "tag.serverName" : "ubuntu,39267,1517713227316", "tag.clusterId" : "990c040f-15f4-4bef-a30c-08f39bc273d8", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "regionCount" : 10, "storeCount" : 1, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 0, "memStoreSize" : 78, "storeFileSize" : 0, "maxStoreFileAge" : 0, "minStoreFileAge" : 0, "avgStoreFileAge" : 0, "numReferenceFiles" : 0, "regionServerStartTime" : 1517713227316, "averageRegionSize" : 78, "totalRequestCount" : 9, "totalRowActionRequestCount" : 4, "readRequestCount" : 2, "filteredReadRequestCount" : 0, "writeRequestCount" : 2, "rpcGetRequestCount" : 4, "rpcScanRequestCount" : 1, "rpcMultiRequestCount" : 0, "rpcMutateRequestCount" : 2, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 0, "staticIndexSize" : 0, "staticBloomSize" : 0, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1116962296, "blockCacheCount" : 0, "blockCacheSize" : 819720, "blockCacheHitCount" : 0, "blockCacheHitCountPrimary" : 0, "blockCacheMissCount" : 0, "blockCacheMissCountPrimary" : 0, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 0.0, "blockCacheExpressHitPercent" : 0.0, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 0, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 0, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 0, "l1CacheMissCount" : 0, "l1CacheHitRatio" : NaN, "l1CacheMissRatio" : NaN, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 0, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 0, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 1, "ScanTime_min" : 1, "ScanTime_max" : 1, "ScanTime_mean" : 1, "ScanTime_25th_percentile" : 1, "ScanTime_median" : 1, "ScanTime_75th_percentile" : 1, "ScanTime_90th_percentile" : 1, "ScanTime_95th_percentile" : 1, "ScanTime_98th_percentile" : 1, "ScanTime_99th_percentile" : 1, "ScanTime_99.9th_percentile" : 1, "ScanTime_TimeRangeCount_0-1" : 1, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 2, "Put_min" : 2, "Put_max" : 9, "Put_mean" : 5, "Put_25th_percentile" : 2, "Put_median" : 3, "Put_75th_percentile" : 9, "Put_90th_percentile" : 9, "Put_95th_percentile" : 9, "Put_98th_percentile" : 9, "Put_99th_percentile" : 9, "Put_99.9th_percentile" : 9, "Put_TimeRangeCount_1-3" : 1, "Put_TimeRangeCount_3-10" : 1, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 0, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 0, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 4, "Get_min" : 0, "Get_max" : 1, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 1, "Get_75th_percentile" : 1, "Get_90th_percentile" : 1, "Get_95th_percentile" : 1, "Get_98th_percentile" : 1, "Get_99th_percentile" : 1, "Get_99.9th_percentile" : 1, "Get_TimeRangeCount_0-1" : 4, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 0, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 0, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 1, "ScanSize_min" : 86, "ScanSize_max" : 86, "ScanSize_mean" : 86, "ScanSize_25th_percentile" : 86, "ScanSize_median" : 86, "ScanSize_75th_percentile" : 86, "ScanSize_90th_percentile" : 86, "ScanSize_95th_percentile" : 86, "ScanSize_98th_percentile" : 86, "ScanSize_99th_percentile" : 86, "ScanSize_99.9th_percentile" : 86, "ScanSize_SizeRangeCount_10-100" : 1, "slowGetCount" : 0, "flushedMemstoreBytes" : 0, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 0, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-02-04 11:00:39,933 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HStore(1030): Added hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/6a7b42fe3d02ea7aca1d05fa666e87ac/family/4a40ff3513884f8ca965cde522e81245, entries=703, sequenceid=9, filesize=32.5 K 2018-02-04 11:00:39,967 ERROR [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=39063] master.MasterRpcServices(503): Region server ubuntu,39267,1517713227316 reported a fatal error: ***** ABORTING region server ubuntu,39267,1517713227316: Replay of WAL required. Forcing server shutdown ***** Cause: org.apache.hadoop.hbase.DroppedSnapshotException: region: tabletest,hhh,1517713232695.ad4f7bcab0a803ddae20b2866466d222. at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2667) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2356) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): Parent directory doesn't exist: /user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/WALs/ubuntu,39267,1517713227316 at org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyParentDir(FSDirectory.java:1776) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2533) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2465) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2349) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:624) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:398) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2213) at org.apache.hadoop.ipc.Client.call(Client.java:1476) at org.apache.hadoop.ipc.Client.call(Client.java:1413) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy28.create(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296) at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy29.create(Unknown Source) at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:372) at com.sun.proxy.$Proxy30.create(Unknown Source) at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.lambda$1(FanOutOneBlockAsyncDFSOutputHelper.java:519) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$FileCreator.create(FanOutOneBlockAsyncDFSOutputHelper.java:210) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createOutput(FanOutOneBlockAsyncDFSOutputHelper.java:757) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$4(FanOutOneBlockAsyncDFSOutputHelper.java:743) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$16.doCall(FanOutOneBlockAsyncDFSOutputHelper.java:851) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$16.doCall(FanOutOneBlockAsyncDFSOutputHelper.java:1) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createOutput(FanOutOneBlockAsyncDFSOutputHelper.java:859) at org.apache.hadoop.hbase.io.asyncfs.AsyncFSOutputHelper.createOutput(AsyncFSOutputHelper.java:51) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.initOutput(AsyncProtobufLogWriter.java:159) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:167) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:99) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:612) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:123) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:766) at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:169) ... 1 more 2018-02-04 11:00:39,969 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(2714): Finished memstore flush of ~130.44 KB/133570, currentsize=0 B/0 for region tabletest,nnn,1517713232695.7429e32b4c4e0aadde8a8154cbd1f071. in 427ms, sequenceid=9, compaction requested=false 2018-02-04 11:00:39,974 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(2714): Finished memstore flush of ~130.44 KB/133570, currentsize=0 B/0 for region tabletest,qqq,1517713232695.6a7b42fe3d02ea7aca1d05fa666e87ac. in 419ms, sequenceid=9, compaction requested=false 2018-02-04 11:00:39,975 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-1] helpers.MarkerIgnoringBase(159): ***** ABORTING region server ubuntu,39267,1517713227316: Unrecoverable exception while closing region tabletest,hhh,1517713232695.ad4f7bcab0a803ddae20b2866466d222., still finishing close ***** org.apache.hadoop.hbase.DroppedSnapshotException: region: tabletest,hhh,1517713232695.ad4f7bcab0a803ddae20b2866466d222. at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2667) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2356) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): Parent directory doesn't exist: /user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/WALs/ubuntu,39267,1517713227316 at org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyParentDir(FSDirectory.java:1776) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2533) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2465) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2349) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:624) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:398) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2213) at org.apache.hadoop.ipc.Client.call(Client.java:1476) at org.apache.hadoop.ipc.Client.call(Client.java:1413) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy28.create(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296) at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy29.create(Unknown Source) at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:372) at com.sun.proxy.$Proxy30.create(Unknown Source) at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.lambda$1(FanOutOneBlockAsyncDFSOutputHelper.java:519) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$FileCreator.create(FanOutOneBlockAsyncDFSOutputHelper.java:210) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createOutput(FanOutOneBlockAsyncDFSOutputHelper.java:757) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$4(FanOutOneBlockAsyncDFSOutputHelper.java:743) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$16.doCall(FanOutOneBlockAsyncDFSOutputHelper.java:851) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$16.doCall(FanOutOneBlockAsyncDFSOutputHelper.java:1) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createOutput(FanOutOneBlockAsyncDFSOutputHelper.java:859) at org.apache.hadoop.hbase.io.asyncfs.AsyncFSOutputHelper.createOutput(AsyncFSOutputHelper.java:51) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.initOutput(AsyncProtobufLogWriter.java:159) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:167) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:99) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:612) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:123) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:766) at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:169) ... 1 more 2018-02-04 11:00:39,976 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-1] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-02-04 11:00:39,978 ERROR [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=39063] master.MasterRpcServices(503): Region server ubuntu,39267,1517713227316 reported a fatal error: ***** ABORTING region server ubuntu,39267,1517713227316: IOE in log roller ***** Cause: java.io.FileNotFoundException: Parent directory doesn't exist: /user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/WALs/ubuntu,39267,1517713227316 at org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyParentDir(FSDirectory.java:1776) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2533) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2465) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2349) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:624) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:398) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2213) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:183) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): Parent directory doesn't exist: /user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/WALs/ubuntu,39267,1517713227316 at org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyParentDir(FSDirectory.java:1776) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2533) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2465) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2349) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:624) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:398) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2213) at org.apache.hadoop.ipc.Client.call(Client.java:1476) at org.apache.hadoop.ipc.Client.call(Client.java:1413) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy28.create(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296) at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy29.create(Unknown Source) at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:372) at com.sun.proxy.$Proxy30.create(Unknown Source) at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.lambda$1(FanOutOneBlockAsyncDFSOutputHelper.java:519) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$FileCreator.create(FanOutOneBlockAsyncDFSOutputHelper.java:210) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createOutput(FanOutOneBlockAsyncDFSOutputHelper.java:757) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$4(FanOutOneBlockAsyncDFSOutputHelper.java:743) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$16.doCall(FanOutOneBlockAsyncDFSOutputHelper.java:851) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$16.doCall(FanOutOneBlockAsyncDFSOutputHelper.java:1) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createOutput(FanOutOneBlockAsyncDFSOutputHelper.java:859) at org.apache.hadoop.hbase.io.asyncfs.AsyncFSOutputHelper.createOutput(AsyncFSOutputHelper.java:51) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.initOutput(AsyncProtobufLogWriter.java:159) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:167) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:99) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:612) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:123) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:766) at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:169) ... 1 more 2018-02-04 11:00:39,980 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegionServer(2350): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 855638016, "init" : 197132288, "max" : 2794455040, "used" : 623552000 }, "NonHeapMemoryUsage" : { "committed" : 118317056, "init" : 2555904, "max" : -1, "used" : 115801576 }, "Verbose" : false, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : 1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 50164, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 113, "TotalCallTime_min" : 0, "TotalCallTime_max" : 120, "TotalCallTime_mean" : 12, "TotalCallTime_25th_percentile" : 1, "TotalCallTime_median" : 4, "TotalCallTime_75th_percentile" : 12, "TotalCallTime_90th_percentile" : 38, "TotalCallTime_95th_percentile" : 45, "TotalCallTime_98th_percentile" : 52, "TotalCallTime_99th_percentile" : 53, "TotalCallTime_99.9th_percentile" : 120, "TotalCallTime_TimeRangeCount_0-1" : 29, "TotalCallTime_TimeRangeCount_1-3" : 17, "TotalCallTime_TimeRangeCount_3-10" : 36, "TotalCallTime_TimeRangeCount_10-30" : 12, "TotalCallTime_TimeRangeCount_30-100" : 18, "TotalCallTime_TimeRangeCount_100-300" : 1, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 113, "ResponseSize_min" : 0, "ResponseSize_max" : 787, "ResponseSize_mean" : 37, "ResponseSize_25th_percentile" : 10, "ResponseSize_median" : 10, "ResponseSize_75th_percentile" : 51, "ResponseSize_90th_percentile" : 51, "ResponseSize_95th_percentile" : 57, "ResponseSize_98th_percentile" : 161, "ResponseSize_99th_percentile" : 486, "ResponseSize_99.9th_percentile" : 756, "ResponseSize_SizeRangeCount_0-10" : 73, "ResponseSize_SizeRangeCount_10-100" : 36, "ResponseSize_SizeRangeCount_100-1000" : 4, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 113, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 120, "ProcessCallTime_mean" : 9, "ProcessCallTime_25th_percentile" : 1, "ProcessCallTime_median" : 4, "ProcessCallTime_75th_percentile" : 11, "ProcessCallTime_90th_percentile" : 28, "ProcessCallTime_95th_percentile" : 38, "ProcessCallTime_98th_percentile" : 51, "ProcessCallTime_99th_percentile" : 53, "ProcessCallTime_99.9th_percentile" : 120, "ProcessCallTime_TimeRangeCount_0-1" : 36, "ProcessCallTime_TimeRangeCount_1-3" : 15, "ProcessCallTime_TimeRangeCount_3-10" : 33, "ProcessCallTime_TimeRangeCount_10-30" : 19, "ProcessCallTime_TimeRangeCount_30-100" : 9, "ProcessCallTime_TimeRangeCount_100-300" : 1, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 7, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 113, "RequestSize_min" : 70, "RequestSize_max" : 8691, "RequestSize_mean" : 424, "RequestSize_25th_percentile" : 81, "RequestSize_median" : 613, "RequestSize_75th_percentile" : 4652, "RequestSize_90th_percentile" : 7075, "RequestSize_95th_percentile" : 7883, "RequestSize_98th_percentile" : 8367, "RequestSize_99th_percentile" : 8529, "RequestSize_99.9th_percentile" : 8674, "RequestSize_SizeRangeCount_10-100" : 42, "RequestSize_SizeRangeCount_100-1000" : 71, "sentBytes" : 5263, "QueueCallTime_num_ops" : 113, "QueueCallTime_min" : 0, "QueueCallTime_max" : 31, "QueueCallTime_mean" : 2, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 1, "QueueCallTime_90th_percentile" : 9, "QueueCallTime_95th_percentile" : 19, "QueueCallTime_98th_percentile" : 29, "QueueCallTime_99th_percentile" : 29, "QueueCallTime_99.9th_percentile" : 31, "QueueCallTime_TimeRangeCount_0-1" : 92, "QueueCallTime_TimeRangeCount_1-3" : 6, "QueueCallTime_TimeRangeCount_3-10" : 4, "QueueCallTime_TimeRangeCount_10-30" : 10, "QueueCallTime_TimeRangeCount_30-100" : 1, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:56841", "tag.serverName" : "ubuntu,39267,1517713227316", "tag.clusterId" : "990c040f-15f4-4bef-a30c-08f39bc273d8", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "regionCount" : 10, "storeCount" : 1, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 0, "memStoreSize" : 78, "storeFileSize" : 0, "maxStoreFileAge" : 0, "minStoreFileAge" : 0, "avgStoreFileAge" : 0, "numReferenceFiles" : 0, "regionServerStartTime" : 1517713227316, "averageRegionSize" : 78, "totalRequestCount" : 9, "totalRowActionRequestCount" : 4, "readRequestCount" : 2, "filteredReadRequestCount" : 0, "writeRequestCount" : 2, "rpcGetRequestCount" : 4, "rpcScanRequestCount" : 1, "rpcMultiRequestCount" : 0, "rpcMutateRequestCount" : 2, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 0, "staticIndexSize" : 0, "staticBloomSize" : 0, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1116962296, "blockCacheCount" : 0, "blockCacheSize" : 819720, "blockCacheHitCount" : 0, "blockCacheHitCountPrimary" : 0, "blockCacheMissCount" : 0, "blockCacheMissCountPrimary" : 0, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 0.0, "blockCacheExpressHitPercent" : 0.0, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 0, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 0, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 0, "l1CacheMissCount" : 0, "l1CacheHitRatio" : NaN, "l1CacheMissRatio" : NaN, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 0, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 0, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 1, "ScanTime_min" : 1, "ScanTime_max" : 1, "ScanTime_mean" : 1, "ScanTime_25th_percentile" : 1, "ScanTime_median" : 1, "ScanTime_75th_percentile" : 1, "ScanTime_90th_percentile" : 1, "ScanTime_95th_percentile" : 1, "ScanTime_98th_percentile" : 1, "ScanTime_99th_percentile" : 1, "ScanTime_99.9th_percentile" : 1, "ScanTime_TimeRangeCount_0-1" : 1, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 2, "Put_min" : 2, "Put_max" : 9, "Put_mean" : 5, "Put_25th_percentile" : 2, "Put_median" : 3, "Put_75th_percentile" : 9, "Put_90th_percentile" : 9, "Put_95th_percentile" : 9, "Put_98th_percentile" : 9, "Put_99th_percentile" : 9, "Put_99.9th_percentile" : 9, "Put_TimeRangeCount_1-3" : 1, "Put_TimeRangeCount_3-10" : 1, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 0, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 0, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 4, "Get_min" : 0, "Get_max" : 1, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 1, "Get_75th_percentile" : 1, "Get_90th_percentile" : 1, "Get_95th_percentile" : 1, "Get_98th_percentile" : 1, "Get_99th_percentile" : 1, "Get_99.9th_percentile" : 1, "Get_TimeRangeCount_0-1" : 4, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 0, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 0, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 1, "ScanSize_min" : 86, "ScanSize_max" : 86, "ScanSize_mean" : 86, "ScanSize_25th_percentile" : 86, "ScanSize_median" : 86, "ScanSize_75th_percentile" : 86, "ScanSize_90th_percentile" : 86, "ScanSize_95th_percentile" : 86, "ScanSize_98th_percentile" : 86, "ScanSize_99th_percentile" : 86, "ScanSize_99.9th_percentile" : 86, "ScanSize_SizeRangeCount_10-100" : 1, "slowGetCount" : 0, "flushedMemstoreBytes" : 0, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 0, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-02-04 11:00:39,983 ERROR [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=39063] master.MasterRpcServices(503): Region server ubuntu,39267,1517713227316 reported a fatal error: ***** ABORTING region server ubuntu,39267,1517713227316: Unrecoverable exception while closing region tabletest,hhh,1517713232695.ad4f7bcab0a803ddae20b2866466d222., still finishing close ***** Cause: org.apache.hadoop.hbase.DroppedSnapshotException: region: tabletest,hhh,1517713232695.ad4f7bcab0a803ddae20b2866466d222. at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2667) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2356) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): Parent directory doesn't exist: /user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/WALs/ubuntu,39267,1517713227316 at org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyParentDir(FSDirectory.java:1776) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2533) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2465) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2349) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:624) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:398) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2213) at org.apache.hadoop.ipc.Client.call(Client.java:1476) at org.apache.hadoop.ipc.Client.call(Client.java:1413) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy28.create(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296) at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy29.create(Unknown Source) at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:372) at com.sun.proxy.$Proxy30.create(Unknown Source) at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.lambda$1(FanOutOneBlockAsyncDFSOutputHelper.java:519) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$FileCreator.create(FanOutOneBlockAsyncDFSOutputHelper.java:210) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createOutput(FanOutOneBlockAsyncDFSOutputHelper.java:757) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$4(FanOutOneBlockAsyncDFSOutputHelper.java:743) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$16.doCall(FanOutOneBlockAsyncDFSOutputHelper.java:851) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$16.doCall(FanOutOneBlockAsyncDFSOutputHelper.java:1) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createOutput(FanOutOneBlockAsyncDFSOutputHelper.java:859) at org.apache.hadoop.hbase.io.asyncfs.AsyncFSOutputHelper.createOutput(AsyncFSOutputHelper.java:51) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.initOutput(AsyncProtobufLogWriter.java:159) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:167) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:99) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:612) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:123) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:766) at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:169) ... 1 more 2018-02-04 11:00:39,983 INFO [RS:2;ubuntu:36383] regionserver.HRegionServer(1383): Waiting on 5 regions to close 2018-02-04 11:00:39,984 DEBUG [RS:2;ubuntu:36383] regionserver.HRegionServer(1387): Online Regions={9579a7c2f22113dfe574ba9569101416=tabletest,uuu,1517713232695.9579a7c2f22113dfe574ba9569101416., 7429e32b4c4e0aadde8a8154cbd1f071=tabletest,nnn,1517713232695.7429e32b4c4e0aadde8a8154cbd1f071., 6a7b42fe3d02ea7aca1d05fa666e87ac=tabletest,qqq,1517713232695.6a7b42fe3d02ea7aca1d05fa666e87ac., 756819a3f903733cb342d3913aab52a6=tabletest,bbb,1517713232695.756819a3f903733cb342d3913aab52a6., 12bc4be2348528b0bd4575ae03200fc3=tabletest,ppp,1517713232695.12bc4be2348528b0bd4575ae03200fc3.} 2018-02-04 11:00:39,984 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-1] executor.EventHandler(186): Caught throwable while processing event M_RS_CLOSE_REGION java.lang.RuntimeException: org.apache.hadoop.hbase.DroppedSnapshotException: region: tabletest,hhh,1517713232695.ad4f7bcab0a803ddae20b2866466d222. at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:117) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.hbase.DroppedSnapshotException: region: tabletest,hhh,1517713232695.ad4f7bcab0a803ddae20b2866466d222. at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2667) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2356) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) ... 4 more Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): Parent directory doesn't exist: /user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/WALs/ubuntu,39267,1517713227316 at org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyParentDir(FSDirectory.java:1776) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2533) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2465) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2349) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:624) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:398) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2213) at org.apache.hadoop.ipc.Client.call(Client.java:1476) at org.apache.hadoop.ipc.Client.call(Client.java:1413) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy28.create(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296) at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy29.create(Unknown Source) at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:372) at com.sun.proxy.$Proxy30.create(Unknown Source) at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.lambda$1(FanOutOneBlockAsyncDFSOutputHelper.java:519) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$FileCreator.create(FanOutOneBlockAsyncDFSOutputHelper.java:210) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createOutput(FanOutOneBlockAsyncDFSOutputHelper.java:757) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$4(FanOutOneBlockAsyncDFSOutputHelper.java:743) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$16.doCall(FanOutOneBlockAsyncDFSOutputHelper.java:851) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$16.doCall(FanOutOneBlockAsyncDFSOutputHelper.java:1) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createOutput(FanOutOneBlockAsyncDFSOutputHelper.java:859) at org.apache.hadoop.hbase.io.asyncfs.AsyncFSOutputHelper.createOutput(AsyncFSOutputHelper.java:51) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.initOutput(AsyncProtobufLogWriter.java:159) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:167) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:99) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:612) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:123) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:766) at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:169) ... 1 more 2018-02-04 11:00:39,985 INFO [StoreCloserThread-tabletest,ooo,1517713232695.41f447d2065af74b219164f0a70d08b3.-1] regionserver.HStore(932): Closed family 2018-02-04 11:00:39,986 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-1] helpers.MarkerIgnoringBase(159): ***** ABORTING region server ubuntu,39267,1517713227316: Caught throwable while processing event M_RS_CLOSE_REGION ***** java.lang.RuntimeException: org.apache.hadoop.hbase.DroppedSnapshotException: region: tabletest,hhh,1517713232695.ad4f7bcab0a803ddae20b2866466d222. at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:117) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.hbase.DroppedSnapshotException: region: tabletest,hhh,1517713232695.ad4f7bcab0a803ddae20b2866466d222. at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2667) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2356) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) ... 4 more Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): Parent directory doesn't exist: /user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/WALs/ubuntu,39267,1517713227316 at org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyParentDir(FSDirectory.java:1776) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2533) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2465) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2349) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:624) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:398) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2213) at org.apache.hadoop.ipc.Client.call(Client.java:1476) at org.apache.hadoop.ipc.Client.call(Client.java:1413) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy28.create(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296) at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy29.create(Unknown Source) at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:372) at com.sun.proxy.$Proxy30.create(Unknown Source) at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.lambda$1(FanOutOneBlockAsyncDFSOutputHelper.java:519) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$FileCreator.create(FanOutOneBlockAsyncDFSOutputHelper.java:210) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createOutput(FanOutOneBlockAsyncDFSOutputHelper.java:757) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$4(FanOutOneBlockAsyncDFSOutputHelper.java:743) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$16.doCall(FanOutOneBlockAsyncDFSOutputHelper.java:851) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$16.doCall(FanOutOneBlockAsyncDFSOutputHelper.java:1) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createOutput(FanOutOneBlockAsyncDFSOutputHelper.java:859) at org.apache.hadoop.hbase.io.asyncfs.AsyncFSOutputHelper.createOutput(AsyncFSOutputHelper.java:51) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.initOutput(AsyncProtobufLogWriter.java:159) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:167) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:99) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:612) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:123) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:766) at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:169) ... 1 more 2018-02-04 11:00:39,987 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-1] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-02-04 11:00:40,001 INFO [RS:0;ubuntu:39267] regionserver.HRegionServer(1383): Waiting on 8 regions to close 2018-02-04 11:00:40,001 DEBUG [RS:0;ubuntu:39267] regionserver.HRegionServer(1387): Online Regions={ad4f7bcab0a803ddae20b2866466d222=tabletest,hhh,1517713232695.ad4f7bcab0a803ddae20b2866466d222., 31b3e09f2db65dd6057a91f45cb8b163=tabletest,ttt,1517713232695.31b3e09f2db65dd6057a91f45cb8b163., 68c9676b485038d63fd2cf8a435420e5=tabletest,ddd,1517713232695.68c9676b485038d63fd2cf8a435420e5., cf37e425ee5be11cb09b53c3671e1599=hbase:namespace,,1517713230960.cf37e425ee5be11cb09b53c3671e1599., f1f365dfd0648c0fb5d9848c1db452cc=tabletest,iii,1517713232695.f1f365dfd0648c0fb5d9848c1db452cc., e063e6b2c2640eb6eafc8737a71fb5f5=tabletest,sss,1517713232695.e063e6b2c2640eb6eafc8737a71fb5f5., 14343105008c3cdd39a78990296fbe94=tabletest,eee,1517713232695.14343105008c3cdd39a78990296fbe94., 7b96ff3277efd9751a2175b9c2b4d481=tabletest,fff,1517713232695.7b96ff3277efd9751a2175b9c2b4d481.} 2018-02-04 11:00:40,014 INFO [RS:1;ubuntu:46739] regionserver.HRegionServer(1383): Waiting on 7 regions to close 2018-02-04 11:00:40,014 DEBUG [RS:1;ubuntu:46739] regionserver.HRegionServer(1387): Online Regions={41f447d2065af74b219164f0a70d08b3=tabletest,ooo,1517713232695.41f447d2065af74b219164f0a70d08b3., 20457e81e436062b31548a08737ebba4=tabletest,yyy,1517713232695.20457e81e436062b31548a08737ebba4., cd5c06da9ebf17554a264cbdd6fcd9ef=tabletest,ccc,1517713232695.cd5c06da9ebf17554a264cbdd6fcd9ef., 85d5a900ae52e80b00b89bb74335621c=tabletest,lll,1517713232695.85d5a900ae52e80b00b89bb74335621c., 1da7d397be67579013b1f635bfbcb064=tabletest,zzz,1517713232695.1da7d397be67579013b1f635bfbcb064., e4481a648e0b377eb5bf59bfb99dfd88=tabletest,jjj,1517713232695.e4481a648e0b377eb5bf59bfb99dfd88., 7514f60148f3a526baae5c7ce2c78c1b=tabletest,www,1517713232695.7514f60148f3a526baae5c7ce2c78c1b.} 2018-02-04 11:00:40,016 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegionServer(2350): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 855638016, "init" : 197132288, "max" : 2794455040, "used" : 625646608 }, "NonHeapMemoryUsage" : { "committed" : 118317056, "init" : 2555904, "max" : -1, "used" : 115773928 }, "Verbose" : false, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : 1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 50164, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 113, "TotalCallTime_min" : 0, "TotalCallTime_max" : 120, "TotalCallTime_mean" : 12, "TotalCallTime_25th_percentile" : 1, "TotalCallTime_median" : 4, "TotalCallTime_75th_percentile" : 12, "TotalCallTime_90th_percentile" : 38, "TotalCallTime_95th_percentile" : 45, "TotalCallTime_98th_percentile" : 52, "TotalCallTime_99th_percentile" : 53, "TotalCallTime_99.9th_percentile" : 120, "TotalCallTime_TimeRangeCount_0-1" : 29, "TotalCallTime_TimeRangeCount_1-3" : 17, "TotalCallTime_TimeRangeCount_3-10" : 36, "TotalCallTime_TimeRangeCount_10-30" : 12, "TotalCallTime_TimeRangeCount_30-100" : 18, "TotalCallTime_TimeRangeCount_100-300" : 1, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 113, "ResponseSize_min" : 0, "ResponseSize_max" : 787, "ResponseSize_mean" : 37, "ResponseSize_25th_percentile" : 10, "ResponseSize_median" : 10, "ResponseSize_75th_percentile" : 51, "ResponseSize_90th_percentile" : 51, "ResponseSize_95th_percentile" : 57, "ResponseSize_98th_percentile" : 161, "ResponseSize_99th_percentile" : 486, "ResponseSize_99.9th_percentile" : 756, "ResponseSize_SizeRangeCount_0-10" : 73, "ResponseSize_SizeRangeCount_10-100" : 36, "ResponseSize_SizeRangeCount_100-1000" : 4, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 113, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 120, "ProcessCallTime_mean" : 9, "ProcessCallTime_25th_percentile" : 1, "ProcessCallTime_median" : 4, "ProcessCallTime_75th_percentile" : 11, "ProcessCallTime_90th_percentile" : 28, "ProcessCallTime_95th_percentile" : 38, "ProcessCallTime_98th_percentile" : 51, "ProcessCallTime_99th_percentile" : 53, "ProcessCallTime_99.9th_percentile" : 120, "ProcessCallTime_TimeRangeCount_0-1" : 36, "ProcessCallTime_TimeRangeCount_1-3" : 15, "ProcessCallTime_TimeRangeCount_3-10" : 33, "ProcessCallTime_TimeRangeCount_10-30" : 19, "ProcessCallTime_TimeRangeCount_30-100" : 9, "ProcessCallTime_TimeRangeCount_100-300" : 1, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 7, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 113, "RequestSize_min" : 70, "RequestSize_max" : 8691, "RequestSize_mean" : 424, "RequestSize_25th_percentile" : 81, "RequestSize_median" : 613, "RequestSize_75th_percentile" : 4652, "RequestSize_90th_percentile" : 7075, "RequestSize_95th_percentile" : 7883, "RequestSize_98th_percentile" : 8367, "RequestSize_99th_percentile" : 8529, "RequestSize_99.9th_percentile" : 8674, "RequestSize_SizeRangeCount_10-100" : 42, "RequestSize_SizeRangeCount_100-1000" : 71, "sentBytes" : 5263, "QueueCallTime_num_ops" : 113, "QueueCallTime_min" : 0, "QueueCallTime_max" : 31, "QueueCallTime_mean" : 2, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 1, "QueueCallTime_90th_percentile" : 9, "QueueCallTime_95th_percentile" : 19, "QueueCallTime_98th_percentile" : 29, "QueueCallTime_99th_percentile" : 29, "QueueCallTime_99.9th_percentile" : 31, "QueueCallTime_TimeRangeCount_0-1" : 92, "QueueCallTime_TimeRangeCount_1-3" : 6, "QueueCallTime_TimeRangeCount_3-10" : 4, "QueueCallTime_TimeRangeCount_10-30" : 10, "QueueCallTime_TimeRangeCount_30-100" : 1, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:56841", "tag.serverName" : "ubuntu,39267,1517713227316", "tag.clusterId" : "990c040f-15f4-4bef-a30c-08f39bc273d8", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "regionCount" : 10, "storeCount" : 1, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 0, "memStoreSize" : 78, "storeFileSize" : 0, "maxStoreFileAge" : 0, "minStoreFileAge" : 0, "avgStoreFileAge" : 0, "numReferenceFiles" : 0, "regionServerStartTime" : 1517713227316, "averageRegionSize" : 78, "totalRequestCount" : 9, "totalRowActionRequestCount" : 4, "readRequestCount" : 2, "filteredReadRequestCount" : 0, "writeRequestCount" : 2, "rpcGetRequestCount" : 4, "rpcScanRequestCount" : 1, "rpcMultiRequestCount" : 0, "rpcMutateRequestCount" : 2, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 0, "staticIndexSize" : 0, "staticBloomSize" : 0, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1116962296, "blockCacheCount" : 0, "blockCacheSize" : 819720, "blockCacheHitCount" : 0, "blockCacheHitCountPrimary" : 0, "blockCacheMissCount" : 0, "blockCacheMissCountPrimary" : 0, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 0.0, "blockCacheExpressHitPercent" : 0.0, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 0, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 0, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 0, "l1CacheMissCount" : 0, "l1CacheHitRatio" : NaN, "l1CacheMissRatio" : NaN, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 0, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 0, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 1, "ScanTime_min" : 1, "ScanTime_max" : 1, "ScanTime_mean" : 1, "ScanTime_25th_percentile" : 1, "ScanTime_median" : 1, "ScanTime_75th_percentile" : 1, "ScanTime_90th_percentile" : 1, "ScanTime_95th_percentile" : 1, "ScanTime_98th_percentile" : 1, "ScanTime_99th_percentile" : 1, "ScanTime_99.9th_percentile" : 1, "ScanTime_TimeRangeCount_0-1" : 1, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 2, "Put_min" : 2, "Put_max" : 9, "Put_mean" : 5, "Put_25th_percentile" : 2, "Put_median" : 3, "Put_75th_percentile" : 9, "Put_90th_percentile" : 9, "Put_95th_percentile" : 9, "Put_98th_percentile" : 9, "Put_99th_percentile" : 9, "Put_99.9th_percentile" : 9, "Put_TimeRangeCount_1-3" : 1, "Put_TimeRangeCount_3-10" : 1, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 0, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 0, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 4, "Get_min" : 0, "Get_max" : 1, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 1, "Get_75th_percentile" : 1, "Get_90th_percentile" : 1, "Get_95th_percentile" : 1, "Get_98th_percentile" : 1, "Get_99th_percentile" : 1, "Get_99.9th_percentile" : 1, "Get_TimeRangeCount_0-1" : 4, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 0, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 0, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 1, "ScanSize_min" : 86, "ScanSize_max" : 86, "ScanSize_mean" : 86, "ScanSize_25th_percentile" : 86, "ScanSize_median" : 86, "ScanSize_75th_percentile" : 86, "ScanSize_90th_percentile" : 86, "ScanSize_95th_percentile" : 86, "ScanSize_98th_percentile" : 86, "ScanSize_99th_percentile" : 86, "ScanSize_99.9th_percentile" : 86, "ScanSize_SizeRangeCount_10-100" : 1, "slowGetCount" : 0, "flushedMemstoreBytes" : 0, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 0, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-02-04 11:00:40,018 ERROR [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=39063] master.MasterRpcServices(503): Region server ubuntu,39267,1517713227316 reported a fatal error: ***** ABORTING region server ubuntu,39267,1517713227316: Caught throwable while processing event M_RS_CLOSE_REGION ***** Cause: java.lang.RuntimeException: org.apache.hadoop.hbase.DroppedSnapshotException: region: tabletest,hhh,1517713232695.ad4f7bcab0a803ddae20b2866466d222. at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:117) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.hbase.DroppedSnapshotException: region: tabletest,hhh,1517713232695.ad4f7bcab0a803ddae20b2866466d222. at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2667) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2356) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) ... 4 more Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): Parent directory doesn't exist: /user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/WALs/ubuntu,39267,1517713227316 at org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyParentDir(FSDirectory.java:1776) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2533) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2465) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2349) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:624) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:398) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2213) at org.apache.hadoop.ipc.Client.call(Client.java:1476) at org.apache.hadoop.ipc.Client.call(Client.java:1413) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy28.create(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296) at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy29.create(Unknown Source) at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:372) at com.sun.proxy.$Proxy30.create(Unknown Source) at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.lambda$1(FanOutOneBlockAsyncDFSOutputHelper.java:519) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$FileCreator.create(FanOutOneBlockAsyncDFSOutputHelper.java:210) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createOutput(FanOutOneBlockAsyncDFSOutputHelper.java:757) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$4(FanOutOneBlockAsyncDFSOutputHelper.java:743) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$16.doCall(FanOutOneBlockAsyncDFSOutputHelper.java:851) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$16.doCall(FanOutOneBlockAsyncDFSOutputHelper.java:1) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createOutput(FanOutOneBlockAsyncDFSOutputHelper.java:859) at org.apache.hadoop.hbase.io.asyncfs.AsyncFSOutputHelper.createOutput(AsyncFSOutputHelper.java:51) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.initOutput(AsyncProtobufLogWriter.java:159) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:167) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:99) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:612) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:123) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:766) at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:169) ... 1 more 2018-02-04 11:00:40,019 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] handler.CloseRegionHandler(92): Processing close of hbase:namespace,,1517713230960.cf37e425ee5be11cb09b53c3671e1599. 2018-02-04 11:00:40,030 INFO [StoreCloserThread-tabletest,nnn,1517713232695.7429e32b4c4e0aadde8a8154cbd1f071.-1] regionserver.HStore(932): Closed family 2018-02-04 11:00:40,031 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/41f447d2065af74b219164f0a70d08b3/recovered.edits/12.seqid, newSeqId=12, maxSeqId=2 2018-02-04 11:00:40,039 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(1483): Closing hbase:namespace,,1517713230960.cf37e425ee5be11cb09b53c3671e1599.: disabling compactions & flushes 2018-02-04 11:00:40,040 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(1522): Updates disabled for region hbase:namespace,,1517713230960.cf37e425ee5be11cb09b53c3671e1599. 2018-02-04 11:00:40,041 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-1] helpers.MarkerIgnoringBase(159): ***** ABORTING region server ubuntu,39267,1517713227316: Unrecoverable exception while closing region hbase:namespace,,1517713230960.cf37e425ee5be11cb09b53c3671e1599., still finishing close ***** java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2369) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2354) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-02-04 11:00:40,041 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-1] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-02-04 11:00:40,044 INFO [StoreCloserThread-tabletest,qqq,1517713232695.6a7b42fe3d02ea7aca1d05fa666e87ac.-1] regionserver.HStore(932): Closed family 2018-02-04 11:00:40,045 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.DefaultStoreFlusher(85): Flushed, sequenceid=9, memsize=130.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/20457e81e436062b31548a08737ebba4/.tmp/family/0e78a66aac13416ca390923167054ee7 2018-02-04 11:00:40,065 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegionServer(2350): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 855638016, "init" : 197132288, "max" : 2794455040, "used" : 627199376 }, "NonHeapMemoryUsage" : { "committed" : 118317056, "init" : 2555904, "max" : -1, "used" : 115830320 }, "Verbose" : false, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : 1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 50164, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 113, "TotalCallTime_min" : 0, "TotalCallTime_max" : 120, "TotalCallTime_mean" : 12, "TotalCallTime_25th_percentile" : 1, "TotalCallTime_median" : 4, "TotalCallTime_75th_percentile" : 12, "TotalCallTime_90th_percentile" : 38, "TotalCallTime_95th_percentile" : 45, "TotalCallTime_98th_percentile" : 52, "TotalCallTime_99th_percentile" : 53, "TotalCallTime_99.9th_percentile" : 120, "TotalCallTime_TimeRangeCount_0-1" : 29, "TotalCallTime_TimeRangeCount_1-3" : 17, "TotalCallTime_TimeRangeCount_3-10" : 36, "TotalCallTime_TimeRangeCount_10-30" : 12, "TotalCallTime_TimeRangeCount_30-100" : 18, "TotalCallTime_TimeRangeCount_100-300" : 1, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 113, "ResponseSize_min" : 0, "ResponseSize_max" : 787, "ResponseSize_mean" : 37, "ResponseSize_25th_percentile" : 10, "ResponseSize_median" : 10, "ResponseSize_75th_percentile" : 51, "ResponseSize_90th_percentile" : 51, "ResponseSize_95th_percentile" : 57, "ResponseSize_98th_percentile" : 161, "ResponseSize_99th_percentile" : 486, "ResponseSize_99.9th_percentile" : 756, "ResponseSize_SizeRangeCount_0-10" : 73, "ResponseSize_SizeRangeCount_10-100" : 36, "ResponseSize_SizeRangeCount_100-1000" : 4, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 113, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 120, "ProcessCallTime_mean" : 9, "ProcessCallTime_25th_percentile" : 1, "ProcessCallTime_median" : 4, "ProcessCallTime_75th_percentile" : 11, "ProcessCallTime_90th_percentile" : 28, "ProcessCallTime_95th_percentile" : 38, "ProcessCallTime_98th_percentile" : 51, "ProcessCallTime_99th_percentile" : 53, "ProcessCallTime_99.9th_percentile" : 120, "ProcessCallTime_TimeRangeCount_0-1" : 36, "ProcessCallTime_TimeRangeCount_1-3" : 15, "ProcessCallTime_TimeRangeCount_3-10" : 33, "ProcessCallTime_TimeRangeCount_10-30" : 19, "ProcessCallTime_TimeRangeCount_30-100" : 9, "ProcessCallTime_TimeRangeCount_100-300" : 1, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 7, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 113, "RequestSize_min" : 70, "RequestSize_max" : 8691, "RequestSize_mean" : 424, "RequestSize_25th_percentile" : 81, "RequestSize_median" : 613, "RequestSize_75th_percentile" : 4652, "RequestSize_90th_percentile" : 7075, "RequestSize_95th_percentile" : 7883, "RequestSize_98th_percentile" : 8367, "RequestSize_99th_percentile" : 8529, "RequestSize_99.9th_percentile" : 8674, "RequestSize_SizeRangeCount_10-100" : 42, "RequestSize_SizeRangeCount_100-1000" : 71, "sentBytes" : 5263, "QueueCallTime_num_ops" : 113, "QueueCallTime_min" : 0, "QueueCallTime_max" : 31, "QueueCallTime_mean" : 2, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 1, "QueueCallTime_90th_percentile" : 9, "QueueCallTime_95th_percentile" : 19, "QueueCallTime_98th_percentile" : 29, "QueueCallTime_99th_percentile" : 29, "QueueCallTime_99.9th_percentile" : 31, "QueueCallTime_TimeRangeCount_0-1" : 92, "QueueCallTime_TimeRangeCount_1-3" : 6, "QueueCallTime_TimeRangeCount_3-10" : 4, "QueueCallTime_TimeRangeCount_10-30" : 10, "QueueCallTime_TimeRangeCount_30-100" : 1, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:56841", "tag.serverName" : "ubuntu,39267,1517713227316", "tag.clusterId" : "990c040f-15f4-4bef-a30c-08f39bc273d8", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "regionCount" : 10, "storeCount" : 1, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 0, "memStoreSize" : 78, "storeFileSize" : 0, "maxStoreFileAge" : 0, "minStoreFileAge" : 0, "avgStoreFileAge" : 0, "numReferenceFiles" : 0, "regionServerStartTime" : 1517713227316, "averageRegionSize" : 78, "totalRequestCount" : 9, "totalRowActionRequestCount" : 4, "readRequestCount" : 2, "filteredReadRequestCount" : 0, "writeRequestCount" : 2, "rpcGetRequestCount" : 4, "rpcScanRequestCount" : 1, "rpcMultiRequestCount" : 0, "rpcMutateRequestCount" : 2, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 0, "staticIndexSize" : 0, "staticBloomSize" : 0, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1116962296, "blockCacheCount" : 0, "blockCacheSize" : 819720, "blockCacheHitCount" : 0, "blockCacheHitCountPrimary" : 0, "blockCacheMissCount" : 0, "blockCacheMissCountPrimary" : 0, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 0.0, "blockCacheExpressHitPercent" : 0.0, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 0, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 0, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 0, "l1CacheMissCount" : 0, "l1CacheHitRatio" : NaN, "l1CacheMissRatio" : NaN, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 0, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 0, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 1, "ScanTime_min" : 1, "ScanTime_max" : 1, "ScanTime_mean" : 1, "ScanTime_25th_percentile" : 1, "ScanTime_median" : 1, "ScanTime_75th_percentile" : 1, "ScanTime_90th_percentile" : 1, "ScanTime_95th_percentile" : 1, "ScanTime_98th_percentile" : 1, "ScanTime_99th_percentile" : 1, "ScanTime_99.9th_percentile" : 1, "ScanTime_TimeRangeCount_0-1" : 1, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 2, "Put_min" : 2, "Put_max" : 9, "Put_mean" : 5, "Put_25th_percentile" : 2, "Put_median" : 3, "Put_75th_percentile" : 9, "Put_90th_percentile" : 9, "Put_95th_percentile" : 9, "Put_98th_percentile" : 9, "Put_99th_percentile" : 9, "Put_99.9th_percentile" : 9, "Put_TimeRangeCount_1-3" : 1, "Put_TimeRangeCount_3-10" : 1, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 0, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 0, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 4, "Get_min" : 0, "Get_max" : 1, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 1, "Get_75th_percentile" : 1, "Get_90th_percentile" : 1, "Get_95th_percentile" : 1, "Get_98th_percentile" : 1, "Get_99th_percentile" : 1, "Get_99.9th_percentile" : 1, "Get_TimeRangeCount_0-1" : 4, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 0, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 0, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 1, "ScanSize_min" : 86, "ScanSize_max" : 86, "ScanSize_mean" : 86, "ScanSize_25th_percentile" : 86, "ScanSize_median" : 86, "ScanSize_75th_percentile" : 86, "ScanSize_90th_percentile" : 86, "ScanSize_95th_percentile" : 86, "ScanSize_98th_percentile" : 86, "ScanSize_99th_percentile" : 86, "ScanSize_99.9th_percentile" : 86, "ScanSize_SizeRangeCount_10-100" : 1, "slowGetCount" : 0, "flushedMemstoreBytes" : 0, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 0, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-02-04 11:00:40,067 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(1635): Closed tabletest,ooo,1517713232695.41f447d2065af74b219164f0a70d08b3. 2018-02-04 11:00:40,067 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] handler.CloseRegionHandler(125): Closed tabletest,ooo,1517713232695.41f447d2065af74b219164f0a70d08b3. 2018-02-04 11:00:40,067 ERROR [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=39063] master.MasterRpcServices(503): Region server ubuntu,39267,1517713227316 reported a fatal error: ***** ABORTING region server ubuntu,39267,1517713227316: Unrecoverable exception while closing region hbase:namespace,,1517713230960.cf37e425ee5be11cb09b53c3671e1599., still finishing close ***** Cause: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2369) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2354) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-02-04 11:00:40,067 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] handler.CloseRegionHandler(92): Processing close of tabletest,lll,1517713232695.85d5a900ae52e80b00b89bb74335621c. 2018-02-04 11:00:40,068 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-1] executor.EventHandler(186): Caught throwable while processing event M_RS_CLOSE_REGION java.lang.RuntimeException: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:117) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2369) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2354) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) ... 4 more 2018-02-04 11:00:40,069 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-1] helpers.MarkerIgnoringBase(159): ***** ABORTING region server ubuntu,39267,1517713227316: Caught throwable while processing event M_RS_CLOSE_REGION ***** java.lang.RuntimeException: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:117) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2369) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2354) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) ... 4 more 2018-02-04 11:00:40,069 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-1] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-02-04 11:00:40,074 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegionServer(2350): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 855638016, "init" : 197132288, "max" : 2794455040, "used" : 628711488 }, "NonHeapMemoryUsage" : { "committed" : 118317056, "init" : 2555904, "max" : -1, "used" : 115910008 }, "Verbose" : false, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : 1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 50164, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 113, "TotalCallTime_min" : 0, "TotalCallTime_max" : 120, "TotalCallTime_mean" : 12, "TotalCallTime_25th_percentile" : 1, "TotalCallTime_median" : 4, "TotalCallTime_75th_percentile" : 12, "TotalCallTime_90th_percentile" : 38, "TotalCallTime_95th_percentile" : 45, "TotalCallTime_98th_percentile" : 52, "TotalCallTime_99th_percentile" : 53, "TotalCallTime_99.9th_percentile" : 120, "TotalCallTime_TimeRangeCount_0-1" : 29, "TotalCallTime_TimeRangeCount_1-3" : 17, "TotalCallTime_TimeRangeCount_3-10" : 36, "TotalCallTime_TimeRangeCount_10-30" : 12, "TotalCallTime_TimeRangeCount_30-100" : 18, "TotalCallTime_TimeRangeCount_100-300" : 1, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 113, "ResponseSize_min" : 0, "ResponseSize_max" : 787, "ResponseSize_mean" : 37, "ResponseSize_25th_percentile" : 10, "ResponseSize_median" : 10, "ResponseSize_75th_percentile" : 51, "ResponseSize_90th_percentile" : 51, "ResponseSize_95th_percentile" : 57, "ResponseSize_98th_percentile" : 161, "ResponseSize_99th_percentile" : 486, "ResponseSize_99.9th_percentile" : 756, "ResponseSize_SizeRangeCount_0-10" : 73, "ResponseSize_SizeRangeCount_10-100" : 36, "ResponseSize_SizeRangeCount_100-1000" : 4, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 113, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 120, "ProcessCallTime_mean" : 9, "ProcessCallTime_25th_percentile" : 1, "ProcessCallTime_median" : 4, "ProcessCallTime_75th_percentile" : 11, "ProcessCallTime_90th_percentile" : 28, "ProcessCallTime_95th_percentile" : 38, "ProcessCallTime_98th_percentile" : 51, "ProcessCallTime_99th_percentile" : 53, "ProcessCallTime_99.9th_percentile" : 120, "ProcessCallTime_TimeRangeCount_0-1" : 36, "ProcessCallTime_TimeRangeCount_1-3" : 15, "ProcessCallTime_TimeRangeCount_3-10" : 33, "ProcessCallTime_TimeRangeCount_10-30" : 19, "ProcessCallTime_TimeRangeCount_30-100" : 9, "ProcessCallTime_TimeRangeCount_100-300" : 1, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 7, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 113, "RequestSize_min" : 70, "RequestSize_max" : 8691, "RequestSize_mean" : 424, "RequestSize_25th_percentile" : 81, "RequestSize_median" : 613, "RequestSize_75th_percentile" : 4652, "RequestSize_90th_percentile" : 7075, "RequestSize_95th_percentile" : 7883, "RequestSize_98th_percentile" : 8367, "RequestSize_99th_percentile" : 8529, "RequestSize_99.9th_percentile" : 8674, "RequestSize_SizeRangeCount_10-100" : 42, "RequestSize_SizeRangeCount_100-1000" : 71, "sentBytes" : 5263, "QueueCallTime_num_ops" : 113, "QueueCallTime_min" : 0, "QueueCallTime_max" : 31, "QueueCallTime_mean" : 2, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 1, "QueueCallTime_90th_percentile" : 9, "QueueCallTime_95th_percentile" : 19, "QueueCallTime_98th_percentile" : 29, "QueueCallTime_99th_percentile" : 29, "QueueCallTime_99.9th_percentile" : 31, "QueueCallTime_TimeRangeCount_0-1" : 92, "QueueCallTime_TimeRangeCount_1-3" : 6, "QueueCallTime_TimeRangeCount_3-10" : 4, "QueueCallTime_TimeRangeCount_10-30" : 10, "QueueCallTime_TimeRangeCount_30-100" : 1, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:56841", "tag.serverName" : "ubuntu,39267,1517713227316", "tag.clusterId" : "990c040f-15f4-4bef-a30c-08f39bc273d8", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "regionCount" : 10, "storeCount" : 1, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 0, "memStoreSize" : 78, "storeFileSize" : 0, "maxStoreFileAge" : 0, "minStoreFileAge" : 0, "avgStoreFileAge" : 0, "numReferenceFiles" : 0, "regionServerStartTime" : 1517713227316, "averageRegionSize" : 78, "totalRequestCount" : 9, "totalRowActionRequestCount" : 4, "readRequestCount" : 2, "filteredReadRequestCount" : 0, "writeRequestCount" : 2, "rpcGetRequestCount" : 4, "rpcScanRequestCount" : 1, "rpcMultiRequestCount" : 0, "rpcMutateRequestCount" : 2, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 0, "staticIndexSize" : 0, "staticBloomSize" : 0, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1116962296, "blockCacheCount" : 0, "blockCacheSize" : 819720, "blockCacheHitCount" : 0, "blockCacheHitCountPrimary" : 0, "blockCacheMissCount" : 0, "blockCacheMissCountPrimary" : 0, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 0.0, "blockCacheExpressHitPercent" : 0.0, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 0, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 0, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 0, "l1CacheMissCount" : 0, "l1CacheHitRatio" : NaN, "l1CacheMissRatio" : NaN, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 0, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 0, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 1, "ScanTime_min" : 1, "ScanTime_max" : 1, "ScanTime_mean" : 1, "ScanTime_25th_percentile" : 1, "ScanTime_median" : 1, "ScanTime_75th_percentile" : 1, "ScanTime_90th_percentile" : 1, "ScanTime_95th_percentile" : 1, "ScanTime_98th_percentile" : 1, "ScanTime_99th_percentile" : 1, "ScanTime_99.9th_percentile" : 1, "ScanTime_TimeRangeCount_0-1" : 1, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 2, "Put_min" : 2, "Put_max" : 9, "Put_mean" : 5, "Put_25th_percentile" : 2, "Put_median" : 3, "Put_75th_percentile" : 9, "Put_90th_percentile" : 9, "Put_95th_percentile" : 9, "Put_98th_percentile" : 9, "Put_99th_percentile" : 9, "Put_99.9th_percentile" : 9, "Put_TimeRangeCount_1-3" : 1, "Put_TimeRangeCount_3-10" : 1, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 0, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 0, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 4, "Get_min" : 0, "Get_max" : 1, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 1, "Get_75th_percentile" : 1, "Get_90th_percentile" : 1, "Get_95th_percentile" : 1, "Get_98th_percentile" : 1, "Get_99th_percentile" : 1, "Get_99.9th_percentile" : 1, "Get_TimeRangeCount_0-1" : 4, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 0, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 0, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 1, "ScanSize_min" : 86, "ScanSize_max" : 86, "ScanSize_mean" : 86, "ScanSize_25th_percentile" : 86, "ScanSize_median" : 86, "ScanSize_75th_percentile" : 86, "ScanSize_90th_percentile" : 86, "ScanSize_95th_percentile" : 86, "ScanSize_98th_percentile" : 86, "ScanSize_99th_percentile" : 86, "ScanSize_99.9th_percentile" : 86, "ScanSize_SizeRangeCount_10-100" : 1, "slowGetCount" : 0, "flushedMemstoreBytes" : 0, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 0, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-02-04 11:00:40,076 ERROR [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=39063] master.MasterRpcServices(503): Region server ubuntu,39267,1517713227316 reported a fatal error: ***** ABORTING region server ubuntu,39267,1517713227316: Caught throwable while processing event M_RS_CLOSE_REGION ***** Cause: java.lang.RuntimeException: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:117) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2369) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2354) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) ... 4 more 2018-02-04 11:00:40,089 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] handler.CloseRegionHandler(92): Processing close of tabletest,iii,1517713232695.f1f365dfd0648c0fb5d9848c1db452cc. 2018-02-04 11:00:40,103 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(1483): Closing tabletest,lll,1517713232695.85d5a900ae52e80b00b89bb74335621c.: disabling compactions & flushes 2018-02-04 11:00:40,104 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(1522): Updates disabled for region tabletest,lll,1517713232695.85d5a900ae52e80b00b89bb74335621c. 2018-02-04 11:00:40,104 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(2518): Flushing 1/1 column families, memstore=130.44 KB 2018-02-04 11:00:40,105 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.CompactingMemStore(205): FLUSHING TO DISK: region tabletest,lll,1517713232695.85d5a900ae52e80b00b89bb74335621c.store: family 2018-02-04 11:00:40,105 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.CompactionPipeline(128): Swapping pipeline suffix; before=1, new segement=null 2018-02-04 11:00:40,114 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(1483): Closing tabletest,iii,1517713232695.f1f365dfd0648c0fb5d9848c1db452cc.: disabling compactions & flushes 2018-02-04 11:00:40,114 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(1522): Updates disabled for region tabletest,iii,1517713232695.f1f365dfd0648c0fb5d9848c1db452cc. 2018-02-04 11:00:40,115 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-1] helpers.MarkerIgnoringBase(159): ***** ABORTING region server ubuntu,39267,1517713227316: Unrecoverable exception while closing region tabletest,iii,1517713232695.f1f365dfd0648c0fb5d9848c1db452cc., still finishing close ***** java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2369) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2354) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-02-04 11:00:40,115 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-1] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-02-04 11:00:40,123 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegionServer(2350): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 855638016, "init" : 197132288, "max" : 2794455040, "used" : 630359720 }, "NonHeapMemoryUsage" : { "committed" : 118317056, "init" : 2555904, "max" : -1, "used" : 115990304 }, "Verbose" : false, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : 1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 50164, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 113, "TotalCallTime_min" : 0, "TotalCallTime_max" : 120, "TotalCallTime_mean" : 12, "TotalCallTime_25th_percentile" : 1, "TotalCallTime_median" : 4, "TotalCallTime_75th_percentile" : 12, "TotalCallTime_90th_percentile" : 38, "TotalCallTime_95th_percentile" : 45, "TotalCallTime_98th_percentile" : 52, "TotalCallTime_99th_percentile" : 53, "TotalCallTime_99.9th_percentile" : 120, "TotalCallTime_TimeRangeCount_0-1" : 29, "TotalCallTime_TimeRangeCount_1-3" : 17, "TotalCallTime_TimeRangeCount_3-10" : 36, "TotalCallTime_TimeRangeCount_10-30" : 12, "TotalCallTime_TimeRangeCount_30-100" : 18, "TotalCallTime_TimeRangeCount_100-300" : 1, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 113, "ResponseSize_min" : 0, "ResponseSize_max" : 787, "ResponseSize_mean" : 37, "ResponseSize_25th_percentile" : 10, "ResponseSize_median" : 10, "ResponseSize_75th_percentile" : 51, "ResponseSize_90th_percentile" : 51, "ResponseSize_95th_percentile" : 57, "ResponseSize_98th_percentile" : 161, "ResponseSize_99th_percentile" : 486, "ResponseSize_99.9th_percentile" : 756, "ResponseSize_SizeRangeCount_0-10" : 73, "ResponseSize_SizeRangeCount_10-100" : 36, "ResponseSize_SizeRangeCount_100-1000" : 4, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 113, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 120, "ProcessCallTime_mean" : 9, "ProcessCallTime_25th_percentile" : 1, "ProcessCallTime_median" : 4, "ProcessCallTime_75th_percentile" : 11, "ProcessCallTime_90th_percentile" : 28, "ProcessCallTime_95th_percentile" : 38, "ProcessCallTime_98th_percentile" : 51, "ProcessCallTime_99th_percentile" : 53, "ProcessCallTime_99.9th_percentile" : 120, "ProcessCallTime_TimeRangeCount_0-1" : 36, "ProcessCallTime_TimeRangeCount_1-3" : 15, "ProcessCallTime_TimeRangeCount_3-10" : 33, "ProcessCallTime_TimeRangeCount_10-30" : 19, "ProcessCallTime_TimeRangeCount_30-100" : 9, "ProcessCallTime_TimeRangeCount_100-300" : 1, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 7, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 113, "RequestSize_min" : 70, "RequestSize_max" : 8691, "RequestSize_mean" : 424, "RequestSize_25th_percentile" : 81, "RequestSize_median" : 613, "RequestSize_75th_percentile" : 4652, "RequestSize_90th_percentile" : 7075, "RequestSize_95th_percentile" : 7883, "RequestSize_98th_percentile" : 8367, "RequestSize_99th_percentile" : 8529, "RequestSize_99.9th_percentile" : 8674, "RequestSize_SizeRangeCount_10-100" : 42, "RequestSize_SizeRangeCount_100-1000" : 71, "sentBytes" : 5263, "QueueCallTime_num_ops" : 113, "QueueCallTime_min" : 0, "QueueCallTime_max" : 31, "QueueCallTime_mean" : 2, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 1, "QueueCallTime_90th_percentile" : 9, "QueueCallTime_95th_percentile" : 19, "QueueCallTime_98th_percentile" : 29, "QueueCallTime_99th_percentile" : 29, "QueueCallTime_99.9th_percentile" : 31, "QueueCallTime_TimeRangeCount_0-1" : 92, "QueueCallTime_TimeRangeCount_1-3" : 6, "QueueCallTime_TimeRangeCount_3-10" : 4, "QueueCallTime_TimeRangeCount_10-30" : 10, "QueueCallTime_TimeRangeCount_30-100" : 1, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:56841", "tag.serverName" : "ubuntu,39267,1517713227316", "tag.clusterId" : "990c040f-15f4-4bef-a30c-08f39bc273d8", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "regionCount" : 10, "storeCount" : 1, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 0, "memStoreSize" : 78, "storeFileSize" : 0, "maxStoreFileAge" : 0, "minStoreFileAge" : 0, "avgStoreFileAge" : 0, "numReferenceFiles" : 0, "regionServerStartTime" : 1517713227316, "averageRegionSize" : 78, "totalRequestCount" : 9, "totalRowActionRequestCount" : 4, "readRequestCount" : 2, "filteredReadRequestCount" : 0, "writeRequestCount" : 2, "rpcGetRequestCount" : 4, "rpcScanRequestCount" : 1, "rpcMultiRequestCount" : 0, "rpcMutateRequestCount" : 2, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 0, "staticIndexSize" : 0, "staticBloomSize" : 0, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1116962296, "blockCacheCount" : 0, "blockCacheSize" : 819720, "blockCacheHitCount" : 0, "blockCacheHitCountPrimary" : 0, "blockCacheMissCount" : 0, "blockCacheMissCountPrimary" : 0, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 0.0, "blockCacheExpressHitPercent" : 0.0, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 0, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 0, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 0, "l1CacheMissCount" : 0, "l1CacheHitRatio" : NaN, "l1CacheMissRatio" : NaN, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 0, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 0, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 1, "ScanTime_min" : 1, "ScanTime_max" : 1, "ScanTime_mean" : 1, "ScanTime_25th_percentile" : 1, "ScanTime_median" : 1, "ScanTime_75th_percentile" : 1, "ScanTime_90th_percentile" : 1, "ScanTime_95th_percentile" : 1, "ScanTime_98th_percentile" : 1, "ScanTime_99th_percentile" : 1, "ScanTime_99.9th_percentile" : 1, "ScanTime_TimeRangeCount_0-1" : 1, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 2, "Put_min" : 2, "Put_max" : 9, "Put_mean" : 5, "Put_25th_percentile" : 2, "Put_median" : 3, "Put_75th_percentile" : 9, "Put_90th_percentile" : 9, "Put_95th_percentile" : 9, "Put_98th_percentile" : 9, "Put_99th_percentile" : 9, "Put_99.9th_percentile" : 9, "Put_TimeRangeCount_1-3" : 1, "Put_TimeRangeCount_3-10" : 1, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 0, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 0, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 4, "Get_min" : 0, "Get_max" : 1, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 1, "Get_75th_percentile" : 1, "Get_90th_percentile" : 1, "Get_95th_percentile" : 1, "Get_98th_percentile" : 1, "Get_99th_percentile" : 1, "Get_99.9th_percentile" : 1, "Get_TimeRangeCount_0-1" : 4, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 0, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 0, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 1, "ScanSize_min" : 86, "ScanSize_max" : 86, "ScanSize_mean" : 86, "ScanSize_25th_percentile" : 86, "ScanSize_median" : 86, "ScanSize_75th_percentile" : 86, "ScanSize_90th_percentile" : 86, "ScanSize_95th_percentile" : 86, "ScanSize_98th_percentile" : 86, "ScanSize_99th_percentile" : 86, "ScanSize_99.9th_percentile" : 86, "ScanSize_SizeRangeCount_10-100" : 1, "slowGetCount" : 0, "flushedMemstoreBytes" : 0, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 0, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-02-04 11:00:40,139 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] wal.WALSplitter(675): Wrote file=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/6a7b42fe3d02ea7aca1d05fa666e87ac/recovered.edits/12.seqid, newSeqId=12, maxSeqId=2 2018-02-04 11:00:40,157 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] wal.WALSplitter(675): Wrote file=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/7429e32b4c4e0aadde8a8154cbd1f071/recovered.edits/12.seqid, newSeqId=12, maxSeqId=2 2018-02-04 11:00:40,179 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegionFileSystem(463): Committing store file hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/20457e81e436062b31548a08737ebba4/.tmp/family/0e78a66aac13416ca390923167054ee7 as hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/20457e81e436062b31548a08737ebba4/family/0e78a66aac13416ca390923167054ee7 2018-02-04 11:00:40,185 ERROR [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=39063] master.MasterRpcServices(503): Region server ubuntu,39267,1517713227316 reported a fatal error: ***** ABORTING region server ubuntu,39267,1517713227316: Unrecoverable exception while closing region tabletest,iii,1517713232695.f1f365dfd0648c0fb5d9848c1db452cc., still finishing close ***** Cause: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2369) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2354) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-02-04 11:00:40,186 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(443): regionserver:39267-0x1615ec1f6c10001, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Expired, path=null 2018-02-04 11:00:40,186 ERROR [Time-limited test-EventThread] helpers.MarkerIgnoringBase(159): ***** ABORTING region server ubuntu,39267,1517713227316: regionserver:39267-0x1615ec1f6c10001, quorum=localhost:56841, baseZNode=/hbase regionserver:39267-0x1615ec1f6c10001 received expired from ZooKeeper, aborting ***** org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired at org.apache.hadoop.hbase.zookeeper.ZKWatcher.connectionEvent(ZKWatcher.java:520) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:452) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505) 2018-02-04 11:00:40,187 ERROR [Time-limited test-EventThread] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-02-04 11:00:40,192 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-1] executor.EventHandler(186): Caught throwable while processing event M_RS_CLOSE_REGION java.lang.RuntimeException: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:117) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2369) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2354) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) ... 4 more 2018-02-04 11:00:40,192 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-1] helpers.MarkerIgnoringBase(159): ***** ABORTING region server ubuntu,39267,1517713227316: Caught throwable while processing event M_RS_CLOSE_REGION ***** java.lang.RuntimeException: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:117) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2369) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2354) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) ... 4 more 2018-02-04 11:00:40,193 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-1] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-02-04 11:00:40,204 INFO [Time-limited test-EventThread] regionserver.HRegionServer(2350): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 855638016, "init" : 197132288, "max" : 2794455040, "used" : 632484376 }, "NonHeapMemoryUsage" : { "committed" : 118317056, "init" : 2555904, "max" : -1, "used" : 116031112 }, "Verbose" : false, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : 1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 50164, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 113, "TotalCallTime_min" : 0, "TotalCallTime_max" : 120, "TotalCallTime_mean" : 12, "TotalCallTime_25th_percentile" : 1, "TotalCallTime_median" : 4, "TotalCallTime_75th_percentile" : 12, "TotalCallTime_90th_percentile" : 38, "TotalCallTime_95th_percentile" : 45, "TotalCallTime_98th_percentile" : 52, "TotalCallTime_99th_percentile" : 53, "TotalCallTime_99.9th_percentile" : 120, "TotalCallTime_TimeRangeCount_0-1" : 29, "TotalCallTime_TimeRangeCount_1-3" : 17, "TotalCallTime_TimeRangeCount_3-10" : 36, "TotalCallTime_TimeRangeCount_10-30" : 12, "TotalCallTime_TimeRangeCount_30-100" : 18, "TotalCallTime_TimeRangeCount_100-300" : 1, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 113, "ResponseSize_min" : 0, "ResponseSize_max" : 787, "ResponseSize_mean" : 37, "ResponseSize_25th_percentile" : 10, "ResponseSize_median" : 10, "ResponseSize_75th_percentile" : 51, "ResponseSize_90th_percentile" : 51, "ResponseSize_95th_percentile" : 57, "ResponseSize_98th_percentile" : 161, "ResponseSize_99th_percentile" : 486, "ResponseSize_99.9th_percentile" : 756, "ResponseSize_SizeRangeCount_0-10" : 73, "ResponseSize_SizeRangeCount_10-100" : 36, "ResponseSize_SizeRangeCount_100-1000" : 4, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 113, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 120, "ProcessCallTime_mean" : 9, "ProcessCallTime_25th_percentile" : 1, "ProcessCallTime_median" : 4, "ProcessCallTime_75th_percentile" : 11, "ProcessCallTime_90th_percentile" : 28, "ProcessCallTime_95th_percentile" : 38, "ProcessCallTime_98th_percentile" : 51, "ProcessCallTime_99th_percentile" : 53, "ProcessCallTime_99.9th_percentile" : 120, "ProcessCallTime_TimeRangeCount_0-1" : 36, "ProcessCallTime_TimeRangeCount_1-3" : 15, "ProcessCallTime_TimeRangeCount_3-10" : 33, "ProcessCallTime_TimeRangeCount_10-30" : 19, "ProcessCallTime_TimeRangeCount_30-100" : 9, "ProcessCallTime_TimeRangeCount_100-300" : 1, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 7, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 113, "RequestSize_min" : 70, "RequestSize_max" : 8691, "RequestSize_mean" : 424, "RequestSize_25th_percentile" : 81, "RequestSize_median" : 613, "RequestSize_75th_percentile" : 4652, "RequestSize_90th_percentile" : 7075, "RequestSize_95th_percentile" : 7883, "RequestSize_98th_percentile" : 8367, "RequestSize_99th_percentile" : 8529, "RequestSize_99.9th_percentile" : 8674, "RequestSize_SizeRangeCount_10-100" : 42, "RequestSize_SizeRangeCount_100-1000" : 71, "sentBytes" : 5263, "QueueCallTime_num_ops" : 113, "QueueCallTime_min" : 0, "QueueCallTime_max" : 31, "QueueCallTime_mean" : 2, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 1, "QueueCallTime_90th_percentile" : 9, "QueueCallTime_95th_percentile" : 19, "QueueCallTime_98th_percentile" : 29, "QueueCallTime_99th_percentile" : 29, "QueueCallTime_99.9th_percentile" : 31, "QueueCallTime_TimeRangeCount_0-1" : 92, "QueueCallTime_TimeRangeCount_1-3" : 6, "QueueCallTime_TimeRangeCount_3-10" : 4, "QueueCallTime_TimeRangeCount_10-30" : 10, "QueueCallTime_TimeRangeCount_30-100" : 1, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:56841", "tag.serverName" : "ubuntu,39267,1517713227316", "tag.clusterId" : "990c040f-15f4-4bef-a30c-08f39bc273d8", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "regionCount" : 10, "storeCount" : 1, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 0, "memStoreSize" : 78, "storeFileSize" : 0, "maxStoreFileAge" : 0, "minStoreFileAge" : 0, "avgStoreFileAge" : 0, "numReferenceFiles" : 0, "regionServerStartTime" : 1517713227316, "averageRegionSize" : 78, "totalRequestCount" : 9, "totalRowActionRequestCount" : 4, "readRequestCount" : 2, "filteredReadRequestCount" : 0, "writeRequestCount" : 2, "rpcGetRequestCount" : 4, "rpcScanRequestCount" : 1, "rpcMultiRequestCount" : 0, "rpcMutateRequestCount" : 2, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 0, "staticIndexSize" : 0, "staticBloomSize" : 0, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1116962296, "blockCacheCount" : 0, "blockCacheSize" : 819720, "blockCacheHitCount" : 0, "blockCacheHitCountPrimary" : 0, "blockCacheMissCount" : 0, "blockCacheMissCountPrimary" : 0, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 0.0, "blockCacheExpressHitPercent" : 0.0, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 0, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 0, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 0, "l1CacheMissCount" : 0, "l1CacheHitRatio" : NaN, "l1CacheMissRatio" : NaN, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 0, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 0, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 1, "ScanTime_min" : 1, "ScanTime_max" : 1, "ScanTime_mean" : 1, "ScanTime_25th_percentile" : 1, "ScanTime_median" : 1, "ScanTime_75th_percentile" : 1, "ScanTime_90th_percentile" : 1, "ScanTime_95th_percentile" : 1, "ScanTime_98th_percentile" : 1, "ScanTime_99th_percentile" : 1, "ScanTime_99.9th_percentile" : 1, "ScanTime_TimeRangeCount_0-1" : 1, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 2, "Put_min" : 2, "Put_max" : 9, "Put_mean" : 5, "Put_25th_percentile" : 2, "Put_median" : 3, "Put_75th_percentile" : 9, "Put_90th_percentile" : 9, "Put_95th_percentile" : 9, "Put_98th_percentile" : 9, "Put_99th_percentile" : 9, "Put_99.9th_percentile" : 9, "Put_TimeRangeCount_1-3" : 1, "Put_TimeRangeCount_3-10" : 1, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 0, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 0, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 4, "Get_min" : 0, "Get_max" : 1, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 1, "Get_75th_percentile" : 1, "Get_90th_percentile" : 1, "Get_95th_percentile" : 1, "Get_98th_percentile" : 1, "Get_99th_percentile" : 1, "Get_99.9th_percentile" : 1, "Get_TimeRangeCount_0-1" : 4, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 0, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 0, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 1, "ScanSize_min" : 86, "ScanSize_max" : 86, "ScanSize_mean" : 86, "ScanSize_25th_percentile" : 86, "ScanSize_median" : 86, "ScanSize_75th_percentile" : 86, "ScanSize_90th_percentile" : 86, "ScanSize_95th_percentile" : 86, "ScanSize_98th_percentile" : 86, "ScanSize_99th_percentile" : 86, "ScanSize_99.9th_percentile" : 86, "ScanSize_SizeRangeCount_10-100" : 1, "slowGetCount" : 0, "flushedMemstoreBytes" : 0, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 0, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-02-04 11:00:40,206 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HStore(1030): Added hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/20457e81e436062b31548a08737ebba4/family/0e78a66aac13416ca390923167054ee7, entries=703, sequenceid=9, filesize=32.5 K 2018-02-04 11:00:40,206 ERROR [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=39063] master.MasterRpcServices(503): Region server ubuntu,39267,1517713227316 reported a fatal error: ***** ABORTING region server ubuntu,39267,1517713227316: regionserver:39267-0x1615ec1f6c10001, quorum=localhost:56841, baseZNode=/hbase regionserver:39267-0x1615ec1f6c10001 received expired from ZooKeeper, aborting ***** Cause: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired at org.apache.hadoop.hbase.zookeeper.ZKWatcher.connectionEvent(ZKWatcher.java:520) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:452) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505) 2018-02-04 11:00:40,207 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegionServer(2350): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 855638016, "init" : 197132288, "max" : 2794455040, "used" : 633153832 }, "NonHeapMemoryUsage" : { "committed" : 118317056, "init" : 2555904, "max" : -1, "used" : 116080976 }, "Verbose" : false, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : 1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 50164, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 113, "TotalCallTime_min" : 0, "TotalCallTime_max" : 120, "TotalCallTime_mean" : 12, "TotalCallTime_25th_percentile" : 1, "TotalCallTime_median" : 4, "TotalCallTime_75th_percentile" : 12, "TotalCallTime_90th_percentile" : 38, "TotalCallTime_95th_percentile" : 45, "TotalCallTime_98th_percentile" : 52, "TotalCallTime_99th_percentile" : 53, "TotalCallTime_99.9th_percentile" : 120, "TotalCallTime_TimeRangeCount_0-1" : 29, "TotalCallTime_TimeRangeCount_1-3" : 17, "TotalCallTime_TimeRangeCount_3-10" : 36, "TotalCallTime_TimeRangeCount_10-30" : 12, "TotalCallTime_TimeRangeCount_30-100" : 18, "TotalCallTime_TimeRangeCount_100-300" : 1, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 113, "ResponseSize_min" : 0, "ResponseSize_max" : 787, "ResponseSize_mean" : 37, "ResponseSize_25th_percentile" : 10, "ResponseSize_median" : 10, "ResponseSize_75th_percentile" : 51, "ResponseSize_90th_percentile" : 51, "ResponseSize_95th_percentile" : 57, "ResponseSize_98th_percentile" : 161, "ResponseSize_99th_percentile" : 486, "ResponseSize_99.9th_percentile" : 756, "ResponseSize_SizeRangeCount_0-10" : 73, "ResponseSize_SizeRangeCount_10-100" : 36, "ResponseSize_SizeRangeCount_100-1000" : 4, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 113, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 120, "ProcessCallTime_mean" : 9, "ProcessCallTime_25th_percentile" : 1, "ProcessCallTime_median" : 4, "ProcessCallTime_75th_percentile" : 11, "ProcessCallTime_90th_percentile" : 28, "ProcessCallTime_95th_percentile" : 38, "ProcessCallTime_98th_percentile" : 51, "ProcessCallTime_99th_percentile" : 53, "ProcessCallTime_99.9th_percentile" : 120, "ProcessCallTime_TimeRangeCount_0-1" : 36, "ProcessCallTime_TimeRangeCount_1-3" : 15, "ProcessCallTime_TimeRangeCount_3-10" : 33, "ProcessCallTime_TimeRangeCount_10-30" : 19, "ProcessCallTime_TimeRangeCount_30-100" : 9, "ProcessCallTime_TimeRangeCount_100-300" : 1, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 7, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 113, "RequestSize_min" : 70, "RequestSize_max" : 8691, "RequestSize_mean" : 424, "RequestSize_25th_percentile" : 81, "RequestSize_median" : 613, "RequestSize_75th_percentile" : 4652, "RequestSize_90th_percentile" : 7075, "RequestSize_95th_percentile" : 7883, "RequestSize_98th_percentile" : 8367, "RequestSize_99th_percentile" : 8529, "RequestSize_99.9th_percentile" : 8674, "RequestSize_SizeRangeCount_10-100" : 42, "RequestSize_SizeRangeCount_100-1000" : 71, "sentBytes" : 5263, "QueueCallTime_num_ops" : 113, "QueueCallTime_min" : 0, "QueueCallTime_max" : 31, "QueueCallTime_mean" : 2, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 1, "QueueCallTime_90th_percentile" : 9, "QueueCallTime_95th_percentile" : 19, "QueueCallTime_98th_percentile" : 29, "QueueCallTime_99th_percentile" : 29, "QueueCallTime_99.9th_percentile" : 31, "QueueCallTime_TimeRangeCount_0-1" : 92, "QueueCallTime_TimeRangeCount_1-3" : 6, "QueueCallTime_TimeRangeCount_3-10" : 4, "QueueCallTime_TimeRangeCount_10-30" : 10, "QueueCallTime_TimeRangeCount_30-100" : 1, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:56841", "tag.serverName" : "ubuntu,39267,1517713227316", "tag.clusterId" : "990c040f-15f4-4bef-a30c-08f39bc273d8", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "regionCount" : 10, "storeCount" : 1, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 0, "memStoreSize" : 78, "storeFileSize" : 0, "maxStoreFileAge" : 0, "minStoreFileAge" : 0, "avgStoreFileAge" : 0, "numReferenceFiles" : 0, "regionServerStartTime" : 1517713227316, "averageRegionSize" : 78, "totalRequestCount" : 9, "totalRowActionRequestCount" : 4, "readRequestCount" : 2, "filteredReadRequestCount" : 0, "writeRequestCount" : 2, "rpcGetRequestCount" : 4, "rpcScanRequestCount" : 1, "rpcMultiRequestCount" : 0, "rpcMutateRequestCount" : 2, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 0, "staticIndexSize" : 0, "staticBloomSize" : 0, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1116962296, "blockCacheCount" : 0, "blockCacheSize" : 819720, "blockCacheHitCount" : 0, "blockCacheHitCountPrimary" : 0, "blockCacheMissCount" : 0, "blockCacheMissCountPrimary" : 0, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 0.0, "blockCacheExpressHitPercent" : 0.0, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 0, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 0, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 0, "l1CacheMissCount" : 0, "l1CacheHitRatio" : NaN, "l1CacheMissRatio" : NaN, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 0, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 0, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 1, "ScanTime_min" : 1, "ScanTime_max" : 1, "ScanTime_mean" : 1, "ScanTime_25th_percentile" : 1, "ScanTime_median" : 1, "ScanTime_75th_percentile" : 1, "ScanTime_90th_percentile" : 1, "ScanTime_95th_percentile" : 1, "ScanTime_98th_percentile" : 1, "ScanTime_99th_percentile" : 1, "ScanTime_99.9th_percentile" : 1, "ScanTime_TimeRangeCount_0-1" : 1, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 2, "Put_min" : 2, "Put_max" : 9, "Put_mean" : 5, "Put_25th_percentile" : 2, "Put_median" : 3, "Put_75th_percentile" : 9, "Put_90th_percentile" : 9, "Put_95th_percentile" : 9, "Put_98th_percentile" : 9, "Put_99th_percentile" : 9, "Put_99.9th_percentile" : 9, "Put_TimeRangeCount_1-3" : 1, "Put_TimeRangeCount_3-10" : 1, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 0, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 0, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 4, "Get_min" : 0, "Get_max" : 1, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 1, "Get_75th_percentile" : 1, "Get_90th_percentile" : 1, "Get_95th_percentile" : 1, "Get_98th_percentile" : 1, "Get_99th_percentile" : 1, "Get_99.9th_percentile" : 1, "Get_TimeRangeCount_0-1" : 4, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 0, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 0, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 1, "ScanSize_min" : 86, "ScanSize_max" : 86, "ScanSize_mean" : 86, "ScanSize_25th_percentile" : 86, "ScanSize_median" : 86, "ScanSize_75th_percentile" : 86, "ScanSize_90th_percentile" : 86, "ScanSize_95th_percentile" : 86, "ScanSize_98th_percentile" : 86, "ScanSize_99th_percentile" : 86, "ScanSize_99.9th_percentile" : 86, "ScanSize_SizeRangeCount_10-100" : 1, "slowGetCount" : 0, "flushedMemstoreBytes" : 0, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 0, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-02-04 11:00:40,209 ERROR [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=39063] master.MasterRpcServices(503): Region server ubuntu,39267,1517713227316 reported a fatal error: ***** ABORTING region server ubuntu,39267,1517713227316: Caught throwable while processing event M_RS_CLOSE_REGION ***** Cause: java.lang.RuntimeException: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:117) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2369) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2354) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) ... 4 more 2018-02-04 11:00:40,211 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] handler.CloseRegionHandler(92): Processing close of tabletest,sss,1517713232695.e063e6b2c2640eb6eafc8737a71fb5f5. 2018-02-04 11:00:40,212 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(1483): Closing tabletest,sss,1517713232695.e063e6b2c2640eb6eafc8737a71fb5f5.: disabling compactions & flushes 2018-02-04 11:00:40,212 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(1522): Updates disabled for region tabletest,sss,1517713232695.e063e6b2c2640eb6eafc8737a71fb5f5. 2018-02-04 11:00:40,212 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-1] helpers.MarkerIgnoringBase(159): ***** ABORTING region server ubuntu,39267,1517713227316: Unrecoverable exception while closing region tabletest,sss,1517713232695.e063e6b2c2640eb6eafc8737a71fb5f5., still finishing close ***** java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2369) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2354) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-02-04 11:00:40,213 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-1] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-02-04 11:00:40,217 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegionServer(2350): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 855638016, "init" : 197132288, "max" : 2794455040, "used" : 634819080 }, "NonHeapMemoryUsage" : { "committed" : 118317056, "init" : 2555904, "max" : -1, "used" : 116135368 }, "Verbose" : false, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : 1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 50164, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 113, "TotalCallTime_min" : 0, "TotalCallTime_max" : 120, "TotalCallTime_mean" : 12, "TotalCallTime_25th_percentile" : 1, "TotalCallTime_median" : 4, "TotalCallTime_75th_percentile" : 12, "TotalCallTime_90th_percentile" : 38, "TotalCallTime_95th_percentile" : 45, "TotalCallTime_98th_percentile" : 52, "TotalCallTime_99th_percentile" : 53, "TotalCallTime_99.9th_percentile" : 120, "TotalCallTime_TimeRangeCount_0-1" : 29, "TotalCallTime_TimeRangeCount_1-3" : 17, "TotalCallTime_TimeRangeCount_3-10" : 36, "TotalCallTime_TimeRangeCount_10-30" : 12, "TotalCallTime_TimeRangeCount_30-100" : 18, "TotalCallTime_TimeRangeCount_100-300" : 1, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 113, "ResponseSize_min" : 0, "ResponseSize_max" : 787, "ResponseSize_mean" : 37, "ResponseSize_25th_percentile" : 10, "ResponseSize_median" : 10, "ResponseSize_75th_percentile" : 51, "ResponseSize_90th_percentile" : 51, "ResponseSize_95th_percentile" : 57, "ResponseSize_98th_percentile" : 161, "ResponseSize_99th_percentile" : 486, "ResponseSize_99.9th_percentile" : 756, "ResponseSize_SizeRangeCount_0-10" : 73, "ResponseSize_SizeRangeCount_10-100" : 36, "ResponseSize_SizeRangeCount_100-1000" : 4, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 113, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 120, "ProcessCallTime_mean" : 9, "ProcessCallTime_25th_percentile" : 1, "ProcessCallTime_median" : 4, "ProcessCallTime_75th_percentile" : 11, "ProcessCallTime_90th_percentile" : 28, "ProcessCallTime_95th_percentile" : 38, "ProcessCallTime_98th_percentile" : 51, "ProcessCallTime_99th_percentile" : 53, "ProcessCallTime_99.9th_percentile" : 120, "ProcessCallTime_TimeRangeCount_0-1" : 36, "ProcessCallTime_TimeRangeCount_1-3" : 15, "ProcessCallTime_TimeRangeCount_3-10" : 33, "ProcessCallTime_TimeRangeCount_10-30" : 19, "ProcessCallTime_TimeRangeCount_30-100" : 9, "ProcessCallTime_TimeRangeCount_100-300" : 1, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 7, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 113, "RequestSize_min" : 70, "RequestSize_max" : 8691, "RequestSize_mean" : 424, "RequestSize_25th_percentile" : 81, "RequestSize_median" : 613, "RequestSize_75th_percentile" : 4652, "RequestSize_90th_percentile" : 7075, "RequestSize_95th_percentile" : 7883, "RequestSize_98th_percentile" : 8367, "RequestSize_99th_percentile" : 8529, "RequestSize_99.9th_percentile" : 8674, "RequestSize_SizeRangeCount_10-100" : 42, "RequestSize_SizeRangeCount_100-1000" : 71, "sentBytes" : 5263, "QueueCallTime_num_ops" : 113, "QueueCallTime_min" : 0, "QueueCallTime_max" : 31, "QueueCallTime_mean" : 2, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 1, "QueueCallTime_90th_percentile" : 9, "QueueCallTime_95th_percentile" : 19, "QueueCallTime_98th_percentile" : 29, "QueueCallTime_99th_percentile" : 29, "QueueCallTime_99.9th_percentile" : 31, "QueueCallTime_TimeRangeCount_0-1" : 92, "QueueCallTime_TimeRangeCount_1-3" : 6, "QueueCallTime_TimeRangeCount_3-10" : 4, "QueueCallTime_TimeRangeCount_10-30" : 10, "QueueCallTime_TimeRangeCount_30-100" : 1, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:56841", "tag.serverName" : "ubuntu,39267,1517713227316", "tag.clusterId" : "990c040f-15f4-4bef-a30c-08f39bc273d8", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "regionCount" : 10, "storeCount" : 1, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 0, "memStoreSize" : 78, "storeFileSize" : 0, "maxStoreFileAge" : 0, "minStoreFileAge" : 0, "avgStoreFileAge" : 0, "numReferenceFiles" : 0, "regionServerStartTime" : 1517713227316, "averageRegionSize" : 78, "totalRequestCount" : 9, "totalRowActionRequestCount" : 4, "readRequestCount" : 2, "filteredReadRequestCount" : 0, "writeRequestCount" : 2, "rpcGetRequestCount" : 4, "rpcScanRequestCount" : 1, "rpcMultiRequestCount" : 0, "rpcMutateRequestCount" : 2, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 0, "staticIndexSize" : 0, "staticBloomSize" : 0, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1116962296, "blockCacheCount" : 0, "blockCacheSize" : 819720, "blockCacheHitCount" : 0, "blockCacheHitCountPrimary" : 0, "blockCacheMissCount" : 0, "blockCacheMissCountPrimary" : 0, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 0.0, "blockCacheExpressHitPercent" : 0.0, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 0, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 0, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 0, "l1CacheMissCount" : 0, "l1CacheHitRatio" : NaN, "l1CacheMissRatio" : NaN, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 0, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 0, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 1, "ScanTime_min" : 1, "ScanTime_max" : 1, "ScanTime_mean" : 1, "ScanTime_25th_percentile" : 1, "ScanTime_median" : 1, "ScanTime_75th_percentile" : 1, "ScanTime_90th_percentile" : 1, "ScanTime_95th_percentile" : 1, "ScanTime_98th_percentile" : 1, "ScanTime_99th_percentile" : 1, "ScanTime_99.9th_percentile" : 1, "ScanTime_TimeRangeCount_0-1" : 1, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 2, "Put_min" : 2, "Put_max" : 9, "Put_mean" : 5, "Put_25th_percentile" : 2, "Put_median" : 3, "Put_75th_percentile" : 9, "Put_90th_percentile" : 9, "Put_95th_percentile" : 9, "Put_98th_percentile" : 9, "Put_99th_percentile" : 9, "Put_99.9th_percentile" : 9, "Put_TimeRangeCount_1-3" : 1, "Put_TimeRangeCount_3-10" : 1, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 0, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 0, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 4, "Get_min" : 0, "Get_max" : 1, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 1, "Get_75th_percentile" : 1, "Get_90th_percentile" : 1, "Get_95th_percentile" : 1, "Get_98th_percentile" : 1, "Get_99th_percentile" : 1, "Get_99.9th_percentile" : 1, "Get_TimeRangeCount_0-1" : 4, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 0, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 0, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 1, "ScanSize_min" : 86, "ScanSize_max" : 86, "ScanSize_mean" : 86, "ScanSize_25th_percentile" : 86, "ScanSize_median" : 86, "ScanSize_75th_percentile" : 86, "ScanSize_90th_percentile" : 86, "ScanSize_95th_percentile" : 86, "ScanSize_98th_percentile" : 86, "ScanSize_99th_percentile" : 86, "ScanSize_99.9th_percentile" : 86, "ScanSize_SizeRangeCount_10-100" : 1, "slowGetCount" : 0, "flushedMemstoreBytes" : 0, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 0, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-02-04 11:00:40,219 ERROR [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=39063] master.MasterRpcServices(503): Region server ubuntu,39267,1517713227316 reported a fatal error: ***** ABORTING region server ubuntu,39267,1517713227316: Unrecoverable exception while closing region tabletest,sss,1517713232695.e063e6b2c2640eb6eafc8737a71fb5f5., still finishing close ***** Cause: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2369) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2354) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-02-04 11:00:40,218 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(443): regionserver:39267-0x1615ec1f6c10001, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Expired, path=null 2018-02-04 11:00:40,219 ERROR [Time-limited test-EventThread] helpers.MarkerIgnoringBase(159): ***** ABORTING region server ubuntu,39267,1517713227316: regionserver:39267-0x1615ec1f6c10001, quorum=localhost:56841, baseZNode=/hbase regionserver:39267-0x1615ec1f6c10001 received expired from ZooKeeper, aborting ***** org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired at org.apache.hadoop.hbase.zookeeper.ZKWatcher.connectionEvent(ZKWatcher.java:520) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:452) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:40) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505) 2018-02-04 11:00:40,220 ERROR [Time-limited test-EventThread] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-02-04 11:00:40,221 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-1] executor.EventHandler(186): Caught throwable while processing event M_RS_CLOSE_REGION java.lang.RuntimeException: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:117) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2369) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2354) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) ... 4 more 2018-02-04 11:00:40,222 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-1] helpers.MarkerIgnoringBase(159): ***** ABORTING region server ubuntu,39267,1517713227316: Caught throwable while processing event M_RS_CLOSE_REGION ***** java.lang.RuntimeException: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:117) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2369) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2354) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) ... 4 more 2018-02-04 11:00:40,222 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-1] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-02-04 11:00:40,224 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegionServer(2350): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 855638016, "init" : 197132288, "max" : 2794455040, "used" : 636618184 }, "NonHeapMemoryUsage" : { "committed" : 118317056, "init" : 2555904, "max" : -1, "used" : 116142344 }, "Verbose" : false, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : 1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 50164, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 113, "TotalCallTime_min" : 0, "TotalCallTime_max" : 120, "TotalCallTime_mean" : 12, "TotalCallTime_25th_percentile" : 1, "TotalCallTime_median" : 4, "TotalCallTime_75th_percentile" : 12, "TotalCallTime_90th_percentile" : 38, "TotalCallTime_95th_percentile" : 45, "TotalCallTime_98th_percentile" : 52, "TotalCallTime_99th_percentile" : 53, "TotalCallTime_99.9th_percentile" : 120, "TotalCallTime_TimeRangeCount_0-1" : 29, "TotalCallTime_TimeRangeCount_1-3" : 17, "TotalCallTime_TimeRangeCount_3-10" : 36, "TotalCallTime_TimeRangeCount_10-30" : 12, "TotalCallTime_TimeRangeCount_30-100" : 18, "TotalCallTime_TimeRangeCount_100-300" : 1, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 113, "ResponseSize_min" : 0, "ResponseSize_max" : 787, "ResponseSize_mean" : 37, "ResponseSize_25th_percentile" : 10, "ResponseSize_median" : 10, "ResponseSize_75th_percentile" : 51, "ResponseSize_90th_percentile" : 51, "ResponseSize_95th_percentile" : 57, "ResponseSize_98th_percentile" : 161, "ResponseSize_99th_percentile" : 486, "ResponseSize_99.9th_percentile" : 756, "ResponseSize_SizeRangeCount_0-10" : 73, "ResponseSize_SizeRangeCount_10-100" : 36, "ResponseSize_SizeRangeCount_100-1000" : 4, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 113, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 120, "ProcessCallTime_mean" : 9, "ProcessCallTime_25th_percentile" : 1, "ProcessCallTime_median" : 4, "ProcessCallTime_75th_percentile" : 11, "ProcessCallTime_90th_percentile" : 28, "ProcessCallTime_95th_percentile" : 38, "ProcessCallTime_98th_percentile" : 51, "ProcessCallTime_99th_percentile" : 53, "ProcessCallTime_99.9th_percentile" : 120, "ProcessCallTime_TimeRangeCount_0-1" : 36, "ProcessCallTime_TimeRangeCount_1-3" : 15, "ProcessCallTime_TimeRangeCount_3-10" : 33, "ProcessCallTime_TimeRangeCount_10-30" : 19, "ProcessCallTime_TimeRangeCount_30-100" : 9, "ProcessCallTime_TimeRangeCount_100-300" : 1, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 7, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 113, "RequestSize_min" : 70, "RequestSize_max" : 8691, "RequestSize_mean" : 424, "RequestSize_25th_percentile" : 81, "RequestSize_median" : 613, "RequestSize_75th_percentile" : 4652, "RequestSize_90th_percentile" : 7075, "RequestSize_95th_percentile" : 7883, "RequestSize_98th_percentile" : 8367, "RequestSize_99th_percentile" : 8529, "RequestSize_99.9th_percentile" : 8674, "RequestSize_SizeRangeCount_10-100" : 42, "RequestSize_SizeRangeCount_100-1000" : 71, "sentBytes" : 5263, "QueueCallTime_num_ops" : 113, "QueueCallTime_min" : 0, "QueueCallTime_max" : 31, "QueueCallTime_mean" : 2, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 1, "QueueCallTime_90th_percentile" : 9, "QueueCallTime_95th_percentile" : 19, "QueueCallTime_98th_percentile" : 29, "QueueCallTime_99th_percentile" : 29, "QueueCallTime_99.9th_percentile" : 31, "QueueCallTime_TimeRangeCount_0-1" : 92, "QueueCallTime_TimeRangeCount_1-3" : 6, "QueueCallTime_TimeRangeCount_3-10" : 4, "QueueCallTime_TimeRangeCount_10-30" : 10, "QueueCallTime_TimeRangeCount_30-100" : 1, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:56841", "tag.serverName" : "ubuntu,39267,1517713227316", "tag.clusterId" : "990c040f-15f4-4bef-a30c-08f39bc273d8", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "regionCount" : 10, "storeCount" : 1, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 0, "memStoreSize" : 78, "storeFileSize" : 0, "maxStoreFileAge" : 0, "minStoreFileAge" : 0, "avgStoreFileAge" : 0, "numReferenceFiles" : 0, "regionServerStartTime" : 1517713227316, "averageRegionSize" : 78, "totalRequestCount" : 9, "totalRowActionRequestCount" : 4, "readRequestCount" : 2, "filteredReadRequestCount" : 0, "writeRequestCount" : 2, "rpcGetRequestCount" : 4, "rpcScanRequestCount" : 1, "rpcMultiRequestCount" : 0, "rpcMutateRequestCount" : 2, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 0, "staticIndexSize" : 0, "staticBloomSize" : 0, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1116962296, "blockCacheCount" : 0, "blockCacheSize" : 819720, "blockCacheHitCount" : 0, "blockCacheHitCountPrimary" : 0, "blockCacheMissCount" : 0, "blockCacheMissCountPrimary" : 0, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 0.0, "blockCacheExpressHitPercent" : 0.0, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 0, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 0, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 0, "l1CacheMissCount" : 0, "l1CacheHitRatio" : NaN, "l1CacheMissRatio" : NaN, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 0, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 0, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 1, "ScanTime_min" : 1, "ScanTime_max" : 1, "ScanTime_mean" : 1, "ScanTime_25th_percentile" : 1, "ScanTime_median" : 1, "ScanTime_75th_percentile" : 1, "ScanTime_90th_percentile" : 1, "ScanTime_95th_percentile" : 1, "ScanTime_98th_percentile" : 1, "ScanTime_99th_percentile" : 1, "ScanTime_99.9th_percentile" : 1, "ScanTime_TimeRangeCount_0-1" : 1, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 2, "Put_min" : 2, "Put_max" : 9, "Put_mean" : 5, "Put_25th_percentile" : 2, "Put_median" : 3, "Put_75th_percentile" : 9, "Put_90th_percentile" : 9, "Put_95th_percentile" : 9, "Put_98th_percentile" : 9, "Put_99th_percentile" : 9, "Put_99.9th_percentile" : 9, "Put_TimeRangeCount_1-3" : 1, "Put_TimeRangeCount_3-10" : 1, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 0, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 0, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 4, "Get_min" : 0, "Get_max" : 1, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 1, "Get_75th_percentile" : 1, "Get_90th_percentile" : 1, "Get_95th_percentile" : 1, "Get_98th_percentile" : 1, "Get_99th_percentile" : 1, "Get_99.9th_percentile" : 1, "Get_TimeRangeCount_0-1" : 4, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 0, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 0, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 1, "ScanSize_min" : 86, "ScanSize_max" : 86, "ScanSize_mean" : 86, "ScanSize_25th_percentile" : 86, "ScanSize_median" : 86, "ScanSize_75th_percentile" : 86, "ScanSize_90th_percentile" : 86, "ScanSize_95th_percentile" : 86, "ScanSize_98th_percentile" : 86, "ScanSize_99th_percentile" : 86, "ScanSize_99.9th_percentile" : 86, "ScanSize_SizeRangeCount_10-100" : 1, "slowGetCount" : 0, "flushedMemstoreBytes" : 0, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 0, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-02-04 11:00:40,226 ERROR [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=39063] master.MasterRpcServices(503): Region server ubuntu,39267,1517713227316 reported a fatal error: ***** ABORTING region server ubuntu,39267,1517713227316: Caught throwable while processing event M_RS_CLOSE_REGION ***** Cause: java.lang.RuntimeException: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:117) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2369) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2354) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) ... 4 more 2018-02-04 11:00:40,225 INFO [Time-limited test-EventThread] regionserver.HRegionServer(2350): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 855638016, "init" : 197132288, "max" : 2794455040, "used" : 636052712 }, "NonHeapMemoryUsage" : { "committed" : 118317056, "init" : 2555904, "max" : -1, "used" : 116142344 }, "Verbose" : false, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : 1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 50164, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 113, "TotalCallTime_min" : 0, "TotalCallTime_max" : 120, "TotalCallTime_mean" : 12, "TotalCallTime_25th_percentile" : 1, "TotalCallTime_median" : 4, "TotalCallTime_75th_percentile" : 12, "TotalCallTime_90th_percentile" : 38, "TotalCallTime_95th_percentile" : 45, "TotalCallTime_98th_percentile" : 52, "TotalCallTime_99th_percentile" : 53, "TotalCallTime_99.9th_percentile" : 120, "TotalCallTime_TimeRangeCount_0-1" : 29, "TotalCallTime_TimeRangeCount_1-3" : 17, "TotalCallTime_TimeRangeCount_3-10" : 36, "TotalCallTime_TimeRangeCount_10-30" : 12, "TotalCallTime_TimeRangeCount_30-100" : 18, "TotalCallTime_TimeRangeCount_100-300" : 1, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 113, "ResponseSize_min" : 0, "ResponseSize_max" : 787, "ResponseSize_mean" : 37, "ResponseSize_25th_percentile" : 10, "ResponseSize_median" : 10, "ResponseSize_75th_percentile" : 51, "ResponseSize_90th_percentile" : 51, "ResponseSize_95th_percentile" : 57, "ResponseSize_98th_percentile" : 161, "ResponseSize_99th_percentile" : 486, "ResponseSize_99.9th_percentile" : 756, "ResponseSize_SizeRangeCount_0-10" : 73, "ResponseSize_SizeRangeCount_10-100" : 36, "ResponseSize_SizeRangeCount_100-1000" : 4, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 113, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 120, "ProcessCallTime_mean" : 9, "ProcessCallTime_25th_percentile" : 1, "ProcessCallTime_median" : 4, "ProcessCallTime_75th_percentile" : 11, "ProcessCallTime_90th_percentile" : 28, "ProcessCallTime_95th_percentile" : 38, "ProcessCallTime_98th_percentile" : 51, "ProcessCallTime_99th_percentile" : 53, "ProcessCallTime_99.9th_percentile" : 120, "ProcessCallTime_TimeRangeCount_0-1" : 36, "ProcessCallTime_TimeRangeCount_1-3" : 15, "ProcessCallTime_TimeRangeCount_3-10" : 33, "ProcessCallTime_TimeRangeCount_10-30" : 19, "ProcessCallTime_TimeRangeCount_30-100" : 9, "ProcessCallTime_TimeRangeCount_100-300" : 1, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 7, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 113, "RequestSize_min" : 70, "RequestSize_max" : 8691, "RequestSize_mean" : 424, "RequestSize_25th_percentile" : 81, "RequestSize_median" : 613, "RequestSize_75th_percentile" : 4652, "RequestSize_90th_percentile" : 7075, "RequestSize_95th_percentile" : 7883, "RequestSize_98th_percentile" : 8367, "RequestSize_99th_percentile" : 8529, "RequestSize_99.9th_percentile" : 8674, "RequestSize_SizeRangeCount_10-100" : 42, "RequestSize_SizeRangeCount_100-1000" : 71, "sentBytes" : 5263, "QueueCallTime_num_ops" : 113, "QueueCallTime_min" : 0, "QueueCallTime_max" : 31, "QueueCallTime_mean" : 2, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 1, "QueueCallTime_90th_percentile" : 9, "QueueCallTime_95th_percentile" : 19, "QueueCallTime_98th_percentile" : 29, "QueueCallTime_99th_percentile" : 29, "QueueCallTime_99.9th_percentile" : 31, "QueueCallTime_TimeRangeCount_0-1" : 92, "QueueCallTime_TimeRangeCount_1-3" : 6, "QueueCallTime_TimeRangeCount_3-10" : 4, "QueueCallTime_TimeRangeCount_10-30" : 10, "QueueCallTime_TimeRangeCount_30-100" : 1, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:56841", "tag.serverName" : "ubuntu,39267,1517713227316", "tag.clusterId" : "990c040f-15f4-4bef-a30c-08f39bc273d8", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "regionCount" : 10, "storeCount" : 1, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 0, "memStoreSize" : 78, "storeFileSize" : 0, "maxStoreFileAge" : 0, "minStoreFileAge" : 0, "avgStoreFileAge" : 0, "numReferenceFiles" : 0, "regionServerStartTime" : 1517713227316, "averageRegionSize" : 78, "totalRequestCount" : 9, "totalRowActionRequestCount" : 4, "readRequestCount" : 2, "filteredReadRequestCount" : 0, "writeRequestCount" : 2, "rpcGetRequestCount" : 4, "rpcScanRequestCount" : 1, "rpcMultiRequestCount" : 0, "rpcMutateRequestCount" : 2, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 0, "staticIndexSize" : 0, "staticBloomSize" : 0, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1116962296, "blockCacheCount" : 0, "blockCacheSize" : 819720, "blockCacheHitCount" : 0, "blockCacheHitCountPrimary" : 0, "blockCacheMissCount" : 0, "blockCacheMissCountPrimary" : 0, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 0.0, "blockCacheExpressHitPercent" : 0.0, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 0, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 0, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 0, "l1CacheMissCount" : 0, "l1CacheHitRatio" : NaN, "l1CacheMissRatio" : NaN, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 0, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 0, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 1, "ScanTime_min" : 1, "ScanTime_max" : 1, "ScanTime_mean" : 1, "ScanTime_25th_percentile" : 1, "ScanTime_median" : 1, "ScanTime_75th_percentile" : 1, "ScanTime_90th_percentile" : 1, "ScanTime_95th_percentile" : 1, "ScanTime_98th_percentile" : 1, "ScanTime_99th_percentile" : 1, "ScanTime_99.9th_percentile" : 1, "ScanTime_TimeRangeCount_0-1" : 1, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 2, "Put_min" : 2, "Put_max" : 9, "Put_mean" : 5, "Put_25th_percentile" : 2, "Put_median" : 3, "Put_75th_percentile" : 9, "Put_90th_percentile" : 9, "Put_95th_percentile" : 9, "Put_98th_percentile" : 9, "Put_99th_percentile" : 9, "Put_99.9th_percentile" : 9, "Put_TimeRangeCount_1-3" : 1, "Put_TimeRangeCount_3-10" : 1, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 0, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 0, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 4, "Get_min" : 0, "Get_max" : 1, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 1, "Get_75th_percentile" : 1, "Get_90th_percentile" : 1, "Get_95th_percentile" : 1, "Get_98th_percentile" : 1, "Get_99th_percentile" : 1, "Get_99.9th_percentile" : 1, "Get_TimeRangeCount_0-1" : 4, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 0, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 0, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 1, "ScanSize_min" : 86, "ScanSize_max" : 86, "ScanSize_mean" : 86, "ScanSize_25th_percentile" : 86, "ScanSize_median" : 86, "ScanSize_75th_percentile" : 86, "ScanSize_90th_percentile" : 86, "ScanSize_95th_percentile" : 86, "ScanSize_98th_percentile" : 86, "ScanSize_99th_percentile" : 86, "ScanSize_99.9th_percentile" : 86, "ScanSize_SizeRangeCount_10-100" : 1, "slowGetCount" : 0, "flushedMemstoreBytes" : 0, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 0, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-02-04 11:00:40,253 ERROR [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=39063] master.MasterRpcServices(503): Region server ubuntu,39267,1517713227316 reported a fatal error: ***** ABORTING region server ubuntu,39267,1517713227316: regionserver:39267-0x1615ec1f6c10001, quorum=localhost:56841, baseZNode=/hbase regionserver:39267-0x1615ec1f6c10001 received expired from ZooKeeper, aborting ***** Cause: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired at org.apache.hadoop.hbase.zookeeper.ZKWatcher.connectionEvent(ZKWatcher.java:520) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:452) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:40) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505) 2018-02-04 11:00:40,253 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] handler.CloseRegionHandler(92): Processing close of tabletest,eee,1517713232695.14343105008c3cdd39a78990296fbe94. 2018-02-04 11:00:40,254 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(1483): Closing tabletest,eee,1517713232695.14343105008c3cdd39a78990296fbe94.: disabling compactions & flushes 2018-02-04 11:00:40,254 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(1522): Updates disabled for region tabletest,eee,1517713232695.14343105008c3cdd39a78990296fbe94. 2018-02-04 11:00:40,254 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-1] helpers.MarkerIgnoringBase(159): ***** ABORTING region server ubuntu,39267,1517713227316: Unrecoverable exception while closing region tabletest,eee,1517713232695.14343105008c3cdd39a78990296fbe94., still finishing close ***** java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2369) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2354) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-02-04 11:00:40,254 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-1] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-02-04 11:00:40,256 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegionServer(2350): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 855638016, "init" : 197132288, "max" : 2794455040, "used" : 637921920 }, "NonHeapMemoryUsage" : { "committed" : 118317056, "init" : 2555904, "max" : -1, "used" : 116166528 }, "Verbose" : false, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : 1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 50164, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 113, "TotalCallTime_min" : 0, "TotalCallTime_max" : 120, "TotalCallTime_mean" : 12, "TotalCallTime_25th_percentile" : 1, "TotalCallTime_median" : 4, "TotalCallTime_75th_percentile" : 12, "TotalCallTime_90th_percentile" : 38, "TotalCallTime_95th_percentile" : 45, "TotalCallTime_98th_percentile" : 52, "TotalCallTime_99th_percentile" : 53, "TotalCallTime_99.9th_percentile" : 120, "TotalCallTime_TimeRangeCount_0-1" : 29, "TotalCallTime_TimeRangeCount_1-3" : 17, "TotalCallTime_TimeRangeCount_3-10" : 36, "TotalCallTime_TimeRangeCount_10-30" : 12, "TotalCallTime_TimeRangeCount_30-100" : 18, "TotalCallTime_TimeRangeCount_100-300" : 1, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 113, "ResponseSize_min" : 0, "ResponseSize_max" : 787, "ResponseSize_mean" : 37, "ResponseSize_25th_percentile" : 10, "ResponseSize_median" : 10, "ResponseSize_75th_percentile" : 51, "ResponseSize_90th_percentile" : 51, "ResponseSize_95th_percentile" : 57, "ResponseSize_98th_percentile" : 161, "ResponseSize_99th_percentile" : 486, "ResponseSize_99.9th_percentile" : 756, "ResponseSize_SizeRangeCount_0-10" : 73, "ResponseSize_SizeRangeCount_10-100" : 36, "ResponseSize_SizeRangeCount_100-1000" : 4, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 113, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 120, "ProcessCallTime_mean" : 9, "ProcessCallTime_25th_percentile" : 1, "ProcessCallTime_median" : 4, "ProcessCallTime_75th_percentile" : 11, "ProcessCallTime_90th_percentile" : 28, "ProcessCallTime_95th_percentile" : 38, "ProcessCallTime_98th_percentile" : 51, "ProcessCallTime_99th_percentile" : 53, "ProcessCallTime_99.9th_percentile" : 120, "ProcessCallTime_TimeRangeCount_0-1" : 36, "ProcessCallTime_TimeRangeCount_1-3" : 15, "ProcessCallTime_TimeRangeCount_3-10" : 33, "ProcessCallTime_TimeRangeCount_10-30" : 19, "ProcessCallTime_TimeRangeCount_30-100" : 9, "ProcessCallTime_TimeRangeCount_100-300" : 1, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 7, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 113, "RequestSize_min" : 70, "RequestSize_max" : 8691, "RequestSize_mean" : 424, "RequestSize_25th_percentile" : 81, "RequestSize_median" : 613, "RequestSize_75th_percentile" : 4652, "RequestSize_90th_percentile" : 7075, "RequestSize_95th_percentile" : 7883, "RequestSize_98th_percentile" : 8367, "RequestSize_99th_percentile" : 8529, "RequestSize_99.9th_percentile" : 8674, "RequestSize_SizeRangeCount_10-100" : 42, "RequestSize_SizeRangeCount_100-1000" : 71, "sentBytes" : 5263, "QueueCallTime_num_ops" : 113, "QueueCallTime_min" : 0, "QueueCallTime_max" : 31, "QueueCallTime_mean" : 2, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 1, "QueueCallTime_90th_percentile" : 9, "QueueCallTime_95th_percentile" : 19, "QueueCallTime_98th_percentile" : 29, "QueueCallTime_99th_percentile" : 29, "QueueCallTime_99.9th_percentile" : 31, "QueueCallTime_TimeRangeCount_0-1" : 92, "QueueCallTime_TimeRangeCount_1-3" : 6, "QueueCallTime_TimeRangeCount_3-10" : 4, "QueueCallTime_TimeRangeCount_10-30" : 10, "QueueCallTime_TimeRangeCount_30-100" : 1, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:56841", "tag.serverName" : "ubuntu,39267,1517713227316", "tag.clusterId" : "990c040f-15f4-4bef-a30c-08f39bc273d8", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "regionCount" : 10, "storeCount" : 1, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 0, "memStoreSize" : 78, "storeFileSize" : 0, "maxStoreFileAge" : 0, "minStoreFileAge" : 0, "avgStoreFileAge" : 0, "numReferenceFiles" : 0, "regionServerStartTime" : 1517713227316, "averageRegionSize" : 78, "totalRequestCount" : 9, "totalRowActionRequestCount" : 4, "readRequestCount" : 2, "filteredReadRequestCount" : 0, "writeRequestCount" : 2, "rpcGetRequestCount" : 4, "rpcScanRequestCount" : 1, "rpcMultiRequestCount" : 0, "rpcMutateRequestCount" : 2, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 0, "staticIndexSize" : 0, "staticBloomSize" : 0, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1116962296, "blockCacheCount" : 0, "blockCacheSize" : 819720, "blockCacheHitCount" : 0, "blockCacheHitCountPrimary" : 0, "blockCacheMissCount" : 0, "blockCacheMissCountPrimary" : 0, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 0.0, "blockCacheExpressHitPercent" : 0.0, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 0, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 0, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 0, "l1CacheMissCount" : 0, "l1CacheHitRatio" : NaN, "l1CacheMissRatio" : NaN, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 0, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 0, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 1, "ScanTime_min" : 1, "ScanTime_max" : 1, "ScanTime_mean" : 1, "ScanTime_25th_percentile" : 1, "ScanTime_median" : 1, "ScanTime_75th_percentile" : 1, "ScanTime_90th_percentile" : 1, "ScanTime_95th_percentile" : 1, "ScanTime_98th_percentile" : 1, "ScanTime_99th_percentile" : 1, "ScanTime_99.9th_percentile" : 1, "ScanTime_TimeRangeCount_0-1" : 1, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 2, "Put_min" : 2, "Put_max" : 9, "Put_mean" : 5, "Put_25th_percentile" : 2, "Put_median" : 3, "Put_75th_percentile" : 9, "Put_90th_percentile" : 9, "Put_95th_percentile" : 9, "Put_98th_percentile" : 9, "Put_99th_percentile" : 9, "Put_99.9th_percentile" : 9, "Put_TimeRangeCount_1-3" : 1, "Put_TimeRangeCount_3-10" : 1, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 0, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 0, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 4, "Get_min" : 0, "Get_max" : 1, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 1, "Get_75th_percentile" : 1, "Get_90th_percentile" : 1, "Get_95th_percentile" : 1, "Get_98th_percentile" : 1, "Get_99th_percentile" : 1, "Get_99.9th_percentile" : 1, "Get_TimeRangeCount_0-1" : 4, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 0, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 0, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 1, "ScanSize_min" : 86, "ScanSize_max" : 86, "ScanSize_mean" : 86, "ScanSize_25th_percentile" : 86, "ScanSize_median" : 86, "ScanSize_75th_percentile" : 86, "ScanSize_90th_percentile" : 86, "ScanSize_95th_percentile" : 86, "ScanSize_98th_percentile" : 86, "ScanSize_99th_percentile" : 86, "ScanSize_99.9th_percentile" : 86, "ScanSize_SizeRangeCount_10-100" : 1, "slowGetCount" : 0, "flushedMemstoreBytes" : 0, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 0, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-02-04 11:00:40,257 ERROR [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=39063] master.MasterRpcServices(503): Region server ubuntu,39267,1517713227316 reported a fatal error: ***** ABORTING region server ubuntu,39267,1517713227316: Unrecoverable exception while closing region tabletest,eee,1517713232695.14343105008c3cdd39a78990296fbe94., still finishing close ***** Cause: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2369) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2354) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-02-04 11:00:40,258 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-1] executor.EventHandler(186): Caught throwable while processing event M_RS_CLOSE_REGION java.lang.RuntimeException: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:117) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2369) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2354) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) ... 4 more 2018-02-04 11:00:40,258 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-1] helpers.MarkerIgnoringBase(159): ***** ABORTING region server ubuntu,39267,1517713227316: Caught throwable while processing event M_RS_CLOSE_REGION ***** java.lang.RuntimeException: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:117) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2369) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2354) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) ... 4 more 2018-02-04 11:00:40,266 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.DefaultStoreFlusher(85): Flushed, sequenceid=9, memsize=130.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/9579a7c2f22113dfe574ba9569101416/.tmp/family/fa6031c2b1884098801946f475a95d94 2018-02-04 11:00:40,265 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(1635): Closed tabletest,nnn,1517713232695.7429e32b4c4e0aadde8a8154cbd1f071. 2018-02-04 11:00:40,267 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] handler.CloseRegionHandler(125): Closed tabletest,nnn,1517713232695.7429e32b4c4e0aadde8a8154cbd1f071. 2018-02-04 11:00:40,267 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] handler.CloseRegionHandler(92): Processing close of tabletest,bbb,1517713232695.756819a3f903733cb342d3913aab52a6. 2018-02-04 11:00:40,267 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(1483): Closing tabletest,bbb,1517713232695.756819a3f903733cb342d3913aab52a6.: disabling compactions & flushes 2018-02-04 11:00:40,267 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(1522): Updates disabled for region tabletest,bbb,1517713232695.756819a3f903733cb342d3913aab52a6. 2018-02-04 11:00:40,267 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(2518): Flushing 1/1 column families, memstore=130.44 KB 2018-02-04 11:00:40,267 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.CompactingMemStore(205): FLUSHING TO DISK: region tabletest,bbb,1517713232695.756819a3f903733cb342d3913aab52a6.store: family 2018-02-04 11:00:40,267 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.CompactionPipeline(128): Swapping pipeline suffix; before=1, new segement=null 2018-02-04 11:00:40,265 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.DefaultStoreFlusher(85): Flushed, sequenceid=9, memsize=130.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/68c9676b485038d63fd2cf8a435420e5/.tmp/family/6a34fbf7441641b592c77283921544a9 2018-02-04 11:00:40,265 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(1635): Closed tabletest,qqq,1517713232695.6a7b42fe3d02ea7aca1d05fa666e87ac. 2018-02-04 11:00:40,269 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] handler.CloseRegionHandler(125): Closed tabletest,qqq,1517713232695.6a7b42fe3d02ea7aca1d05fa666e87ac. 2018-02-04 11:00:40,269 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] handler.CloseRegionHandler(92): Processing close of tabletest,ppp,1517713232695.12bc4be2348528b0bd4575ae03200fc3. 2018-02-04 11:00:40,269 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(1483): Closing tabletest,ppp,1517713232695.12bc4be2348528b0bd4575ae03200fc3.: disabling compactions & flushes 2018-02-04 11:00:40,269 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(1522): Updates disabled for region tabletest,ppp,1517713232695.12bc4be2348528b0bd4575ae03200fc3. 2018-02-04 11:00:40,269 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(2518): Flushing 1/1 column families, memstore=130.44 KB 2018-02-04 11:00:40,270 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.CompactingMemStore(205): FLUSHING TO DISK: region tabletest,ppp,1517713232695.12bc4be2348528b0bd4575ae03200fc3.store: family 2018-02-04 11:00:40,270 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.CompactionPipeline(128): Swapping pipeline suffix; before=1, new segement=null 2018-02-04 11:00:40,267 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(2714): Finished memstore flush of ~130.44 KB/133570, currentsize=0 B/0 for region tabletest,yyy,1517713232695.20457e81e436062b31548a08737ebba4. in 797ms, sequenceid=9, compaction requested=false 2018-02-04 11:00:40,291 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-1] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-02-04 11:00:40,310 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741880_1057{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|RBW]]} size 0 2018-02-04 11:00:40,313 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741880_1057{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|RBW]]} size 0 2018-02-04 11:00:40,328 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741880_1057{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|RBW]]} size 0 2018-02-04 11:00:40,330 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.DefaultStoreFlusher(85): Flushed, sequenceid=9, memsize=130.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/cd5c06da9ebf17554a264cbdd6fcd9ef/.tmp/family/c410acf329e24620ad6b3634c829c1cb 2018-02-04 11:00:40,370 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegionServer(2350): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 855638016, "init" : 197132288, "max" : 2794455040, "used" : 638984376 }, "NonHeapMemoryUsage" : { "committed" : 118317056, "init" : 2555904, "max" : -1, "used" : 116208224 }, "Verbose" : false, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : 1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 50164, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 113, "TotalCallTime_min" : 0, "TotalCallTime_max" : 120, "TotalCallTime_mean" : 12, "TotalCallTime_25th_percentile" : 1, "TotalCallTime_median" : 4, "TotalCallTime_75th_percentile" : 12, "TotalCallTime_90th_percentile" : 38, "TotalCallTime_95th_percentile" : 45, "TotalCallTime_98th_percentile" : 52, "TotalCallTime_99th_percentile" : 53, "TotalCallTime_99.9th_percentile" : 120, "TotalCallTime_TimeRangeCount_0-1" : 29, "TotalCallTime_TimeRangeCount_1-3" : 17, "TotalCallTime_TimeRangeCount_3-10" : 36, "TotalCallTime_TimeRangeCount_10-30" : 12, "TotalCallTime_TimeRangeCount_30-100" : 18, "TotalCallTime_TimeRangeCount_100-300" : 1, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 113, "ResponseSize_min" : 0, "ResponseSize_max" : 787, "ResponseSize_mean" : 37, "ResponseSize_25th_percentile" : 10, "ResponseSize_median" : 10, "ResponseSize_75th_percentile" : 51, "ResponseSize_90th_percentile" : 51, "ResponseSize_95th_percentile" : 57, "ResponseSize_98th_percentile" : 161, "ResponseSize_99th_percentile" : 486, "ResponseSize_99.9th_percentile" : 756, "ResponseSize_SizeRangeCount_0-10" : 73, "ResponseSize_SizeRangeCount_10-100" : 36, "ResponseSize_SizeRangeCount_100-1000" : 4, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 113, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 120, "ProcessCallTime_mean" : 9, "ProcessCallTime_25th_percentile" : 1, "ProcessCallTime_median" : 4, "ProcessCallTime_75th_percentile" : 11, "ProcessCallTime_90th_percentile" : 28, "ProcessCallTime_95th_percentile" : 38, "ProcessCallTime_98th_percentile" : 51, "ProcessCallTime_99th_percentile" : 53, "ProcessCallTime_99.9th_percentile" : 120, "ProcessCallTime_TimeRangeCount_0-1" : 36, "ProcessCallTime_TimeRangeCount_1-3" : 15, "ProcessCallTime_TimeRangeCount_3-10" : 33, "ProcessCallTime_TimeRangeCount_10-30" : 19, "ProcessCallTime_TimeRangeCount_30-100" : 9, "ProcessCallTime_TimeRangeCount_100-300" : 1, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 7, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 113, "RequestSize_min" : 70, "RequestSize_max" : 8691, "RequestSize_mean" : 424, "RequestSize_25th_percentile" : 81, "RequestSize_median" : 613, "RequestSize_75th_percentile" : 4652, "RequestSize_90th_percentile" : 7075, "RequestSize_95th_percentile" : 7883, "RequestSize_98th_percentile" : 8367, "RequestSize_99th_percentile" : 8529, "RequestSize_99.9th_percentile" : 8674, "RequestSize_SizeRangeCount_10-100" : 42, "RequestSize_SizeRangeCount_100-1000" : 71, "sentBytes" : 5263, "QueueCallTime_num_ops" : 113, "QueueCallTime_min" : 0, "QueueCallTime_max" : 31, "QueueCallTime_mean" : 2, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 1, "QueueCallTime_90th_percentile" : 9, "QueueCallTime_95th_percentile" : 19, "QueueCallTime_98th_percentile" : 29, "QueueCallTime_99th_percentile" : 29, "QueueCallTime_99.9th_percentile" : 31, "QueueCallTime_TimeRangeCount_0-1" : 92, "QueueCallTime_TimeRangeCount_1-3" : 6, "QueueCallTime_TimeRangeCount_3-10" : 4, "QueueCallTime_TimeRangeCount_10-30" : 10, "QueueCallTime_TimeRangeCount_30-100" : 1, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:56841", "tag.serverName" : "ubuntu,39267,1517713227316", "tag.clusterId" : "990c040f-15f4-4bef-a30c-08f39bc273d8", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "regionCount" : 10, "storeCount" : 1, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 0, "memStoreSize" : 78, "storeFileSize" : 0, "maxStoreFileAge" : 0, "minStoreFileAge" : 0, "avgStoreFileAge" : 0, "numReferenceFiles" : 0, "regionServerStartTime" : 1517713227316, "averageRegionSize" : 78, "totalRequestCount" : 9, "totalRowActionRequestCount" : 4, "readRequestCount" : 2, "filteredReadRequestCount" : 0, "writeRequestCount" : 2, "rpcGetRequestCount" : 4, "rpcScanRequestCount" : 1, "rpcMultiRequestCount" : 0, "rpcMutateRequestCount" : 2, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 0, "staticIndexSize" : 0, "staticBloomSize" : 0, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1116962296, "blockCacheCount" : 0, "blockCacheSize" : 819720, "blockCacheHitCount" : 0, "blockCacheHitCountPrimary" : 0, "blockCacheMissCount" : 0, "blockCacheMissCountPrimary" : 0, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 0.0, "blockCacheExpressHitPercent" : 0.0, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 0, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 0, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 0, "l1CacheMissCount" : 0, "l1CacheHitRatio" : NaN, "l1CacheMissRatio" : NaN, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 0, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 0, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 1, "ScanTime_min" : 1, "ScanTime_max" : 1, "ScanTime_mean" : 1, "ScanTime_25th_percentile" : 1, "ScanTime_median" : 1, "ScanTime_75th_percentile" : 1, "ScanTime_90th_percentile" : 1, "ScanTime_95th_percentile" : 1, "ScanTime_98th_percentile" : 1, "ScanTime_99th_percentile" : 1, "ScanTime_99.9th_percentile" : 1, "ScanTime_TimeRangeCount_0-1" : 1, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 2, "Put_min" : 2, "Put_max" : 9, "Put_mean" : 5, "Put_25th_percentile" : 2, "Put_median" : 3, "Put_75th_percentile" : 9, "Put_90th_percentile" : 9, "Put_95th_percentile" : 9, "Put_98th_percentile" : 9, "Put_99th_percentile" : 9, "Put_99.9th_percentile" : 9, "Put_TimeRangeCount_1-3" : 1, "Put_TimeRangeCount_3-10" : 1, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 0, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 0, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 4, "Get_min" : 0, "Get_max" : 1, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 1, "Get_75th_percentile" : 1, "Get_90th_percentile" : 1, "Get_95th_percentile" : 1, "Get_98th_percentile" : 1, "Get_99th_percentile" : 1, "Get_99.9th_percentile" : 1, "Get_TimeRangeCount_0-1" : 4, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 0, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 0, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 1, "ScanSize_min" : 86, "ScanSize_max" : 86, "ScanSize_mean" : 86, "ScanSize_25th_percentile" : 86, "ScanSize_median" : 86, "ScanSize_75th_percentile" : 86, "ScanSize_90th_percentile" : 86, "ScanSize_95th_percentile" : 86, "ScanSize_98th_percentile" : 86, "ScanSize_99th_percentile" : 86, "ScanSize_99.9th_percentile" : 86, "ScanSize_SizeRangeCount_10-100" : 1, "slowGetCount" : 0, "flushedMemstoreBytes" : 0, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 0, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-02-04 11:00:40,371 ERROR [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=39063] master.MasterRpcServices(503): Region server ubuntu,39267,1517713227316 reported a fatal error: ***** ABORTING region server ubuntu,39267,1517713227316: Caught throwable while processing event M_RS_CLOSE_REGION ***** Cause: java.lang.RuntimeException: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:117) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2369) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2354) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) ... 4 more 2018-02-04 11:00:40,374 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] handler.CloseRegionHandler(92): Processing close of tabletest,fff,1517713232695.7b96ff3277efd9751a2175b9c2b4d481. 2018-02-04 11:00:40,375 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(1483): Closing tabletest,fff,1517713232695.7b96ff3277efd9751a2175b9c2b4d481.: disabling compactions & flushes 2018-02-04 11:00:40,375 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(1522): Updates disabled for region tabletest,fff,1517713232695.7b96ff3277efd9751a2175b9c2b4d481. 2018-02-04 11:00:40,375 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegionFileSystem(463): Committing store file hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/68c9676b485038d63fd2cf8a435420e5/.tmp/family/6a34fbf7441641b592c77283921544a9 as hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/68c9676b485038d63fd2cf8a435420e5/family/6a34fbf7441641b592c77283921544a9 2018-02-04 11:00:40,375 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-1] helpers.MarkerIgnoringBase(159): ***** ABORTING region server ubuntu,39267,1517713227316: Unrecoverable exception while closing region tabletest,fff,1517713232695.7b96ff3277efd9751a2175b9c2b4d481., still finishing close ***** java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2369) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2354) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-02-04 11:00:40,376 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-1] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-02-04 11:00:40,379 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegionFileSystem(463): Committing store file hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/9579a7c2f22113dfe574ba9569101416/.tmp/family/fa6031c2b1884098801946f475a95d94 as hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/9579a7c2f22113dfe574ba9569101416/family/fa6031c2b1884098801946f475a95d94 2018-02-04 11:00:40,381 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegionServer(2350): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 855638016, "init" : 197132288, "max" : 2794455040, "used" : 642206272 }, "NonHeapMemoryUsage" : { "committed" : 118579200, "init" : 2555904, "max" : -1, "used" : 116267264 }, "Verbose" : false, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : 1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 50164, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 113, "TotalCallTime_min" : 0, "TotalCallTime_max" : 120, "TotalCallTime_mean" : 12, "TotalCallTime_25th_percentile" : 1, "TotalCallTime_median" : 4, "TotalCallTime_75th_percentile" : 12, "TotalCallTime_90th_percentile" : 38, "TotalCallTime_95th_percentile" : 45, "TotalCallTime_98th_percentile" : 52, "TotalCallTime_99th_percentile" : 53, "TotalCallTime_99.9th_percentile" : 120, "TotalCallTime_TimeRangeCount_0-1" : 29, "TotalCallTime_TimeRangeCount_1-3" : 17, "TotalCallTime_TimeRangeCount_3-10" : 36, "TotalCallTime_TimeRangeCount_10-30" : 12, "TotalCallTime_TimeRangeCount_30-100" : 18, "TotalCallTime_TimeRangeCount_100-300" : 1, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 113, "ResponseSize_min" : 0, "ResponseSize_max" : 787, "ResponseSize_mean" : 37, "ResponseSize_25th_percentile" : 10, "ResponseSize_median" : 10, "ResponseSize_75th_percentile" : 51, "ResponseSize_90th_percentile" : 51, "ResponseSize_95th_percentile" : 57, "ResponseSize_98th_percentile" : 161, "ResponseSize_99th_percentile" : 486, "ResponseSize_99.9th_percentile" : 756, "ResponseSize_SizeRangeCount_0-10" : 73, "ResponseSize_SizeRangeCount_10-100" : 36, "ResponseSize_SizeRangeCount_100-1000" : 4, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 113, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 120, "ProcessCallTime_mean" : 9, "ProcessCallTime_25th_percentile" : 1, "ProcessCallTime_median" : 4, "ProcessCallTime_75th_percentile" : 11, "ProcessCallTime_90th_percentile" : 28, "ProcessCallTime_95th_percentile" : 38, "ProcessCallTime_98th_percentile" : 51, "ProcessCallTime_99th_percentile" : 53, "ProcessCallTime_99.9th_percentile" : 120, "ProcessCallTime_TimeRangeCount_0-1" : 36, "ProcessCallTime_TimeRangeCount_1-3" : 15, "ProcessCallTime_TimeRangeCount_3-10" : 33, "ProcessCallTime_TimeRangeCount_10-30" : 19, "ProcessCallTime_TimeRangeCount_30-100" : 9, "ProcessCallTime_TimeRangeCount_100-300" : 1, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 7, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 113, "RequestSize_min" : 70, "RequestSize_max" : 8691, "RequestSize_mean" : 424, "RequestSize_25th_percentile" : 81, "RequestSize_median" : 613, "RequestSize_75th_percentile" : 4652, "RequestSize_90th_percentile" : 7075, "RequestSize_95th_percentile" : 7883, "RequestSize_98th_percentile" : 8367, "RequestSize_99th_percentile" : 8529, "RequestSize_99.9th_percentile" : 8674, "RequestSize_SizeRangeCount_10-100" : 42, "RequestSize_SizeRangeCount_100-1000" : 71, "sentBytes" : 5263, "QueueCallTime_num_ops" : 113, "QueueCallTime_min" : 0, "QueueCallTime_max" : 31, "QueueCallTime_mean" : 2, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 1, "QueueCallTime_90th_percentile" : 9, "QueueCallTime_95th_percentile" : 19, "QueueCallTime_98th_percentile" : 29, "QueueCallTime_99th_percentile" : 29, "QueueCallTime_99.9th_percentile" : 31, "QueueCallTime_TimeRangeCount_0-1" : 92, "QueueCallTime_TimeRangeCount_1-3" : 6, "QueueCallTime_TimeRangeCount_3-10" : 4, "QueueCallTime_TimeRangeCount_10-30" : 10, "QueueCallTime_TimeRangeCount_30-100" : 1, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:56841", "tag.serverName" : "ubuntu,39267,1517713227316", "tag.clusterId" : "990c040f-15f4-4bef-a30c-08f39bc273d8", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "regionCount" : 10, "storeCount" : 1, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 0, "memStoreSize" : 78, "storeFileSize" : 0, "maxStoreFileAge" : 0, "minStoreFileAge" : 0, "avgStoreFileAge" : 0, "numReferenceFiles" : 0, "regionServerStartTime" : 1517713227316, "averageRegionSize" : 78, "totalRequestCount" : 9, "totalRowActionRequestCount" : 4, "readRequestCount" : 2, "filteredReadRequestCount" : 0, "writeRequestCount" : 2, "rpcGetRequestCount" : 4, "rpcScanRequestCount" : 1, "rpcMultiRequestCount" : 0, "rpcMutateRequestCount" : 2, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 0, "staticIndexSize" : 0, "staticBloomSize" : 0, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1116962296, "blockCacheCount" : 0, "blockCacheSize" : 819720, "blockCacheHitCount" : 0, "blockCacheHitCountPrimary" : 0, "blockCacheMissCount" : 0, "blockCacheMissCountPrimary" : 0, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 0.0, "blockCacheExpressHitPercent" : 0.0, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 0, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 0, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 0, "l1CacheMissCount" : 0, "l1CacheHitRatio" : NaN, "l1CacheMissRatio" : NaN, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 0, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 0, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 1, "ScanTime_min" : 1, "ScanTime_max" : 1, "ScanTime_mean" : 1, "ScanTime_25th_percentile" : 1, "ScanTime_median" : 1, "ScanTime_75th_percentile" : 1, "ScanTime_90th_percentile" : 1, "ScanTime_95th_percentile" : 1, "ScanTime_98th_percentile" : 1, "ScanTime_99th_percentile" : 1, "ScanTime_99.9th_percentile" : 1, "ScanTime_TimeRangeCount_0-1" : 1, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 2, "Put_min" : 2, "Put_max" : 9, "Put_mean" : 5, "Put_25th_percentile" : 2, "Put_median" : 3, "Put_75th_percentile" : 9, "Put_90th_percentile" : 9, "Put_95th_percentile" : 9, "Put_98th_percentile" : 9, "Put_99th_percentile" : 9, "Put_99.9th_percentile" : 9, "Put_TimeRangeCount_1-3" : 1, "Put_TimeRangeCount_3-10" : 1, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 0, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 0, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 4, "Get_min" : 0, "Get_max" : 1, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 1, "Get_75th_percentile" : 1, "Get_90th_percentile" : 1, "Get_95th_percentile" : 1, "Get_98th_percentile" : 1, "Get_99th_percentile" : 1, "Get_99.9th_percentile" : 1, "Get_TimeRangeCount_0-1" : 4, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 0, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 0, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 1, "ScanSize_min" : 86, "ScanSize_max" : 86, "ScanSize_mean" : 86, "ScanSize_25th_percentile" : 86, "ScanSize_median" : 86, "ScanSize_75th_percentile" : 86, "ScanSize_90th_percentile" : 86, "ScanSize_95th_percentile" : 86, "ScanSize_98th_percentile" : 86, "ScanSize_99th_percentile" : 86, "ScanSize_99.9th_percentile" : 86, "ScanSize_SizeRangeCount_10-100" : 1, "slowGetCount" : 0, "flushedMemstoreBytes" : 0, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 0, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-02-04 11:00:40,383 ERROR [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=39063] master.MasterRpcServices(503): Region server ubuntu,39267,1517713227316 reported a fatal error: ***** ABORTING region server ubuntu,39267,1517713227316: Unrecoverable exception while closing region tabletest,fff,1517713232695.7b96ff3277efd9751a2175b9c2b4d481., still finishing close ***** Cause: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2369) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2354) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-02-04 11:00:40,384 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-1] executor.EventHandler(186): Caught throwable while processing event M_RS_CLOSE_REGION java.lang.RuntimeException: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:117) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2369) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2354) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) ... 4 more 2018-02-04 11:00:40,385 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-1] helpers.MarkerIgnoringBase(159): ***** ABORTING region server ubuntu,39267,1517713227316: Caught throwable while processing event M_RS_CLOSE_REGION ***** java.lang.RuntimeException: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:117) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2369) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2354) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) ... 4 more 2018-02-04 11:00:40,386 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-1] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-02-04 11:00:40,391 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegionServer(2350): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 855638016, "init" : 197132288, "max" : 2794455040, "used" : 643529248 }, "NonHeapMemoryUsage" : { "committed" : 118579200, "init" : 2555904, "max" : -1, "used" : 116276464 }, "Verbose" : false, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : 1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 50164, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 113, "TotalCallTime_min" : 0, "TotalCallTime_max" : 120, "TotalCallTime_mean" : 12, "TotalCallTime_25th_percentile" : 1, "TotalCallTime_median" : 4, "TotalCallTime_75th_percentile" : 12, "TotalCallTime_90th_percentile" : 38, "TotalCallTime_95th_percentile" : 45, "TotalCallTime_98th_percentile" : 52, "TotalCallTime_99th_percentile" : 53, "TotalCallTime_99.9th_percentile" : 120, "TotalCallTime_TimeRangeCount_0-1" : 29, "TotalCallTime_TimeRangeCount_1-3" : 17, "TotalCallTime_TimeRangeCount_3-10" : 36, "TotalCallTime_TimeRangeCount_10-30" : 12, "TotalCallTime_TimeRangeCount_30-100" : 18, "TotalCallTime_TimeRangeCount_100-300" : 1, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 113, "ResponseSize_min" : 0, "ResponseSize_max" : 787, "ResponseSize_mean" : 37, "ResponseSize_25th_percentile" : 10, "ResponseSize_median" : 10, "ResponseSize_75th_percentile" : 51, "ResponseSize_90th_percentile" : 51, "ResponseSize_95th_percentile" : 57, "ResponseSize_98th_percentile" : 161, "ResponseSize_99th_percentile" : 486, "ResponseSize_99.9th_percentile" : 756, "ResponseSize_SizeRangeCount_0-10" : 73, "ResponseSize_SizeRangeCount_10-100" : 36, "ResponseSize_SizeRangeCount_100-1000" : 4, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 113, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 120, "ProcessCallTime_mean" : 9, "ProcessCallTime_25th_percentile" : 1, "ProcessCallTime_median" : 4, "ProcessCallTime_75th_percentile" : 11, "ProcessCallTime_90th_percentile" : 28, "ProcessCallTime_95th_percentile" : 38, "ProcessCallTime_98th_percentile" : 51, "ProcessCallTime_99th_percentile" : 53, "ProcessCallTime_99.9th_percentile" : 120, "ProcessCallTime_TimeRangeCount_0-1" : 36, "ProcessCallTime_TimeRangeCount_1-3" : 15, "ProcessCallTime_TimeRangeCount_3-10" : 33, "ProcessCallTime_TimeRangeCount_10-30" : 19, "ProcessCallTime_TimeRangeCount_30-100" : 9, "ProcessCallTime_TimeRangeCount_100-300" : 1, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 7, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 113, "RequestSize_min" : 70, "RequestSize_max" : 8691, "RequestSize_mean" : 424, "RequestSize_25th_percentile" : 81, "RequestSize_median" : 613, "RequestSize_75th_percentile" : 4652, "RequestSize_90th_percentile" : 7075, "RequestSize_95th_percentile" : 7883, "RequestSize_98th_percentile" : 8367, "RequestSize_99th_percentile" : 8529, "RequestSize_99.9th_percentile" : 8674, "RequestSize_SizeRangeCount_10-100" : 42, "RequestSize_SizeRangeCount_100-1000" : 71, "sentBytes" : 5263, "QueueCallTime_num_ops" : 113, "QueueCallTime_min" : 0, "QueueCallTime_max" : 31, "QueueCallTime_mean" : 2, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 1, "QueueCallTime_90th_percentile" : 9, "QueueCallTime_95th_percentile" : 19, "QueueCallTime_98th_percentile" : 29, "QueueCallTime_99th_percentile" : 29, "QueueCallTime_99.9th_percentile" : 31, "QueueCallTime_TimeRangeCount_0-1" : 92, "QueueCallTime_TimeRangeCount_1-3" : 6, "QueueCallTime_TimeRangeCount_3-10" : 4, "QueueCallTime_TimeRangeCount_10-30" : 10, "QueueCallTime_TimeRangeCount_30-100" : 1, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:56841", "tag.serverName" : "ubuntu,39267,1517713227316", "tag.clusterId" : "990c040f-15f4-4bef-a30c-08f39bc273d8", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "regionCount" : 10, "storeCount" : 1, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 0, "memStoreSize" : 78, "storeFileSize" : 0, "maxStoreFileAge" : 0, "minStoreFileAge" : 0, "avgStoreFileAge" : 0, "numReferenceFiles" : 0, "regionServerStartTime" : 1517713227316, "averageRegionSize" : 78, "totalRequestCount" : 9, "totalRowActionRequestCount" : 4, "readRequestCount" : 2, "filteredReadRequestCount" : 0, "writeRequestCount" : 2, "rpcGetRequestCount" : 4, "rpcScanRequestCount" : 1, "rpcMultiRequestCount" : 0, "rpcMutateRequestCount" : 2, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 0, "staticIndexSize" : 0, "staticBloomSize" : 0, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1116962296, "blockCacheCount" : 0, "blockCacheSize" : 819720, "blockCacheHitCount" : 0, "blockCacheHitCountPrimary" : 0, "blockCacheMissCount" : 0, "blockCacheMissCountPrimary" : 0, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 0.0, "blockCacheExpressHitPercent" : 0.0, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 0, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 0, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 0, "l1CacheMissCount" : 0, "l1CacheHitRatio" : NaN, "l1CacheMissRatio" : NaN, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 0, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 0, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 1, "ScanTime_min" : 1, "ScanTime_max" : 1, "ScanTime_mean" : 1, "ScanTime_25th_percentile" : 1, "ScanTime_median" : 1, "ScanTime_75th_percentile" : 1, "ScanTime_90th_percentile" : 1, "ScanTime_95th_percentile" : 1, "ScanTime_98th_percentile" : 1, "ScanTime_99th_percentile" : 1, "ScanTime_99.9th_percentile" : 1, "ScanTime_TimeRangeCount_0-1" : 1, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 2, "Put_min" : 2, "Put_max" : 9, "Put_mean" : 5, "Put_25th_percentile" : 2, "Put_median" : 3, "Put_75th_percentile" : 9, "Put_90th_percentile" : 9, "Put_95th_percentile" : 9, "Put_98th_percentile" : 9, "Put_99th_percentile" : 9, "Put_99.9th_percentile" : 9, "Put_TimeRangeCount_1-3" : 1, "Put_TimeRangeCount_3-10" : 1, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 0, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 0, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 4, "Get_min" : 0, "Get_max" : 1, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 1, "Get_75th_percentile" : 1, "Get_90th_percentile" : 1, "Get_95th_percentile" : 1, "Get_98th_percentile" : 1, "Get_99th_percentile" : 1, "Get_99.9th_percentile" : 1, "Get_TimeRangeCount_0-1" : 4, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 0, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 0, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 1, "ScanSize_min" : 86, "ScanSize_max" : 86, "ScanSize_mean" : 86, "ScanSize_25th_percentile" : 86, "ScanSize_median" : 86, "ScanSize_75th_percentile" : 86, "ScanSize_90th_percentile" : 86, "ScanSize_95th_percentile" : 86, "ScanSize_98th_percentile" : 86, "ScanSize_99th_percentile" : 86, "ScanSize_99.9th_percentile" : 86, "ScanSize_SizeRangeCount_10-100" : 1, "slowGetCount" : 0, "flushedMemstoreBytes" : 0, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 0, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-02-04 11:00:40,396 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegionFileSystem(463): Committing store file hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/cd5c06da9ebf17554a264cbdd6fcd9ef/.tmp/family/c410acf329e24620ad6b3634c829c1cb as hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/cd5c06da9ebf17554a264cbdd6fcd9ef/family/c410acf329e24620ad6b3634c829c1cb 2018-02-04 11:00:40,403 ERROR [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=39063] master.MasterRpcServices(503): Region server ubuntu,39267,1517713227316 reported a fatal error: ***** ABORTING region server ubuntu,39267,1517713227316: Caught throwable while processing event M_RS_CLOSE_REGION ***** Cause: java.lang.RuntimeException: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:117) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2369) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2354) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) ... 4 more 2018-02-04 11:00:40,404 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] handler.CloseRegionHandler(92): Processing close of tabletest,hhh,1517713232695.ad4f7bcab0a803ddae20b2866466d222. 2018-02-04 11:00:40,404 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(1483): Closing tabletest,hhh,1517713232695.ad4f7bcab0a803ddae20b2866466d222.: disabling compactions & flushes 2018-02-04 11:00:40,405 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(1522): Updates disabled for region tabletest,hhh,1517713232695.ad4f7bcab0a803ddae20b2866466d222. 2018-02-04 11:00:40,416 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HStore(1030): Added hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/cd5c06da9ebf17554a264cbdd6fcd9ef/family/c410acf329e24620ad6b3634c829c1cb, entries=703, sequenceid=9, filesize=32.5 K 2018-02-04 11:00:40,416 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HStore(1030): Added hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/68c9676b485038d63fd2cf8a435420e5/family/6a34fbf7441641b592c77283921544a9, entries=703, sequenceid=9, filesize=32.5 K 2018-02-04 11:00:40,425 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HStore(1030): Added hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/9579a7c2f22113dfe574ba9569101416/family/fa6031c2b1884098801946f475a95d94, entries=703, sequenceid=9, filesize=32.5 K 2018-02-04 11:00:40,440 INFO [StoreCloserThread-tabletest,yyy,1517713232695.20457e81e436062b31548a08737ebba4.-1] regionserver.HStore(932): Closed family 2018-02-04 11:00:40,452 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(2714): Finished memstore flush of ~130.44 KB/133570, currentsize=0 B/0 for region tabletest,uuu,1517713232695.9579a7c2f22113dfe574ba9569101416. in 921ms, sequenceid=9, compaction requested=false 2018-02-04 11:00:40,454 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(2714): Finished memstore flush of ~130.44 KB/133570, currentsize=0 B/0 for region tabletest,ccc,1517713232695.cd5c06da9ebf17554a264cbdd6fcd9ef. in 548ms, sequenceid=9, compaction requested=false 2018-02-04 11:00:40,488 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741881_1058{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW]]} size 33261 2018-02-04 11:00:40,489 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741881_1058 size 33261 2018-02-04 11:00:40,489 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741881_1058 size 33261 2018-02-04 11:00:40,529 INFO [StoreCloserThread-tabletest,hhh,1517713232695.ad4f7bcab0a803ddae20b2866466d222.-1] regionserver.HStore(932): Closed family 2018-02-04 11:00:40,529 INFO [StoreCloserThread-tabletest,uuu,1517713232695.9579a7c2f22113dfe574ba9569101416.-1] regionserver.HStore(932): Closed family 2018-02-04 11:00:40,536 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] wal.WALSplitter(675): Wrote file=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/20457e81e436062b31548a08737ebba4/recovered.edits/12.seqid, newSeqId=12, maxSeqId=2 2018-02-04 11:00:40,549 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741882_1059{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW]]} size 0 2018-02-04 11:00:40,551 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741882_1059{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW]]} size 0 2018-02-04 11:00:40,555 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741882_1059 size 33261 2018-02-04 11:00:40,568 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741883_1060{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619:NORMAL:127.0.0.1:45015|RBW]]} size 33261 2018-02-04 11:00:40,568 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741883_1060 size 33261 2018-02-04 11:00:40,568 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741883_1060 size 33261 2018-02-04 11:00:40,572 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.DefaultStoreFlusher(85): Flushed, sequenceid=9, memsize=130.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/12bc4be2348528b0bd4575ae03200fc3/.tmp/family/33658e5aeb0f453ea1808844a4a4a117 2018-02-04 11:00:40,582 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(1635): Closed tabletest,yyy,1517713232695.20457e81e436062b31548a08737ebba4. 2018-02-04 11:00:40,582 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] handler.CloseRegionHandler(125): Closed tabletest,yyy,1517713232695.20457e81e436062b31548a08737ebba4. 2018-02-04 11:00:40,583 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] handler.CloseRegionHandler(92): Processing close of tabletest,zzz,1517713232695.1da7d397be67579013b1f635bfbcb064. 2018-02-04 11:00:40,583 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(1483): Closing tabletest,zzz,1517713232695.1da7d397be67579013b1f635bfbcb064.: disabling compactions & flushes 2018-02-04 11:00:40,583 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(1522): Updates disabled for region tabletest,zzz,1517713232695.1da7d397be67579013b1f635bfbcb064. 2018-02-04 11:00:40,583 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(2518): Flushing 1/1 column families, memstore=190 B 2018-02-04 11:00:40,584 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.CompactingMemStore(205): FLUSHING TO DISK: region tabletest,zzz,1517713232695.1da7d397be67579013b1f635bfbcb064.store: family 2018-02-04 11:00:40,585 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.CompactionPipeline(128): Swapping pipeline suffix; before=1, new segement=null 2018-02-04 11:00:40,592 INFO [StoreCloserThread-tabletest,ccc,1517713232695.cd5c06da9ebf17554a264cbdd6fcd9ef.-1] regionserver.HStore(932): Closed family 2018-02-04 11:00:40,594 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/9579a7c2f22113dfe574ba9569101416/recovered.edits/12.seqid, newSeqId=12, maxSeqId=2 2018-02-04 11:00:40,599 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(1635): Closed tabletest,uuu,1517713232695.9579a7c2f22113dfe574ba9569101416. 2018-02-04 11:00:40,599 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] handler.CloseRegionHandler(125): Closed tabletest,uuu,1517713232695.9579a7c2f22113dfe574ba9569101416. 2018-02-04 11:00:40,617 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegionFileSystem(463): Committing store file hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/12bc4be2348528b0bd4575ae03200fc3/.tmp/family/33658e5aeb0f453ea1808844a4a4a117 as hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/12bc4be2348528b0bd4575ae03200fc3/family/33658e5aeb0f453ea1808844a4a4a117 2018-02-04 11:00:40,623 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] wal.WALSplitter(675): Wrote file=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/cd5c06da9ebf17554a264cbdd6fcd9ef/recovered.edits/12.seqid, newSeqId=12, maxSeqId=2 2018-02-04 11:00:40,636 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(1635): Closed tabletest,ccc,1517713232695.cd5c06da9ebf17554a264cbdd6fcd9ef. 2018-02-04 11:00:40,636 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] handler.CloseRegionHandler(125): Closed tabletest,ccc,1517713232695.cd5c06da9ebf17554a264cbdd6fcd9ef. 2018-02-04 11:00:40,637 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] handler.CloseRegionHandler(92): Processing close of tabletest,jjj,1517713232695.e4481a648e0b377eb5bf59bfb99dfd88. 2018-02-04 11:00:40,637 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(1483): Closing tabletest,jjj,1517713232695.e4481a648e0b377eb5bf59bfb99dfd88.: disabling compactions & flushes 2018-02-04 11:00:40,637 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(1522): Updates disabled for region tabletest,jjj,1517713232695.e4481a648e0b377eb5bf59bfb99dfd88. 2018-02-04 11:00:40,637 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(2518): Flushing 1/1 column families, memstore=130.44 KB 2018-02-04 11:00:40,637 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.CompactingMemStore(205): FLUSHING TO DISK: region tabletest,jjj,1517713232695.e4481a648e0b377eb5bf59bfb99dfd88.store: family 2018-02-04 11:00:40,637 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.CompactionPipeline(128): Swapping pipeline suffix; before=1, new segement=null 2018-02-04 11:00:40,650 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HStore(1030): Added hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/12bc4be2348528b0bd4575ae03200fc3/family/33658e5aeb0f453ea1808844a4a4a117, entries=703, sequenceid=9, filesize=32.5 K 2018-02-04 11:00:40,652 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(2714): Finished memstore flush of ~130.44 KB/133570, currentsize=0 B/0 for region tabletest,ppp,1517713232695.12bc4be2348528b0bd4575ae03200fc3. in 383ms, sequenceid=9, compaction requested=false 2018-02-04 11:00:40,664 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741884_1061{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|FINALIZED]]} size 0 2018-02-04 11:00:40,665 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741884_1061{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|FINALIZED]]} size 0 2018-02-04 11:00:40,666 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741884_1061{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|FINALIZED], ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|FINALIZED]]} size 0 2018-02-04 11:00:40,674 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.DefaultStoreFlusher(85): Flushed, sequenceid=9, memsize=190, hasBloomFilter=true, into tmp file hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/1da7d397be67579013b1f635bfbcb064/.tmp/family/d342fdd681854ef3a1c0ad8f6bece25e 2018-02-04 11:00:40,683 INFO [StoreCloserThread-tabletest,ppp,1517713232695.12bc4be2348528b0bd4575ae03200fc3.-1] regionserver.HStore(932): Closed family 2018-02-04 11:00:40,709 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegionFileSystem(463): Committing store file hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/1da7d397be67579013b1f635bfbcb064/.tmp/family/d342fdd681854ef3a1c0ad8f6bece25e as hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/1da7d397be67579013b1f635bfbcb064/family/d342fdd681854ef3a1c0ad8f6bece25e 2018-02-04 11:00:40,715 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741885_1062{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW]]} size 0 2018-02-04 11:00:40,716 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741885_1062{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW]]} size 0 2018-02-04 11:00:40,718 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741885_1062 size 33261 2018-02-04 11:00:40,720 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] wal.WALSplitter(675): Wrote file=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/12bc4be2348528b0bd4575ae03200fc3/recovered.edits/12.seqid, newSeqId=12, maxSeqId=2 2018-02-04 11:00:40,720 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.DefaultStoreFlusher(85): Flushed, sequenceid=9, memsize=130.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/e4481a648e0b377eb5bf59bfb99dfd88/.tmp/family/bf06f57326f046c9ab6f7e300bad230d 2018-02-04 11:00:40,724 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(1635): Closed tabletest,ppp,1517713232695.12bc4be2348528b0bd4575ae03200fc3. 2018-02-04 11:00:40,725 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] handler.CloseRegionHandler(125): Closed tabletest,ppp,1517713232695.12bc4be2348528b0bd4575ae03200fc3. 2018-02-04 11:00:40,725 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HStore(1030): Added hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/1da7d397be67579013b1f635bfbcb064/family/d342fdd681854ef3a1c0ad8f6bece25e, entries=1, sequenceid=9, filesize=4.7 K 2018-02-04 11:00:40,726 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(2714): Finished memstore flush of ~190 B/190, currentsize=0 B/0 for region tabletest,zzz,1517713232695.1da7d397be67579013b1f635bfbcb064. in 143ms, sequenceid=9, compaction requested=false 2018-02-04 11:00:40,743 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegionFileSystem(463): Committing store file hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/e4481a648e0b377eb5bf59bfb99dfd88/.tmp/family/bf06f57326f046c9ab6f7e300bad230d as hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/e4481a648e0b377eb5bf59bfb99dfd88/family/bf06f57326f046c9ab6f7e300bad230d 2018-02-04 11:00:40,754 INFO [StoreCloserThread-tabletest,zzz,1517713232695.1da7d397be67579013b1f635bfbcb064.-1] regionserver.HStore(932): Closed family 2018-02-04 11:00:40,756 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HStore(1030): Added hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/e4481a648e0b377eb5bf59bfb99dfd88/family/bf06f57326f046c9ab6f7e300bad230d, entries=703, sequenceid=9, filesize=32.5 K 2018-02-04 11:00:40,761 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(2714): Finished memstore flush of ~130.44 KB/133570, currentsize=0 B/0 for region tabletest,jjj,1517713232695.e4481a648e0b377eb5bf59bfb99dfd88. in 123ms, sequenceid=9, compaction requested=false 2018-02-04 11:00:40,768 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] wal.WALSplitter(675): Wrote file=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/1da7d397be67579013b1f635bfbcb064/recovered.edits/12.seqid, newSeqId=12, maxSeqId=2 2018-02-04 11:00:40,774 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(1635): Closed tabletest,zzz,1517713232695.1da7d397be67579013b1f635bfbcb064. 2018-02-04 11:00:40,775 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] handler.CloseRegionHandler(125): Closed tabletest,zzz,1517713232695.1da7d397be67579013b1f635bfbcb064. 2018-02-04 11:00:40,775 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] handler.CloseRegionHandler(92): Processing close of tabletest,www,1517713232695.7514f60148f3a526baae5c7ce2c78c1b. 2018-02-04 11:00:40,775 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(1483): Closing tabletest,www,1517713232695.7514f60148f3a526baae5c7ce2c78c1b.: disabling compactions & flushes 2018-02-04 11:00:40,775 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(1522): Updates disabled for region tabletest,www,1517713232695.7514f60148f3a526baae5c7ce2c78c1b. 2018-02-04 11:00:40,777 INFO [StoreCloserThread-tabletest,jjj,1517713232695.e4481a648e0b377eb5bf59bfb99dfd88.-1] regionserver.HStore(932): Closed family 2018-02-04 11:00:40,785 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(2518): Flushing 1/1 column families, memstore=130.44 KB 2018-02-04 11:00:40,786 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.CompactingMemStore(205): FLUSHING TO DISK: region tabletest,www,1517713232695.7514f60148f3a526baae5c7ce2c78c1b.store: family 2018-02-04 11:00:40,786 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.CompactionPipeline(128): Swapping pipeline suffix; before=1, new segement=null 2018-02-04 11:00:40,803 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] wal.WALSplitter(675): Wrote file=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/e4481a648e0b377eb5bf59bfb99dfd88/recovered.edits/12.seqid, newSeqId=12, maxSeqId=2 2018-02-04 11:00:40,808 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(1635): Closed tabletest,jjj,1517713232695.e4481a648e0b377eb5bf59bfb99dfd88. 2018-02-04 11:00:40,809 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] handler.CloseRegionHandler(125): Closed tabletest,jjj,1517713232695.e4481a648e0b377eb5bf59bfb99dfd88. 2018-02-04 11:00:40,833 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741886_1063{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|RBW]]} size 33261 2018-02-04 11:00:40,835 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741886_1063 size 33261 2018-02-04 11:00:40,836 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741886_1063 size 33261 2018-02-04 11:00:40,895 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.DefaultStoreFlusher(85): Flushed, sequenceid=9, memsize=130.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/85d5a900ae52e80b00b89bb74335621c/.tmp/family/e3ea7532263a4e16abbe0b748c859fa6 2018-02-04 11:00:40,924 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegionFileSystem(463): Committing store file hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/85d5a900ae52e80b00b89bb74335621c/.tmp/family/e3ea7532263a4e16abbe0b748c859fa6 as hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/85d5a900ae52e80b00b89bb74335621c/family/e3ea7532263a4e16abbe0b748c859fa6 2018-02-04 11:00:40,945 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HStore(1030): Added hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/85d5a900ae52e80b00b89bb74335621c/family/e3ea7532263a4e16abbe0b748c859fa6, entries=703, sequenceid=9, filesize=32.5 K 2018-02-04 11:00:40,954 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(2714): Finished memstore flush of ~130.44 KB/133570, currentsize=0 B/0 for region tabletest,lll,1517713232695.85d5a900ae52e80b00b89bb74335621c. in 850ms, sequenceid=9, compaction requested=false 2018-02-04 11:00:40,987 INFO [RS:2;ubuntu:36383] regionserver.HRegionServer(1383): Waiting on 1 regions to close 2018-02-04 11:00:40,987 DEBUG [RS:2;ubuntu:36383] regionserver.HRegionServer(1387): Online Regions={756819a3f903733cb342d3913aab52a6=tabletest,bbb,1517713232695.756819a3f903733cb342d3913aab52a6.} 2018-02-04 11:00:40,989 INFO [StoreCloserThread-tabletest,lll,1517713232695.85d5a900ae52e80b00b89bb74335621c.-1] regionserver.HStore(932): Closed family 2018-02-04 11:00:40,990 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.DefaultStoreFlusher(85): Flushed, sequenceid=9, memsize=130.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/756819a3f903733cb342d3913aab52a6/.tmp/family/fa1d7189751c40fcbfe026c8654d65d1 2018-02-04 11:00:41,008 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/85d5a900ae52e80b00b89bb74335621c/recovered.edits/12.seqid, newSeqId=12, maxSeqId=2 2018-02-04 11:00:41,014 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(1635): Closed tabletest,lll,1517713232695.85d5a900ae52e80b00b89bb74335621c. 2018-02-04 11:00:41,014 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] handler.CloseRegionHandler(125): Closed tabletest,lll,1517713232695.85d5a900ae52e80b00b89bb74335621c. 2018-02-04 11:00:41,016 INFO [RS:1;ubuntu:46739] regionserver.HRegionServer(1383): Waiting on 1 regions to close 2018-02-04 11:00:41,017 DEBUG [RS:1;ubuntu:46739] regionserver.HRegionServer(1387): Online Regions={7514f60148f3a526baae5c7ce2c78c1b=tabletest,www,1517713232695.7514f60148f3a526baae5c7ce2c78c1b.} 2018-02-04 11:00:41,019 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegionFileSystem(463): Committing store file hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/756819a3f903733cb342d3913aab52a6/.tmp/family/fa1d7189751c40fcbfe026c8654d65d1 as hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/756819a3f903733cb342d3913aab52a6/family/fa1d7189751c40fcbfe026c8654d65d1 2018-02-04 11:00:41,030 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HStore(1030): Added hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/756819a3f903733cb342d3913aab52a6/family/fa1d7189751c40fcbfe026c8654d65d1, entries=703, sequenceid=9, filesize=32.5 K 2018-02-04 11:00:41,032 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(2714): Finished memstore flush of ~130.44 KB/133570, currentsize=0 B/0 for region tabletest,bbb,1517713232695.756819a3f903733cb342d3913aab52a6. in 765ms, sequenceid=9, compaction requested=false 2018-02-04 11:00:41,037 INFO [StoreCloserThread-tabletest,bbb,1517713232695.756819a3f903733cb342d3913aab52a6.-1] regionserver.HStore(932): Closed family 2018-02-04 11:00:41,058 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] wal.WALSplitter(675): Wrote file=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/756819a3f903733cb342d3913aab52a6/recovered.edits/12.seqid, newSeqId=12, maxSeqId=2 2018-02-04 11:00:41,061 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegion(1635): Closed tabletest,bbb,1517713232695.756819a3f903733cb342d3913aab52a6. 2018-02-04 11:00:41,062 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] handler.CloseRegionHandler(125): Closed tabletest,bbb,1517713232695.756819a3f903733cb342d3913aab52a6. 2018-02-04 11:00:41,188 INFO [RS:2;ubuntu:36383] regionserver.HRegionServer(1104): stopping server ubuntu,36383,1517713227466; all regions closed. 2018-02-04 11:00:41,203 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741831_1007{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|RBW]]} size 0 2018-02-04 11:00:41,203 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741831_1007{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|RBW]]} size 0 2018-02-04 11:00:41,214 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741831_1007{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW], ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|RBW]]} size 0 2018-02-04 11:00:41,222 DEBUG [RS:2;ubuntu:36383] wal.AbstractFSWAL(861): Moved 1 WAL file(s) to /user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/oldWALs 2018-02-04 11:00:41,222 INFO [RS:2;ubuntu:36383] wal.AbstractFSWAL(864): Closed WAL: AsyncFSWAL ubuntu%2C36383%2C1517713227466:(num 1517713230056) 2018-02-04 11:00:41,222 DEBUG [RS:2;ubuntu:36383] ipc.AbstractRpcClient(483): Stopping rpc client 2018-02-04 11:00:41,222 INFO [RS:2;ubuntu:36383] regionserver.Leases(147): RS:2;ubuntu:36383 closing leases 2018-02-04 11:00:41,222 INFO [RS:2;ubuntu:36383] regionserver.Leases(150): RS:2;ubuntu:36383 closed leases 2018-02-04 11:00:41,223 INFO [RS:2;ubuntu:36383] hbase.ChoreService(327): Chore service for: regionserver/ubuntu:0 had [[ScheduledChore: Name: CompactedHFilesCleaner Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: MovedRegionsCleaner for region ubuntu,36383,1517713227466 Period: 120000 Unit: MILLISECONDS]] on shutdown 2018-02-04 11:00:41,223 INFO [RS:2;ubuntu:36383] regionserver.CompactSplit(394): Waiting for Split Thread to finish... 2018-02-04 11:00:41,223 INFO [regionserver/ubuntu:0.logRoller] regionserver.LogRoller(195): LogRoller exiting. 2018-02-04 11:00:41,223 INFO [RS:2;ubuntu:36383] regionserver.CompactSplit(394): Waiting for Large Compaction Thread to finish... 2018-02-04 11:00:41,223 INFO [RS:2;ubuntu:36383] regionserver.CompactSplit(394): Waiting for Small Compaction Thread to finish... 2018-02-04 11:00:41,223 DEBUG [RS_LOG_REPLAY_OPS-regionserver/ubuntu:0-0] wal.WALSplitter(336): Finishing writing output logs and closing down 2018-02-04 11:00:41,224 INFO [RS_LOG_REPLAY_OPS-regionserver/ubuntu:0-0] wal.WALSplitter(357): Processed 0 edits across 0 regions; edits skipped=0; log file=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/WALs/ubuntu,39267,1517713227316-splitting/ubuntu%2C39267%2C1517713227316.1517713230044, length=0, corrupted=false, progress failed=false 2018-02-04 11:00:41,224 WARN [RS_LOG_REPLAY_OPS-regionserver/ubuntu:0-0] regionserver.SplitLogWorker$1(105): log splitting of WALs/ubuntu,39267,1517713227316-splitting/ubuntu%2C39267%2C1517713227316.1517713230044 interrupted, resigning java.io.InterruptedIOException at org.apache.hadoop.hbase.util.FSHDFSUtils.recoverDFSFileLease(FSHDFSUtils.java:249) at org.apache.hadoop.hbase.util.FSHDFSUtils.recoverFileLease(FSHDFSUtils.java:163) at org.apache.hadoop.hbase.wal.WALSplitter.getReader(WALSplitter.java:714) at org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:257) at org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:196) at org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:99) at org.apache.hadoop.hbase.regionserver.handler.WALSplitterHandler.process(WALSplitterHandler.java:70) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.InterruptedException: sleep interrupted at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.hbase.util.FSHDFSUtils.recoverDFSFileLease(FSHDFSUtils.java:223) ... 10 more 2018-02-04 11:00:41,225 INFO [RS:2;ubuntu:36383] ipc.NettyRpcServer(143): Stopping server on 0 2018-02-04 11:00:41,226 INFO [RS_LOG_REPLAY_OPS-regionserver/ubuntu:0-0] handler.WALSplitterHandler(91): task execution interrupted because worker is exiting org.apache.hadoop.hbase.coordination.ZkSplitLogWorkerCoordination$ZkSplitTaskDetails@7c24a8ac 2018-02-04 11:00:41,227 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(443): master:39063-0x1615ec1f6c10000, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fubuntu%2C39267%2C1517713227316-splitting%2Fubuntu%252C39267%252C1517713227316.1517713230044 2018-02-04 11:00:41,227 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(443): regionserver:36383-0x1615ec1f6c10003, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fubuntu%2C39267%2C1517713227316-splitting%2Fubuntu%252C39267%252C1517713227316.1517713230044 2018-02-04 11:00:41,227 INFO [RS_LOG_REPLAY_OPS-regionserver/ubuntu:0-0] coordination.ZkSplitLogWorkerCoordination(574): successfully transitioned task /hbase/splitWAL/WALs%2Fubuntu%2C39267%2C1517713227316-splitting%2Fubuntu%252C39267%252C1517713227316.1517713230044 to final state RESIGNED ubuntu,36383,1517713227466 2018-02-04 11:00:41,227 INFO [RS_LOG_REPLAY_OPS-regionserver/ubuntu:0-0] handler.WALSplitterHandler(99): Worker ubuntu,36383,1517713227466 done with task org.apache.hadoop.hbase.coordination.ZkSplitLogWorkerCoordination$ZkSplitTaskDetails@7c24a8ac in 2443ms. Status = RESIGNED 2018-02-04 11:00:41,227 INFO [Time-limited test-EventThread] coordination.ZKSplitLogManagerCoordination(368): Task /hbase/splitWAL/WALs%2Fubuntu%2C39267%2C1517713227316-splitting%2Fubuntu%252C39267%252C1517713227316.1517713230044 entered state=RESIGNED ubuntu,36383,1517713227466 2018-02-04 11:00:41,228 INFO [Time-limited test-EventThread] coordination.ZKSplitLogManagerCoordination(205): Resubmitting task /hbase/splitWAL/WALs%2Fubuntu%2C39267%2C1517713227316-splitting%2Fubuntu%252C39267%252C1517713227316.1517713230044 2018-02-04 11:00:41,229 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(443): master:39063-0x1615ec1f6c10000, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fubuntu%2C39267%2C1517713227316-splitting%2Fubuntu%252C39267%252C1517713227316.1517713230044 2018-02-04 11:00:41,230 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(443): regionserver:36383-0x1615ec1f6c10003, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2018-02-04 11:00:41,230 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(443): regionserver:46739-0x1615ec1f6c10002, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2018-02-04 11:00:41,230 DEBUG [Time-limited test-EventThread] coordination.ZKSplitLogManagerCoordination(363): Task not yet acquired /hbase/splitWAL/WALs%2Fubuntu%2C39267%2C1517713227316-splitting%2Fubuntu%252C39267%252C1517713227316.1517713230044, ver=4 2018-02-04 11:00:41,232 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.DefaultStoreFlusher(85): Flushed, sequenceid=9, memsize=130.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/7514f60148f3a526baae5c7ce2c78c1b/.tmp/family/2b361bf1a2914d96b06101787e8eda53 2018-02-04 11:00:41,236 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(443): regionserver:46739-0x1615ec1f6c10002, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/ubuntu,36383,1517713227466 2018-02-04 11:00:41,236 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(443): regionserver:36383-0x1615ec1f6c10003, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/ubuntu,36383,1517713227466 2018-02-04 11:00:41,236 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(443): master:39063-0x1615ec1f6c10000, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/ubuntu,36383,1517713227466 2018-02-04 11:00:41,236 INFO [Time-limited test-EventThread] master.RegionServerTracker(122): RegionServer ephemeral node deleted, processing expiration [ubuntu,36383,1517713227466] 2018-02-04 11:00:41,237 INFO [Time-limited test-EventThread] master.ServerManager(588): Cluster shutdown set; ubuntu,36383,1517713227466 expired; onlineServers=1 2018-02-04 11:00:41,237 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(443): regionserver:46739-0x1615ec1f6c10002, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-02-04 11:00:41,237 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(443): regionserver:36383-0x1615ec1f6c10003, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-02-04 11:00:41,238 INFO [RS:2;ubuntu:36383] regionserver.HRegionServer(1150): stopping server ubuntu,36383,1517713227466; zookeeper connection closed. 2018-02-04 11:00:41,238 INFO [RS:2;ubuntu:36383] regionserver.HRegionServer(1152): RS:2;ubuntu:36383 exiting 2018-02-04 11:00:41,240 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(443): master:39063-0x1615ec1f6c10000, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-02-04 11:00:41,241 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@2cb3d99] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(225): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@2cb3d99 2018-02-04 11:00:41,249 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:39063-0x1615ec1f6c10000, quorum=localhost:56841, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ubuntu,46739,1517713227398 2018-02-04 11:00:41,250 INFO [Time-limited test-EventThread] coordination.ZKSplitLogManagerCoordination(371): Task /hbase/splitWAL/RESCAN0000000001 entered state=DONE ubuntu,39063,1517713226282 2018-02-04 11:00:41,253 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(443): master:39063-0x1615ec1f6c10000, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/splitWAL/RESCAN0000000001 2018-02-04 11:00:41,253 DEBUG [Time-limited test-EventThread] coordination.ZKSplitLogManagerCoordination$DeleteAsyncCallback(694): Deleted /hbase/splitWAL/RESCAN0000000001 2018-02-04 11:00:41,253 DEBUG [Time-limited test-EventThread] coordination.ZKSplitLogManagerCoordination(283): Deleted task without in memory state /hbase/splitWAL/RESCAN0000000001 2018-02-04 11:00:41,256 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegionFileSystem(463): Committing store file hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/7514f60148f3a526baae5c7ce2c78c1b/.tmp/family/2b361bf1a2914d96b06101787e8eda53 as hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/7514f60148f3a526baae5c7ce2c78c1b/family/2b361bf1a2914d96b06101787e8eda53 2018-02-04 11:00:41,265 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HStore(1030): Added hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/7514f60148f3a526baae5c7ce2c78c1b/family/2b361bf1a2914d96b06101787e8eda53, entries=703, sequenceid=9, filesize=32.5 K 2018-02-04 11:00:41,270 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(2714): Finished memstore flush of ~130.44 KB/133570, currentsize=0 B/0 for region tabletest,www,1517713232695.7514f60148f3a526baae5c7ce2c78c1b. in 494ms, sequenceid=9, compaction requested=false 2018-02-04 11:00:41,272 INFO [StoreCloserThread-tabletest,www,1517713232695.7514f60148f3a526baae5c7ce2c78c1b.-1] regionserver.HStore(932): Closed family 2018-02-04 11:00:41,289 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] wal.WALSplitter(675): Wrote file=hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/data/default/tabletest/7514f60148f3a526baae5c7ce2c78c1b/recovered.edits/12.seqid, newSeqId=12, maxSeqId=2 2018-02-04 11:00:41,294 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(1635): Closed tabletest,www,1517713232695.7514f60148f3a526baae5c7ce2c78c1b. 2018-02-04 11:00:41,294 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] handler.CloseRegionHandler(125): Closed tabletest,www,1517713232695.7514f60148f3a526baae5c7ce2c78c1b. 2018-02-04 11:00:41,418 INFO [RS:1;ubuntu:46739] regionserver.HRegionServer(1104): stopping server ubuntu,46739,1517713227398; all regions closed. 2018-02-04 11:00:41,435 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741833_1009{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619:NORMAL:127.0.0.1:45015|RBW]]} size 0 2018-02-04 11:00:41,439 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741833_1009{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619:NORMAL:127.0.0.1:45015|RBW]]} size 0 2018-02-04 11:00:41,441 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741833_1009 size 35658 2018-02-04 11:00:41,446 DEBUG [RS:1;ubuntu:46739] wal.AbstractFSWAL(861): Moved 1 WAL file(s) to /user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/oldWALs 2018-02-04 11:00:41,446 INFO [RS:1;ubuntu:46739] wal.AbstractFSWAL(864): Closed WAL: AsyncFSWAL ubuntu%2C46739%2C1517713227398.meta:.meta(num 1517713230529) 2018-02-04 11:00:41,451 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741830_1006{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-1fd5fc50-1375-4985-9716-c32596d90f81:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-dd1fcbd6-0c82-4629-949f-0463aa9af693:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-f1f2b1f7-9b2c-4e73-9c82-9d4c4df317b1:NORMAL:127.0.0.1:45015|RBW]]} size 0 2018-02-04 11:00:41,453 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741830_1006 size 1193257 2018-02-04 11:00:41,453 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741830_1006 size 1193257 2018-02-04 11:00:41,455 DEBUG [RS:1;ubuntu:46739] wal.AbstractFSWAL(861): Moved 1 WAL file(s) to /user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/oldWALs 2018-02-04 11:00:41,455 INFO [RS:1;ubuntu:46739] wal.AbstractFSWAL(864): Closed WAL: AsyncFSWAL ubuntu%2C46739%2C1517713227398:(num 1517713230043) 2018-02-04 11:00:41,455 DEBUG [RS:1;ubuntu:46739] ipc.AbstractRpcClient(483): Stopping rpc client 2018-02-04 11:00:41,455 INFO [RS:1;ubuntu:46739] regionserver.Leases(147): RS:1;ubuntu:46739 closing leases 2018-02-04 11:00:41,455 INFO [RS:1;ubuntu:46739] regionserver.Leases(150): RS:1;ubuntu:46739 closed leases 2018-02-04 11:00:41,456 INFO [RS:1;ubuntu:46739] hbase.ChoreService(327): Chore service for: regionserver/ubuntu:0 had [[ScheduledChore: Name: CompactedHFilesCleaner Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: MovedRegionsCleaner for region ubuntu,46739,1517713227398 Period: 120000 Unit: MILLISECONDS]] on shutdown 2018-02-04 11:00:41,457 INFO [regionserver/ubuntu:0.logRoller] regionserver.LogRoller(195): LogRoller exiting. 2018-02-04 11:00:41,459 INFO [RS:1;ubuntu:46739] ipc.NettyRpcServer(143): Stopping server on 0 2018-02-04 11:00:41,465 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(443): regionserver:46739-0x1615ec1f6c10002, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/ubuntu,46739,1517713227398 2018-02-04 11:00:41,465 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(443): master:39063-0x1615ec1f6c10000, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/ubuntu,46739,1517713227398 2018-02-04 11:00:41,465 INFO [Time-limited test-EventThread] master.RegionServerTracker(122): RegionServer ephemeral node deleted, processing expiration [ubuntu,46739,1517713227398] 2018-02-04 11:00:41,465 INFO [Time-limited test-EventThread] master.ServerManager(588): Cluster shutdown set; ubuntu,46739,1517713227398 expired; onlineServers=0 2018-02-04 11:00:41,465 INFO [Time-limited test-EventThread] regionserver.HRegionServer(2127): ***** STOPPING region server 'ubuntu,39063,1517713226282' ***** 2018-02-04 11:00:41,465 INFO [Time-limited test-EventThread] regionserver.HRegionServer(2141): STOPPED: Cluster shutdown set; onlineServer=0 2018-02-04 11:00:41,466 DEBUG [M:0;ubuntu:39063] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@184d9e1e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=1, fallbackAllowed=true, bind address=ubuntu/127.0.1.1:0 2018-02-04 11:00:41,466 INFO [M:0;ubuntu:39063] regionserver.HRegionServer(1076): stopping server ubuntu,39063,1517713226282 2018-02-04 11:00:41,466 DEBUG [M:0;ubuntu:39063] zookeeper.MetaTableLocator(641): Stopping MetaTableLocator 2018-02-04 11:00:41,466 INFO [M:0;ubuntu:39063] regionserver.HRegionServer(1104): stopping server ubuntu,39063,1517713226282; all regions closed. 2018-02-04 11:00:41,466 DEBUG [M:0;ubuntu:39063] ipc.AbstractRpcClient(483): Stopping rpc client 2018-02-04 11:00:41,466 INFO [M:0;ubuntu:39063] hbase.ChoreService(327): Chore service for: master/ubuntu:0 had [[ScheduledChore: Name: LogsCleaner Period: 600000 Unit: MILLISECONDS], [ScheduledChore: Name: CatalogJanitor-ubuntu:39063 Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: ReplicationMetaCleaner Period: 600000 Unit: MILLISECONDS], [ScheduledChore: Name: ubuntu,39063,1517713226282-ExpiredMobFileCleanerChore Period: 86400 Unit: SECONDS], [ScheduledChore: Name: ubuntu,39063,1517713226282-RegionNormalizerChore Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: ubuntu,39063,1517713226282-BalancerChore Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: HFileCleaner Period: 600000 Unit: MILLISECONDS], [ScheduledChore: Name: ubuntu,39063,1517713226282-MobCompactionChore Period: 604800 Unit: SECONDS], [ScheduledChore: Name: ubuntu,39063,1517713226282-ClusterStatusChore Period: 60000 Unit: MILLISECONDS]] on shutdown 2018-02-04 11:00:41,466 INFO [M:0;ubuntu:39063] master.MasterMobCompactionThread(175): Waiting for Mob Compaction Thread to finish... 2018-02-04 11:00:41,466 INFO [M:0;ubuntu:39063] master.MasterMobCompactionThread(175): Waiting for Region Server Mob Compaction Thread to finish... 2018-02-04 11:00:41,466 DEBUG [M:0;ubuntu:39063] master.HMaster(1192): Stopping service threads 2018-02-04 11:00:41,467 DEBUG [OldWALsCleaner-0] cleaner.LogCleaner(176): Exiting cleaner. 2018-02-04 11:00:41,467 DEBUG [OldWALsCleaner-1] cleaner.LogCleaner(176): Exiting cleaner. 2018-02-04 11:00:41,467 DEBUG [M:0;ubuntu:39063-HFileCleaner.large.0-1517713228419] cleaner.HFileCleaner(253): Exit thread: Thread[M:0;ubuntu:39063-HFileCleaner.large.0-1517713228419,5,FailOnTimeoutGroup] 2018-02-04 11:00:41,467 DEBUG [M:0;ubuntu:39063-HFileCleaner.small.0-1517713228420] cleaner.HFileCleaner(253): Exit thread: Thread[M:0;ubuntu:39063-HFileCleaner.small.0-1517713228420,5,FailOnTimeoutGroup] 2018-02-04 11:00:41,470 INFO [RS:1;ubuntu:46739] regionserver.HRegionServer(1150): stopping server ubuntu,46739,1517713227398; zookeeper connection closed. 2018-02-04 11:00:41,470 INFO [RS:1;ubuntu:46739] regionserver.HRegionServer(1152): RS:1;ubuntu:46739 exiting 2018-02-04 11:00:41,473 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@559016f7] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(225): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@559016f7 2018-02-04 11:00:41,473 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(443): master:39063-0x1615ec1f6c10000, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-02-04 11:00:41,473 DEBUG [M:0;ubuntu:39063] zookeeper.RecoverableZooKeeper(176): Node /hbase/master already deleted, retry=false 2018-02-04 11:00:41,474 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(443): master:39063-0x1615ec1f6c10000, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-02-04 11:00:41,474 DEBUG [M:0;ubuntu:39063] master.ActiveMasterManager(280): master:39063-0x1615ec1f6c10000, quorum=localhost:56841, baseZNode=/hbase Failed delete of our master address node; KeeperErrorCode = NoNode for /hbase/master 2018-02-04 11:00:41,474 INFO [M:0;ubuntu:39063] assignment.AssignmentManager(237): Stopping assignment manager 2018-02-04 11:00:41,474 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(357): master:39063-0x1615ec1f6c10000, quorum=localhost:56841, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-02-04 11:00:41,476 INFO [M:0;ubuntu:39063] zookeeper.ReadOnlyZKClient(344): Close zookeeper connection 0x5db12a7f to localhost:56841 2018-02-04 11:00:41,477 DEBUG [M:0;ubuntu:39063] ipc.AbstractRpcClient(483): Stopping rpc client 2018-02-04 11:00:41,489 INFO [M:0;ubuntu:39063] procedure2.RemoteProcedureDispatcher(116): Stopping procedure remote dispatcher 2018-02-04 11:00:41,539 WARN [PEWorker-14] master.SplitLogManager(341): Stopped while waiting for log splits to be completed 2018-02-04 11:00:41,539 WARN [PEWorker-14] master.SplitLogManager(263): error while splitting logs in [hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/WALs/ubuntu,39267,1517713227316-splitting] installed = 1 but only 0 done 2018-02-04 11:00:41,539 WARN [PEWorker-14] procedure.ServerCrashProcedure(181): Failed state=SERVER_CRASH_SPLIT_LOGS, retry pid=34, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS; ServerCrashProcedure server=ubuntu,39267,1517713227316, splitWal=true, meta=false; cycles=0 java.io.IOException: error or interrupted while splitting logs in [hdfs://localhost:44491/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/WALs/ubuntu,39267,1517713227316-splitting] Task = installed = 1 done = 0 error = 0 at org.apache.hadoop.hbase.master.SplitLogManager.splitLogDistributed(SplitLogManager.java:267) at org.apache.hadoop.hbase.master.MasterWalManager.splitLog(MasterWalManager.java:293) at org.apache.hadoop.hbase.master.MasterWalManager.splitLog(MasterWalManager.java:278) at org.apache.hadoop.hbase.master.MasterWalManager.splitLog(MasterWalManager.java:216) at org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure.splitLogs(ServerCrashProcedure.java:224) at org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure.executeFromState(ServerCrashProcedure.java:150) at org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure.executeFromState(ServerCrashProcedure.java:1) at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:180) at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1455) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1224) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$6(ProcedureExecutor.java:1162) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1734) 2018-02-04 11:00:41,546 ERROR [M:0;ubuntu:39063] procedure2.ProcedureExecutor(594): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[ReadOnlyZKClient-localhost:56841@0x5db12a7f-SendThread(localhost:56841),5,PEWorkerGroup] 2018-02-04 11:00:41,547 INFO [M:0;ubuntu:39063] wal.WALProcedureStore(290): Stopping the WAL Procedure Store, isAbort=false 2018-02-04 11:00:41,569 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45015 is added to blk_1073741829_1005{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619:NORMAL:127.0.0.1:45015|RBW]]} size 214 2018-02-04 11:00:41,573 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46483 is added to blk_1073741829_1005{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619:NORMAL:127.0.0.1:45015|RBW]]} size 214 2018-02-04 11:00:41,574 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33785 is added to blk_1073741829_1005{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-44dd590d-1f51-400a-b8c2-20d260aaa289:NORMAL:127.0.0.1:46483|RBW], ReplicaUC[[DISK]DS-7e56e822-3f2b-4b63-a673-acf15dd3ab33:NORMAL:127.0.0.1:33785|RBW], ReplicaUC[[DISK]DS-1d0a207a-eb09-4c99-b0a8-fd1dd2cef619:NORMAL:127.0.0.1:45015|RBW]]} size 214 2018-02-04 11:00:41,577 INFO [M:0;ubuntu:39063] hbase.ChoreService(327): Chore service for: master/ubuntu:0.splitLogManager. had [[ScheduledChore: Name: SplitLogManager Timeout Monitor Period: 1000 Unit: MILLISECONDS]] on shutdown 2018-02-04 11:00:41,577 INFO [M:0;ubuntu:39063] flush.MasterFlushTableProcedureManager(78): stop: server shutting down. 2018-02-04 11:00:41,578 INFO [M:0;ubuntu:39063] ipc.NettyRpcServer(143): Stopping server on 0 2018-02-04 11:00:41,581 DEBUG [M:0;ubuntu:39063] zookeeper.RecoverableZooKeeper(176): Node /hbase/rs/ubuntu,39063,1517713226282 already deleted, retry=false 2018-02-04 11:00:41,582 INFO [M:0;ubuntu:39063] regionserver.HRegionServer(1150): stopping server ubuntu,39063,1517713226282; zookeeper connection closed. 2018-02-04 11:00:41,583 INFO [M:0;ubuntu:39063] regionserver.HRegionServer(1152): M:0;ubuntu:39063 exiting 2018-02-04 11:00:44,963 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2018-02-04 11:00:46,407 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(156): Removing adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2018-02-04 11:00:51,917 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2018-02-04 11:01:08,969 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:01:09,971 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:01:09,972 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:01:09,972 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:01:10,973 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:01:10,974 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:01:10,974 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:01:11,975 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:01:11,976 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:01:11,976 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:01:12,977 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:01:12,978 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:01:12,978 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:01:13,979 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:01:13,980 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:01:13,980 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:01:14,981 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:01:14,982 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:01:14,982 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:01:15,983 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:01:15,983 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:01:15,984 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:01:16,984 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:01:16,984 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:01:16,984 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:01:17,985 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:01:17,985 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:01:17,985 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:01:18,986 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:01:18,986 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:01:18,987 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:01:19,987 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:01:19,988 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:01:19,988 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:01:20,989 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:01:20,989 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:01:20,990 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:01:21,826 INFO [ReplicationExecutor-0] regionserver.ReplicationSourceManager$NodeFailoverWorker(699): Not transferring queue since we are shutting down 2018-02-04 11:01:21,991 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:01:21,991 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:01:21,992 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:01:22,992 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:01:22,993 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:01:22,993 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:01:23,994 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:01:23,995 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:01:23,995 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:01:24,996 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:01:24,997 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:01:24,997 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:01:25,998 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:01:25,999 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:01:25,999 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:01:27,000 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:01:27,001 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:01:27,001 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:01:28,002 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:01:28,002 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:01:28,003 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:01:28,696 DEBUG [ReadOnlyZKClient-localhost:56841@0x143faef3] zookeeper.ReadOnlyZKClient(316): 0x143faef3 to localhost:56841 inactive for 60000ms; closing (Will reconnect when new requests) 2018-02-04 11:01:28,697 DEBUG [ReadOnlyZKClient-localhost:56841@0x46e70b17] zookeeper.ReadOnlyZKClient(316): 0x46e70b17 to localhost:56841 inactive for 60000ms; closing (Will reconnect when new requests) 2018-02-04 11:01:28,705 DEBUG [ReadOnlyZKClient-localhost:56841@0x796736be] zookeeper.ReadOnlyZKClient(316): 0x796736be to localhost:56841 inactive for 60000ms; closing (Will reconnect when new requests) 2018-02-04 11:01:29,003 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:01:29,004 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:01:29,004 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:01:30,005 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:01:30,006 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:01:30,006 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:01:31,007 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:01:31,007 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:01:31,008 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:01:32,008 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:01:32,009 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:01:32,009 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:01:33,010 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:01:33,011 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:01:33,011 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:01:34,012 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:01:34,012 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:01:34,013 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:01:35,013 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:01:35,014 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:01:35,015 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:01:35,682 INFO [ReplicationExecutor-0] regionserver.ReplicationSourceManager$NodeFailoverWorker(699): Not transferring queue since we are shutting down 2018-02-04 11:01:36,015 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:01:36,015 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:01:36,016 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:01:37,016 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:01:37,016 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:01:37,017 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:01:38,017 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:01:38,018 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:01:38,018 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:01:39,019 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:01:39,019 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:01:39,020 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:01:40,021 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:01:40,022 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:01:40,022 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:01:41,022 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:01:41,023 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:01:41,023 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:01:42,024 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:01:42,025 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:01:42,025 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:01:43,026 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:01:43,026 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:01:43,026 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:01:44,028 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:01:44,028 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:01:44,029 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:01:45,029 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:01:45,030 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:01:45,030 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:01:46,031 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:01:46,031 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:01:46,031 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:01:47,031 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:01:47,032 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:01:47,032 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:01:48,033 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:01:48,034 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:01:48,034 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:01:49,096 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:01:49,096 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:01:49,097 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:01:50,097 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:01:50,098 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:01:50,098 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:01:51,099 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:01:51,100 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:01:51,100 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:01:52,101 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:01:52,102 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:01:52,102 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:01:53,102 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:01:53,103 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:01:53,103 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:01:54,104 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:01:54,104 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:01:54,105 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:01:55,105 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:01:55,106 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:01:55,106 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:01:56,107 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:01:56,107 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:01:56,107 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:01:57,107 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:01:57,108 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:01:57,108 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:01:58,109 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:01:58,110 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:01:58,110 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:01:59,110 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:01:59,111 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:01:59,111 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:02:00,111 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:02:00,112 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:02:00,113 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:02:01,113 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:02:01,114 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:02:01,114 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:02:02,115 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:02:02,116 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:02:02,116 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:02:03,117 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:02:03,117 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:02:03,117 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:02:04,118 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:02:04,119 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:02:04,119 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:02:05,119 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:02:05,120 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:02:05,120 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:02:06,121 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:02:06,121 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:02:06,121 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:02:07,122 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:02:07,123 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:02:07,123 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:02:08,124 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:02:08,125 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:02:08,125 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:02:09,125 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:02:09,126 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:02:09,126 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:02:10,127 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:02:10,128 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:02:10,128 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:02:11,129 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:02:11,129 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:02:11,130 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:02:12,131 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:02:12,131 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:02:12,131 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:02:13,132 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:02:13,133 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:02:13,133 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:02:14,134 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:02:14,135 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:02:14,135 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:02:15,136 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:02:15,137 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:02:15,137 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:02:16,137 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:02:16,138 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:02:16,138 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:02:17,139 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:02:17,139 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:02:17,140 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:02:18,141 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:02:18,141 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:02:18,141 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:02:19,142 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:02:19,143 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:02:19,143 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:02:20,144 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:02:20,145 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:02:20,145 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:02:21,146 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:02:21,146 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:02:21,146 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:02:22,147 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:02:22,148 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:02:22,148 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:02:23,149 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:02:23,150 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:02:23,150 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:02:24,151 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:02:24,151 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:02:24,152 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:02:25,152 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:02:25,153 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:02:25,153 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:02:26,154 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:02:26,155 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:02:26,155 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:02:27,156 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:02:27,156 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:02:27,157 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:02:28,158 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:02:28,158 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:02:28,158 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:02:28,702 DEBUG [ReadOnlyZKClient-localhost:56841@0x143faef3] zookeeper.ReadOnlyZKClient(316): 0x143faef3 to localhost:56841 inactive for 60000ms; closing (Will reconnect when new requests) 2018-02-04 11:02:28,707 DEBUG [ReadOnlyZKClient-localhost:56841@0x46e70b17] zookeeper.ReadOnlyZKClient(316): 0x46e70b17 to localhost:56841 inactive for 60000ms; closing (Will reconnect when new requests) 2018-02-04 11:02:28,710 DEBUG [ReadOnlyZKClient-localhost:56841@0x796736be] zookeeper.ReadOnlyZKClient(316): 0x796736be to localhost:56841 inactive for 60000ms; closing (Will reconnect when new requests) 2018-02-04 11:02:28,829 INFO [regionserver/ubuntu:0.Chore.1] hbase.ScheduledChore(180): Chore: MovedRegionsCleaner for region ubuntu,39267,1517713227316 was stopped 2018-02-04 11:02:28,830 INFO [regionserver/ubuntu:0.Chore.1] hbase.ScheduledChore(180): Chore: CompactedHFilesCleaner was stopped 2018-02-04 11:02:29,159 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:02:29,160 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:02:29,160 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:02:30,161 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:02:30,161 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:02:30,162 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:02:31,162 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:02:31,163 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:02:31,163 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:02:32,164 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:02:32,165 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:02:32,165 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:02:33,166 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:02:33,167 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:02:33,168 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:02:34,168 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:02:34,169 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:02:34,169 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:02:35,170 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:02:35,171 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:02:35,172 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:02:36,173 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:02:36,173 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:02:36,173 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:02:37,174 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:02:37,174 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:02:37,174 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:02:38,175 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:02:38,175 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:02:38,176 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:02:39,176 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:02:39,177 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:02:39,177 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:02:40,177 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:02:40,178 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:02:40,178 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:02:41,179 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:02:41,179 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:02:41,180 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:02:42,181 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:02:42,181 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:02:42,182 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:02:43,182 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:02:43,183 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:02:43,183 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:02:44,184 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:02:44,185 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:02:44,185 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:02:45,186 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:02:45,186 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:02:45,187 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:02:46,187 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:02:46,187 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:02:46,188 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:02:47,188 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:02:47,188 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:02:47,189 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:02:48,190 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:02:48,190 WARN [Time-limited test] util.JVMClusterUtil(289): RegionServerThreads remaining, give one more chance before interrupting 2018-02-04 11:02:48,191 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:02:49,191 WARN [Time-limited test] util.JVMClusterUtil(299): RegionServerThreads taking too long to stop, interrupting 2018-02-04 11:02:49,192 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1427): Interrupted while sleeping 2018-02-04 11:02:49,192 INFO [Time-limited test] util.JVMClusterUtil(322): Shutdown of 1 master(s) and 3 regionserver(s) complete 2018-02-04 11:03:28,703 DEBUG [ReadOnlyZKClient-localhost:56841@0x143faef3] zookeeper.ReadOnlyZKClient(316): 0x143faef3 to localhost:56841 inactive for 60000ms; closing (Will reconnect when new requests) 2018-02-04 11:03:28,708 DEBUG [ReadOnlyZKClient-localhost:56841@0x46e70b17] zookeeper.ReadOnlyZKClient(316): 0x46e70b17 to localhost:56841 inactive for 60000ms; closing (Will reconnect when new requests) 2018-02-04 11:03:28,710 DEBUG [ReadOnlyZKClient-localhost:56841@0x796736be] zookeeper.ReadOnlyZKClient(316): 0x796736be to localhost:56841 inactive for 60000ms; closing (Will reconnect when new requests) Process Thread Dump: Automatic Stack Trace every 60 seconds waiting on RS:0;ubuntu:39267 204 active threads Thread 1345 (Attach Listener): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: Thread 1343 (Timer for 'HBase' metrics system): State: TIMED_WAITING Blocked count: 0 Waited count: 18 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 1069 (RS_CLOSE_REGION-regionserver/ubuntu:0-2): State: TIMED_WAITING Blocked count: 72 Waited count: 274 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:133) org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:718) org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:605) org.apache.hadoop.hbase.regionserver.wal.WALUtil.doFullAppendTransaction(WALUtil.java:154) org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeFlushMarker(WALUtil.java:81) org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2645) org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2356) org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 1062 (RS_CLOSE_REGION-regionserver/ubuntu:0-1): State: TIMED_WAITING Blocked count: 54 Waited count: 232 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:133) org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:718) org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:605) org.apache.hadoop.hbase.regionserver.wal.WALUtil.doFullAppendTransaction(WALUtil.java:154) org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeMarker(WALUtil.java:132) org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeRegionEventMarker(WALUtil.java:97) org.apache.hadoop.hbase.regionserver.HRegion.writeRegionCloseMarker(HRegion.java:1103) org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1615) org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 1061 (RS_CLOSE_REGION-regionserver/ubuntu:0-0): State: TIMED_WAITING Blocked count: 66 Waited count: 280 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:133) org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:718) org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:605) org.apache.hadoop.hbase.regionserver.wal.WALUtil.doFullAppendTransaction(WALUtil.java:154) org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeFlushMarker(WALUtil.java:81) org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2645) org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2356) org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 988 (RS-EventLoopGroup-3-4): State: RUNNABLE Blocked count: 15 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 987 (Default-IPC-NioEventLoopGroup-7-3): State: RUNNABLE Blocked count: 23 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 986 (Default-IPC-NioEventLoopGroup-7-4): State: RUNNABLE Blocked count: 18 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 845 (RS_OPEN_REGION-regionserver/ubuntu:0-2): State: WAITING Blocked count: 50 Waited count: 67 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@72aa087e Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 842 (RS_OPEN_REGION-regionserver/ubuntu:0-1): State: WAITING Blocked count: 42 Waited count: 66 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@72aa087e Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 841 (RS_OPEN_REGION-regionserver/ubuntu:0-0): State: WAITING Blocked count: 42 Waited count: 64 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@72aa087e Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 810 (region-location-4): State: WAITING Blocked count: 5 Waited count: 4 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@282bcdfd Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 809 (region-location-3): State: WAITING Blocked count: 8 Waited count: 4 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@282bcdfd Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 808 (region-location-2): State: WAITING Blocked count: 13 Waited count: 7 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@282bcdfd Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 807 (region-location-1): State: WAITING Blocked count: 11 Waited count: 14 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@282bcdfd Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 571 (RS-EventLoopGroup-1-5): State: RUNNABLE Blocked count: 2 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 570 (Default-IPC-NioEventLoopGroup-7-2): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 568 (RS-EventLoopGroup-4-5): State: RUNNABLE Blocked count: 15 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 567 (Default-IPC-NioEventLoopGroup-7-1): State: RUNNABLE Blocked count: 49 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 560 (RS-EventLoopGroup-3-3): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 553 (RS_OPEN_PRIORITY_REGION-regionserver/ubuntu:0-0): State: WAITING Blocked count: 14 Waited count: 26 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6b2dcd67 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 551 (RS-EventLoopGroup-3-2): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 549 (RS-EventLoopGroup-4-4): State: RUNNABLE Blocked count: 1 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 544 (region-location-0): State: WAITING Blocked count: 12 Waited count: 18 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@282bcdfd Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 521 (RS-EventLoopGroup-4-3): State: RUNNABLE Blocked count: 1 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 508 (RS-EventLoopGroup-4-2): State: RUNNABLE Blocked count: 1 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 504 (AsyncFSWAL-0): State: WAITING Blocked count: 0 Waited count: 169 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1c3dccfe Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 503 (RS-EventLoopGroup-5-8): State: RUNNABLE Blocked count: 7 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 497 (RS-EventLoopGroup-5-7): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 483 (RS-EventLoopGroup-5-6): State: RUNNABLE Blocked count: 3 Waited count: 2 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 480 (LeaseRenewer:zhangduo.hfs.0@localhost:44491): State: TIMED_WAITING Blocked count: 6 Waited count: 211 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:444) org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304) java.lang.Thread.run(Thread.java:748) Thread 482 (RS-EventLoopGroup-5-5): State: RUNNABLE Blocked count: 12 Waited count: 3 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 476 (ubuntu:39267Replication Statistics #0): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 470 (ReplicationExecutor-0): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@153c8699 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 458 (regionserver/ubuntu:0.procedureResultReporter): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@788f3b3e Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:75) Thread 455 (regionserver/ubuntu:0.logRoller): State: TIMED_WAITING Blocked count: 6 Waited count: 206 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:147) java.lang.Thread.run(Thread.java:748) Thread 450 (regionserver/ubuntu:0.procedureResultReporter): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73ec8f20 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:75) Thread 452 (regionserver/ubuntu:0.procedureResultReporter): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@723b1ca8 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:75) Thread 443 (regionserver/ubuntu:0.Chore.1): State: TIMED_WAITING Blocked count: 1 Waited count: 29 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 431 (JvmPauseMonitor): State: TIMED_WAITING Blocked count: 0 Waited count: 401 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:154) java.lang.Thread.run(Thread.java:748) Thread 423 (ReadOnlyZKClient-localhost:56841@0x796736be): State: TIMED_WAITING Blocked count: 1 Waited count: 6 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:307) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$63/1354899920.run(Unknown Source) java.lang.Thread.run(Thread.java:748) Thread 422 (ReadOnlyZKClient-localhost:56841@0x143faef3): State: TIMED_WAITING Blocked count: 1 Waited count: 6 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:307) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$63/1354899920.run(Unknown Source) java.lang.Thread.run(Thread.java:748) Thread 421 (ReadOnlyZKClient-localhost:56841@0x46e70b17): State: TIMED_WAITING Blocked count: 1 Waited count: 6 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:307) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$63/1354899920.run(Unknown Source) java.lang.Thread.run(Thread.java:748) Thread 419 (IPC Parameter Sending Thread #1): State: TIMED_WAITING Blocked count: 0 Waited count: 1401 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 416 (threadDeathWatcher-6-1): State: TIMED_WAITING Blocked count: 0 Waited count: 201 Stack: java.lang.Thread.sleep(Native Method) org.apache.hbase.thirdparty.io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:152) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 415 (RS-EventLoopGroup-1-4): State: RUNNABLE Blocked count: 29 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 414 (RS-EventLoopGroup-1-3): State: RUNNABLE Blocked count: 18 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 413 (RS-EventLoopGroup-1-2): State: RUNNABLE Blocked count: 16 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 367 (RpcClient-timer-pool1-t1): State: TIMED_WAITING Blocked count: 0 Waited count: 20039 Stack: java.lang.Thread.sleep(Native Method) org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:560) org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:459) java.lang.Thread.run(Thread.java:748) Thread 410 (RS-EventLoopGroup-5-4): State: RUNNABLE Blocked count: 25 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 411 (RS-EventLoopGroup-5-2): State: RUNNABLE Blocked count: 56 Waited count: 3 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 412 (RS-EventLoopGroup-5-3): State: RUNNABLE Blocked count: 31 Waited count: 4 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 407 (snapshot-hfile-cleaner-cache-refresher): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 400 (M:0;ubuntu:39063-EventThread): State: WAITING Blocked count: 1 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4fcbce78 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501) Thread 399 (M:0;ubuntu:39063-SendThread(localhost:56841)): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141) Thread 289 (RS:0;ubuntu:39267): State: TIMED_WAITING Blocked count: 198 Waited count: 1009 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.regionserver.HRegionServer.sleep(HRegionServer.java:1425) org.apache.hadoop.hbase.regionserver.HRegionServer.waitOnAllRegionsToClose(HRegionServer.java:1411) org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1103) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:187) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$0(MiniHBaseCluster.java:186) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:171) java.security.AccessController.doPrivileged(Native Method) javax.security.auth.Subject.doAs(Subject.java:360) org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726) org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:168) java.lang.Thread.run(Thread.java:748) Thread 368 (Idle-Rpc-Conn-Sweeper-pool2-t1): State: TIMED_WAITING Blocked count: 0 Waited count: 11 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 334 (org.apache.hadoop.hdfs.PeerCache@2e1946df): State: TIMED_WAITING Blocked count: 1 Waited count: 68 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:255) org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:46) org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:124) java.lang.Thread.run(Thread.java:748) Thread 332 (M:0;ubuntu:39063-MemStoreChunkPool Statistics): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 328 (Monitor thread for TaskMonitor): State: TIMED_WAITING Blocked count: 0 Waited count: 21 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.monitoring.TaskMonitor$MonitorRunnable.run(TaskMonitor.java:302) java.lang.Thread.run(Thread.java:748) Thread 310 (RS-EventLoopGroup-5-1): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 291 (RS-EventLoopGroup-4-1): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 288 (RpcServer.replication.FPBQ.Fifo.handler=2,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@50f24e6c Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 287 (RpcServer.replication.FPBQ.Fifo.handler=1,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@5b366694 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 286 (RpcServer.replication.FPBQ.Fifo.handler=0,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@68239fa7 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 285 (RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=39267): State: WAITING Blocked count: 8 Waited count: 20 Waiting on java.util.concurrent.Semaphore$NonfairSync@64e05a8c Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 284 (RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@26a3f952 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 283 (RpcServer.priority.FPBQ.Fifo.handler=3,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@4ec929d4 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 282 (RpcServer.priority.FPBQ.Fifo.handler=2,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@3ae3490d Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 281 (RpcServer.priority.FPBQ.Fifo.handler=1,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@5cb140e7 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 280 (RpcServer.priority.FPBQ.Fifo.handler=0,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@65614b4a Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 279 (RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39267): State: WAITING Blocked count: 33 Waited count: 56 Waiting on java.util.concurrent.Semaphore$NonfairSync@626159bb Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 278 (RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 2 Waiting on java.util.concurrent.Semaphore$NonfairSync@42eadfdb Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 277 (RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@148a1919 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 276 (RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@7b5e2bd9 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 275 (RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@139a0988 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 272 (RS-EventLoopGroup-3-1): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 248 (RS-EventLoopGroup-1-1): State: RUNNABLE Blocked count: 1 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 247 (HBase-Metrics2-1): State: TIMED_WAITING Blocked count: 0 Waited count: 229 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 235 (ProcessThread(sid:0 cport:56841):): State: WAITING Blocked count: 1 Waited count: 609 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@133caf02 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:122) Thread 234 (SyncThread:0): State: WAITING Blocked count: 0 Waited count: 581 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5c669c88 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:127) Thread 233 (SessionTracker): State: TIMED_WAITING Blocked count: 0 Waited count: 103 Stack: java.lang.Object.wait(Native Method) org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146) Thread 232 (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:56841): State: RUNNABLE Blocked count: 9 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:173) java.lang.Thread.run(Thread.java:748) Thread 229 (java.util.concurrent.ThreadPoolExecutor$Worker@1d726817[State = -1, empty queue]): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 224 (refreshUsed-/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data6/current/BP-35161775-127.0.1.1-1517713219102): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:132) java.lang.Thread.run(Thread.java:748) Thread 222 (refreshUsed-/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data5/current/BP-35161775-127.0.1.1-1517713219102): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:132) java.lang.Thread.run(Thread.java:748) Thread 218 (org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl$LazyWriter@2327f0e2): State: TIMED_WAITING Blocked count: 0 Waited count: 4 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl$LazyWriter.run(FsDatasetImpl.java:3071) java.lang.Thread.run(Thread.java:748) Thread 217 (VolumeScannerThread(/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data6)): State: TIMED_WAITING Blocked count: 1 Waited count: 2 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:622) Thread 216 (VolumeScannerThread(/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data5)): State: TIMED_WAITING Blocked count: 1 Waited count: 2 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:622) Thread 213 (IPC Server handler 9 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 236 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 212 (IPC Server handler 8 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 221 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 211 (IPC Server handler 7 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 226 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 210 (IPC Server handler 6 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 239 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 209 (IPC Server handler 5 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 221 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 208 (IPC Server handler 4 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 248 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 207 (IPC Server handler 3 on 43825): State: TIMED_WAITING Blocked count: 1 Waited count: 209 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 206 (IPC Server handler 2 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 248 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 205 (IPC Server handler 1 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 256 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 204 (IPC Server handler 0 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 207 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 199 (IPC Server listener on 43825): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener.run(Server.java:807) Thread 202 (IPC Server Responder): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.ipc.Server$Responder.doRunLoop(Server.java:982) org.apache.hadoop.ipc.Server$Responder.run(Server.java:965) Thread 161 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@3e65225e): State: RUNNABLE Blocked count: 1 Waited count: 0 Stack: sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:100) org.apache.hadoop.hdfs.net.TcpPeerServer.accept(TcpPeerServer.java:141) org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:135) java.lang.Thread.run(Thread.java:748) Thread 203 (DataNode: [[[DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data5/, [DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data6/]] heartbeating to localhost/127.0.0.1:44491): State: TIMED_WAITING Blocked count: 352 Waited count: 784 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.IncrementalBlockReportManager.waitTillNextIBR(IncrementalBlockReportManager.java:130) org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:542) org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:659) java.lang.Thread.run(Thread.java:748) Thread 201 (IPC Server idle connection scanner for port 43825): State: TIMED_WAITING Blocked count: 1 Waited count: 22 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 200 (Socket Reader #1 for port 43825): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:745) org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:724) Thread 198 (org.apache.hadoop.util.JvmPauseMonitor$Monitor@3369a0f5): State: TIMED_WAITING Blocked count: 0 Waited count: 409 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:182) java.lang.Thread.run(Thread.java:748) Thread 182 (nioEventLoopGroup-6-1): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) java.lang.Thread.run(Thread.java:748) Thread 181 (Timer-3): State: TIMED_WAITING Blocked count: 0 Waited count: 7 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 180 (java.util.concurrent.ThreadPoolExecutor$Worker@4231ed0c[State = -1, empty queue]): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 175 (refreshUsed-/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data4/current/BP-35161775-127.0.1.1-1517713219102): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:132) java.lang.Thread.run(Thread.java:748) Thread 173 (refreshUsed-/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data3/current/BP-35161775-127.0.1.1-1517713219102): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:132) java.lang.Thread.run(Thread.java:748) Thread 169 (org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl$LazyWriter@221a6ce4): State: TIMED_WAITING Blocked count: 0 Waited count: 4 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl$LazyWriter.run(FsDatasetImpl.java:3071) java.lang.Thread.run(Thread.java:748) Thread 168 (VolumeScannerThread(/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data4)): State: TIMED_WAITING Blocked count: 1 Waited count: 2 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:622) Thread 167 (VolumeScannerThread(/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data3)): State: TIMED_WAITING Blocked count: 1 Waited count: 2 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:622) Thread 164 (103115058@qtp-229466626-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:41633): State: RUNNABLE Blocked count: 2 Waited count: 1 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498) org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192) org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Thread 163 (498148987@qtp-229466626-0): State: TIMED_WAITING Blocked count: 0 Waited count: 4 Stack: java.lang.Object.wait(Native Method) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626) Thread 162 (pool-12-thread-1): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 156 (IPC Server handler 9 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 244 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 155 (IPC Server handler 8 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 207 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 154 (IPC Server handler 7 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 208 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 153 (IPC Server handler 6 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 244 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 152 (IPC Server handler 5 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 227 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 151 (IPC Server handler 4 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 213 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 150 (IPC Server handler 3 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 235 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 149 (IPC Server handler 2 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 240 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 148 (IPC Server handler 1 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 209 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 147 (IPC Server handler 0 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 206 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 142 (IPC Server listener on 33493): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener.run(Server.java:807) Thread 145 (IPC Server Responder): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.ipc.Server$Responder.doRunLoop(Server.java:982) org.apache.hadoop.ipc.Server$Responder.run(Server.java:965) Thread 102 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@26506cf6): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:100) org.apache.hadoop.hdfs.net.TcpPeerServer.accept(TcpPeerServer.java:141) org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:135) java.lang.Thread.run(Thread.java:748) Thread 146 (DataNode: [[[DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data3/, [DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data4/]] heartbeating to localhost/127.0.0.1:44491): State: TIMED_WAITING Blocked count: 332 Waited count: 776 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.IncrementalBlockReportManager.waitTillNextIBR(IncrementalBlockReportManager.java:130) org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:542) org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:659) java.lang.Thread.run(Thread.java:748) Thread 144 (IPC Server idle connection scanner for port 33493): State: TIMED_WAITING Blocked count: 1 Waited count: 22 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 143 (Socket Reader #1 for port 33493): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:745) org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:724) Thread 141 (org.apache.hadoop.util.JvmPauseMonitor$Monitor@1d004c0c): State: TIMED_WAITING Blocked count: 0 Waited count: 411 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:182) java.lang.Thread.run(Thread.java:748) Thread 125 (nioEventLoopGroup-4-1): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) java.lang.Thread.run(Thread.java:748) Thread 124 (Timer-2): State: TIMED_WAITING Blocked count: 0 Waited count: 7 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 123 (java.util.concurrent.ThreadPoolExecutor$Worker@3adb50a8[State = -1, empty queue]): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 118 (refreshUsed-/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data1/current/BP-35161775-127.0.1.1-1517713219102): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:132) java.lang.Thread.run(Thread.java:748) Thread 117 (refreshUsed-/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data2/current/BP-35161775-127.0.1.1-1517713219102): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:132) java.lang.Thread.run(Thread.java:748) Thread 112 (org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl$LazyWriter@7e498b6b): State: TIMED_WAITING Blocked count: 0 Waited count: 4 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl$LazyWriter.run(FsDatasetImpl.java:3071) java.lang.Thread.run(Thread.java:748) Thread 111 (VolumeScannerThread(/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data2)): State: TIMED_WAITING Blocked count: 11 Waited count: 2 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:622) Thread 110 (VolumeScannerThread(/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data1)): State: TIMED_WAITING Blocked count: 7 Waited count: 2 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:622) Thread 106 (IPC Client (1670261399) connection to localhost/127.0.0.1:44491 from zhangduo): State: TIMED_WAITING Blocked count: 702 Waited count: 703 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:934) org.apache.hadoop.ipc.Client$Connection.run(Client.java:979) Thread 105 (1512126834@qtp-261780187-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:34039): State: RUNNABLE Blocked count: 1 Waited count: 1 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498) org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192) org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Thread 104 (1495196264@qtp-261780187-0): State: TIMED_WAITING Blocked count: 0 Waited count: 4 Stack: java.lang.Object.wait(Native Method) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626) Thread 103 (pool-6-thread-1): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 97 (IPC Server handler 9 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 237 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 96 (IPC Server handler 8 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 235 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 95 (IPC Server handler 7 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 227 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 94 (IPC Server handler 6 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 229 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 93 (IPC Server handler 5 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 224 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 92 (IPC Server handler 4 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 225 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 91 (IPC Server handler 3 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 256 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 90 (IPC Server handler 2 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 236 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 89 (IPC Server handler 1 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 250 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 88 (IPC Server handler 0 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 235 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 83 (IPC Server listener on 45595): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener.run(Server.java:807) Thread 86 (IPC Server Responder): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.ipc.Server$Responder.doRunLoop(Server.java:982) org.apache.hadoop.ipc.Server$Responder.run(Server.java:965) Thread 60 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@5ec0030b): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:100) org.apache.hadoop.hdfs.net.TcpPeerServer.accept(TcpPeerServer.java:141) org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:135) java.lang.Thread.run(Thread.java:748) Thread 87 (DataNode: [[[DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data1/, [DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data2/]] heartbeating to localhost/127.0.0.1:44491): State: TIMED_WAITING Blocked count: 346 Waited count: 781 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.IncrementalBlockReportManager.waitTillNextIBR(IncrementalBlockReportManager.java:130) org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:542) org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:659) java.lang.Thread.run(Thread.java:748) Thread 85 (IPC Server idle connection scanner for port 45595): State: TIMED_WAITING Blocked count: 1 Waited count: 22 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 84 (Socket Reader #1 for port 45595): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:745) org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:724) Thread 82 (org.apache.hadoop.util.JvmPauseMonitor$Monitor@74095914): State: TIMED_WAITING Blocked count: 0 Waited count: 413 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:182) java.lang.Thread.run(Thread.java:748) Thread 65 (nioEventLoopGroup-2-1): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) java.lang.Thread.run(Thread.java:748) Thread 64 (Timer-1): State: TIMED_WAITING Blocked count: 0 Waited count: 7 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 63 (1020827856@qtp-415074966-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:40851): State: RUNNABLE Blocked count: 1 Waited count: 1 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498) org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192) org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Thread 62 (1498522151@qtp-415074966-0): State: TIMED_WAITING Blocked count: 0 Waited count: 4 Stack: java.lang.Object.wait(Native Method) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626) Thread 61 (pool-4-thread-1): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 55 (CacheReplicationMonitor(1454695942)): State: TIMED_WAITING Blocked count: 0 Waited count: 8 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163) org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor.run(CacheReplicationMonitor.java:181) Thread 54 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$LazyPersistFileScrubber@5654e22b): State: TIMED_WAITING Blocked count: 0 Waited count: 2 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.namenode.FSNamesystem$LazyPersistFileScrubber.run(FSNamesystem.java:4725) java.lang.Thread.run(Thread.java:748) Thread 53 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeEditLogRoller@68702a45): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeEditLogRoller.run(FSNamesystem.java:4642) java.lang.Thread.run(Thread.java:748) Thread 52 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor@3d921e48): State: TIMED_WAITING Blocked count: 0 Waited count: 42 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor.run(FSNamesystem.java:4598) java.lang.Thread.run(Thread.java:748) Thread 51 (org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@fe27903): State: TIMED_WAITING Blocked count: 0 Waited count: 105 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:431) java.lang.Thread.run(Thread.java:748) Thread 50 (IPC Server handler 9 on 44491): State: TIMED_WAITING Blocked count: 3 Waited count: 401 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 49 (IPC Server handler 8 on 44491): State: TIMED_WAITING Blocked count: 2 Waited count: 392 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 48 (IPC Server handler 7 on 44491): State: TIMED_WAITING Blocked count: 3 Waited count: 401 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 47 (IPC Server handler 6 on 44491): State: TIMED_WAITING Blocked count: 3 Waited count: 409 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 46 (IPC Server handler 5 on 44491): State: TIMED_WAITING Blocked count: 17 Waited count: 405 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 45 (IPC Server handler 4 on 44491): State: TIMED_WAITING Blocked count: 5 Waited count: 405 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 44 (IPC Server handler 3 on 44491): State: TIMED_WAITING Blocked count: 6 Waited count: 391 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 43 (IPC Server handler 2 on 44491): State: TIMED_WAITING Blocked count: 8 Waited count: 401 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 42 (IPC Server handler 1 on 44491): State: TIMED_WAITING Blocked count: 4 Waited count: 395 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 41 (IPC Server handler 0 on 44491): State: TIMED_WAITING Blocked count: 7 Waited count: 400 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 31 (IPC Server listener on 44491): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener.run(Server.java:807) Thread 34 (IPC Server Responder): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.ipc.Server$Responder.doRunLoop(Server.java:982) org.apache.hadoop.ipc.Server$Responder.run(Server.java:965) Thread 29 (Block report processor): State: WAITING Blocked count: 2 Waited count: 140 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@c5f3502 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403) org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.processQueue(BlockManager.java:3854) org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.run(BlockManager.java:3843) Thread 28 (org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@2748be45): State: TIMED_WAITING Blocked count: 0 Waited count: 71 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3638) java.lang.Thread.run(Thread.java:748) Thread 30 (org.apache.hadoop.hdfs.server.blockmanagement.HeartbeatManager$Monitor@1706a704): State: TIMED_WAITING Blocked count: 0 Waited count: 42 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.blockmanagement.HeartbeatManager$Monitor.run(HeartbeatManager.java:381) java.lang.Thread.run(Thread.java:748) Thread 40 (DecommissionMonitor-0): State: TIMED_WAITING Blocked count: 0 Waited count: 70 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 39 (org.apache.hadoop.hdfs.server.blockmanagement.PendingReplicationBlocks$PendingReplicationMonitor@28cb973d): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.blockmanagement.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:221) java.lang.Thread.run(Thread.java:748) Thread 35 (org.apache.hadoop.util.JvmPauseMonitor$Monitor@7501869b): State: TIMED_WAITING Blocked count: 0 Waited count: 415 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:182) java.lang.Thread.run(Thread.java:748) Thread 33 (IPC Server idle connection scanner for port 44491): State: TIMED_WAITING Blocked count: 1 Waited count: 22 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 32 (Socket Reader #1 for port 44491): State: RUNNABLE Blocked count: 1 Waited count: 1 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:745) org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:724) Thread 27 (Timer-0): State: TIMED_WAITING Blocked count: 1 Waited count: 8 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 25 (1941945463@qtp-2007783481-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:43361): State: RUNNABLE Blocked count: 1 Waited count: 1 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498) org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192) org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Thread 24 (1848959351@qtp-2007783481-0): State: TIMED_WAITING Blocked count: 0 Waited count: 4 Stack: java.lang.Object.wait(Native Method) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626) Thread 23 (pool-2-thread-1): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 14 (org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner): State: WAITING Blocked count: 4 Waited count: 5 Waiting on java.lang.ref.ReferenceQueue$Lock@688d8764 Stack: java.lang.Object.wait(Native Method) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164) org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:3063) java.lang.Thread.run(Thread.java:748) Thread 13 (Time-limited test): State: RUNNABLE Blocked count: 91 Waited count: 229 Stack: sun.management.ThreadImpl.getThreadInfo1(Native Method) sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:178) sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:139) org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:168) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) org.apache.hadoop.hbase.util.Threads$PrintThreadInfoLazyHolder$1.printThreadInfo(Threads.java:294) org.apache.hadoop.hbase.util.Threads.printThreadInfo(Threads.java:341) org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:135) org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:367) org.apache.hadoop.hbase.MiniHBaseCluster.waitUntilShutDown(MiniHBaseCluster.java:865) org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:1130) org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:1105) org.apache.hadoop.hbase.TestFullLogReconstruction.tearDownAfterClass(TestFullLogReconstruction.java:68) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) Thread 9 (ReaderThread): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: java.net.SocketInputStream.socketRead0(Native Method) java.net.SocketInputStream.socketRead(SocketInputStream.java:116) java.net.SocketInputStream.read(SocketInputStream.java:171) java.net.SocketInputStream.read(SocketInputStream.java:141) sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284) sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326) sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178) java.io.InputStreamReader.read(InputStreamReader.java:184) java.io.BufferedReader.fill(BufferedReader.java:161) java.io.BufferedReader.readLine(BufferedReader.java:324) java.io.BufferedReader.readLine(BufferedReader.java:389) org.eclipse.jdt.internal.junit.runner.RemoteTestRunner$ReaderThread.run(RemoteTestRunner.java:148) Thread 4 (Signal Dispatcher): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: Thread 3 (Finalizer): State: WAITING Blocked count: 266 Waited count: 15 Waiting on java.lang.ref.ReferenceQueue$Lock@60cb01b5 Stack: java.lang.Object.wait(Native Method) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164) java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209) Thread 2 (Reference Handler): State: WAITING Blocked count: 21 Waited count: 14 Waiting on java.lang.ref.Reference$Lock@6a276aab Stack: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) java.lang.ref.Reference.tryHandlePending(Reference.java:191) java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153) Thread 1 (main): State: TIMED_WAITING Blocked count: 0 Waited count: 2 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.FutureTask.awaitDone(FutureTask.java:426) java.util.concurrent.FutureTask.get(FutureTask.java:204) org.junit.internal.runners.statements.FailOnTimeout.getResult(FailOnTimeout.java:141) org.junit.internal.runners.statements.FailOnTimeout.evaluate(FailOnTimeout.java:127) org.junit.rules.RunRules.evaluate(RunRules.java:20) org.junit.runners.ParentRunner.run(ParentRunner.java:363) org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86) org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38) org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:538) org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:760) org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:460) org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:206) 2018-02-04 11:04:28,704 DEBUG [ReadOnlyZKClient-localhost:56841@0x143faef3] zookeeper.ReadOnlyZKClient(316): 0x143faef3 to localhost:56841 inactive for 60000ms; closing (Will reconnect when new requests) 2018-02-04 11:04:28,710 DEBUG [ReadOnlyZKClient-localhost:56841@0x46e70b17] zookeeper.ReadOnlyZKClient(316): 0x46e70b17 to localhost:56841 inactive for 60000ms; closing (Will reconnect when new requests) 2018-02-04 11:04:28,712 DEBUG [ReadOnlyZKClient-localhost:56841@0x796736be] zookeeper.ReadOnlyZKClient(316): 0x796736be to localhost:56841 inactive for 60000ms; closing (Will reconnect when new requests) Process Thread Dump: Automatic Stack Trace every 60 seconds waiting on RS:0;ubuntu:39267 204 active threads Thread 1345 (Attach Listener): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: Thread 1343 (Timer for 'HBase' metrics system): State: TIMED_WAITING Blocked count: 0 Waited count: 24 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 1069 (RS_CLOSE_REGION-regionserver/ubuntu:0-2): State: TIMED_WAITING Blocked count: 72 Waited count: 334 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:133) org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:718) org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:605) org.apache.hadoop.hbase.regionserver.wal.WALUtil.doFullAppendTransaction(WALUtil.java:154) org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeFlushMarker(WALUtil.java:81) org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2645) org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2356) org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 1062 (RS_CLOSE_REGION-regionserver/ubuntu:0-1): State: TIMED_WAITING Blocked count: 54 Waited count: 292 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:133) org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:718) org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:605) org.apache.hadoop.hbase.regionserver.wal.WALUtil.doFullAppendTransaction(WALUtil.java:154) org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeMarker(WALUtil.java:132) org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeRegionEventMarker(WALUtil.java:97) org.apache.hadoop.hbase.regionserver.HRegion.writeRegionCloseMarker(HRegion.java:1103) org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1615) org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 1061 (RS_CLOSE_REGION-regionserver/ubuntu:0-0): State: TIMED_WAITING Blocked count: 66 Waited count: 340 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:133) org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:718) org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:605) org.apache.hadoop.hbase.regionserver.wal.WALUtil.doFullAppendTransaction(WALUtil.java:154) org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeFlushMarker(WALUtil.java:81) org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2645) org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2356) org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 988 (RS-EventLoopGroup-3-4): State: RUNNABLE Blocked count: 15 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 987 (Default-IPC-NioEventLoopGroup-7-3): State: RUNNABLE Blocked count: 23 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 986 (Default-IPC-NioEventLoopGroup-7-4): State: RUNNABLE Blocked count: 18 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 845 (RS_OPEN_REGION-regionserver/ubuntu:0-2): State: WAITING Blocked count: 50 Waited count: 67 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@72aa087e Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 842 (RS_OPEN_REGION-regionserver/ubuntu:0-1): State: WAITING Blocked count: 42 Waited count: 66 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@72aa087e Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 841 (RS_OPEN_REGION-regionserver/ubuntu:0-0): State: WAITING Blocked count: 42 Waited count: 64 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@72aa087e Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 810 (region-location-4): State: WAITING Blocked count: 5 Waited count: 4 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@282bcdfd Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 809 (region-location-3): State: WAITING Blocked count: 8 Waited count: 4 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@282bcdfd Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 808 (region-location-2): State: WAITING Blocked count: 13 Waited count: 7 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@282bcdfd Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 807 (region-location-1): State: WAITING Blocked count: 11 Waited count: 14 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@282bcdfd Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 571 (RS-EventLoopGroup-1-5): State: RUNNABLE Blocked count: 2 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 570 (Default-IPC-NioEventLoopGroup-7-2): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 568 (RS-EventLoopGroup-4-5): State: RUNNABLE Blocked count: 15 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 567 (Default-IPC-NioEventLoopGroup-7-1): State: RUNNABLE Blocked count: 49 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 560 (RS-EventLoopGroup-3-3): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 553 (RS_OPEN_PRIORITY_REGION-regionserver/ubuntu:0-0): State: WAITING Blocked count: 14 Waited count: 26 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6b2dcd67 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 551 (RS-EventLoopGroup-3-2): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 549 (RS-EventLoopGroup-4-4): State: RUNNABLE Blocked count: 1 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 544 (region-location-0): State: WAITING Blocked count: 12 Waited count: 18 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@282bcdfd Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 521 (RS-EventLoopGroup-4-3): State: RUNNABLE Blocked count: 1 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 508 (RS-EventLoopGroup-4-2): State: RUNNABLE Blocked count: 1 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 504 (AsyncFSWAL-0): State: WAITING Blocked count: 0 Waited count: 169 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1c3dccfe Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 503 (RS-EventLoopGroup-5-8): State: RUNNABLE Blocked count: 7 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 497 (RS-EventLoopGroup-5-7): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 483 (RS-EventLoopGroup-5-6): State: RUNNABLE Blocked count: 3 Waited count: 2 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 480 (LeaseRenewer:zhangduo.hfs.0@localhost:44491): State: TIMED_WAITING Blocked count: 8 Waited count: 276 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:444) org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304) java.lang.Thread.run(Thread.java:748) Thread 482 (RS-EventLoopGroup-5-5): State: RUNNABLE Blocked count: 12 Waited count: 3 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 476 (ubuntu:39267Replication Statistics #0): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 470 (ReplicationExecutor-0): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@153c8699 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 458 (regionserver/ubuntu:0.procedureResultReporter): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@788f3b3e Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:75) Thread 455 (regionserver/ubuntu:0.logRoller): State: TIMED_WAITING Blocked count: 6 Waited count: 266 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:147) java.lang.Thread.run(Thread.java:748) Thread 450 (regionserver/ubuntu:0.procedureResultReporter): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73ec8f20 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:75) Thread 452 (regionserver/ubuntu:0.procedureResultReporter): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@723b1ca8 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:75) Thread 443 (regionserver/ubuntu:0.Chore.1): State: TIMED_WAITING Blocked count: 1 Waited count: 32 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 431 (JvmPauseMonitor): State: TIMED_WAITING Blocked count: 0 Waited count: 521 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:154) java.lang.Thread.run(Thread.java:748) Thread 423 (ReadOnlyZKClient-localhost:56841@0x796736be): State: TIMED_WAITING Blocked count: 1 Waited count: 7 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:307) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$63/1354899920.run(Unknown Source) java.lang.Thread.run(Thread.java:748) Thread 422 (ReadOnlyZKClient-localhost:56841@0x143faef3): State: TIMED_WAITING Blocked count: 1 Waited count: 7 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:307) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$63/1354899920.run(Unknown Source) java.lang.Thread.run(Thread.java:748) Thread 421 (ReadOnlyZKClient-localhost:56841@0x46e70b17): State: TIMED_WAITING Blocked count: 1 Waited count: 7 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:307) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$63/1354899920.run(Unknown Source) java.lang.Thread.run(Thread.java:748) Thread 419 (IPC Parameter Sending Thread #1): State: TIMED_WAITING Blocked count: 0 Waited count: 1583 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 416 (threadDeathWatcher-6-1): State: TIMED_WAITING Blocked count: 0 Waited count: 261 Stack: java.lang.Thread.sleep(Native Method) org.apache.hbase.thirdparty.io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:152) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 415 (RS-EventLoopGroup-1-4): State: RUNNABLE Blocked count: 29 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 414 (RS-EventLoopGroup-1-3): State: RUNNABLE Blocked count: 18 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 413 (RS-EventLoopGroup-1-2): State: RUNNABLE Blocked count: 16 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 367 (RpcClient-timer-pool1-t1): State: TIMED_WAITING Blocked count: 0 Waited count: 26056 Stack: java.lang.Thread.sleep(Native Method) org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:560) org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:459) java.lang.Thread.run(Thread.java:748) Thread 410 (RS-EventLoopGroup-5-4): State: RUNNABLE Blocked count: 25 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 411 (RS-EventLoopGroup-5-2): State: RUNNABLE Blocked count: 56 Waited count: 3 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 412 (RS-EventLoopGroup-5-3): State: RUNNABLE Blocked count: 31 Waited count: 4 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 407 (snapshot-hfile-cleaner-cache-refresher): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 400 (M:0;ubuntu:39063-EventThread): State: WAITING Blocked count: 1 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4fcbce78 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501) Thread 399 (M:0;ubuntu:39063-SendThread(localhost:56841)): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141) Thread 289 (RS:0;ubuntu:39267): State: TIMED_WAITING Blocked count: 198 Waited count: 1309 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.regionserver.HRegionServer.sleep(HRegionServer.java:1425) org.apache.hadoop.hbase.regionserver.HRegionServer.waitOnAllRegionsToClose(HRegionServer.java:1411) org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1103) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:187) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$0(MiniHBaseCluster.java:186) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:171) java.security.AccessController.doPrivileged(Native Method) javax.security.auth.Subject.doAs(Subject.java:360) org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726) org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:168) java.lang.Thread.run(Thread.java:748) Thread 368 (Idle-Rpc-Conn-Sweeper-pool2-t1): State: TIMED_WAITING Blocked count: 0 Waited count: 15 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 334 (org.apache.hadoop.hdfs.PeerCache@2e1946df): State: TIMED_WAITING Blocked count: 1 Waited count: 88 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:255) org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:46) org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:124) java.lang.Thread.run(Thread.java:748) Thread 332 (M:0;ubuntu:39063-MemStoreChunkPool Statistics): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 328 (Monitor thread for TaskMonitor): State: TIMED_WAITING Blocked count: 0 Waited count: 27 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.monitoring.TaskMonitor$MonitorRunnable.run(TaskMonitor.java:302) java.lang.Thread.run(Thread.java:748) Thread 310 (RS-EventLoopGroup-5-1): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 291 (RS-EventLoopGroup-4-1): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 288 (RpcServer.replication.FPBQ.Fifo.handler=2,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@50f24e6c Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 287 (RpcServer.replication.FPBQ.Fifo.handler=1,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@5b366694 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 286 (RpcServer.replication.FPBQ.Fifo.handler=0,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@68239fa7 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 285 (RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=39267): State: WAITING Blocked count: 8 Waited count: 20 Waiting on java.util.concurrent.Semaphore$NonfairSync@64e05a8c Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 284 (RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@26a3f952 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 283 (RpcServer.priority.FPBQ.Fifo.handler=3,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@4ec929d4 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 282 (RpcServer.priority.FPBQ.Fifo.handler=2,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@3ae3490d Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 281 (RpcServer.priority.FPBQ.Fifo.handler=1,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@5cb140e7 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 280 (RpcServer.priority.FPBQ.Fifo.handler=0,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@65614b4a Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 279 (RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39267): State: WAITING Blocked count: 33 Waited count: 56 Waiting on java.util.concurrent.Semaphore$NonfairSync@626159bb Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 278 (RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 2 Waiting on java.util.concurrent.Semaphore$NonfairSync@42eadfdb Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 277 (RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@148a1919 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 276 (RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@7b5e2bd9 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 275 (RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@139a0988 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 272 (RS-EventLoopGroup-3-1): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 248 (RS-EventLoopGroup-1-1): State: RUNNABLE Blocked count: 1 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 247 (HBase-Metrics2-1): State: TIMED_WAITING Blocked count: 0 Waited count: 288 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 235 (ProcessThread(sid:0 cport:56841):): State: WAITING Blocked count: 1 Waited count: 654 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@133caf02 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:122) Thread 234 (SyncThread:0): State: WAITING Blocked count: 0 Waited count: 626 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5c669c88 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:127) Thread 233 (SessionTracker): State: TIMED_WAITING Blocked count: 0 Waited count: 133 Stack: java.lang.Object.wait(Native Method) org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146) Thread 232 (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:56841): State: RUNNABLE Blocked count: 9 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:173) java.lang.Thread.run(Thread.java:748) Thread 229 (java.util.concurrent.ThreadPoolExecutor$Worker@1d726817[State = -1, empty queue]): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 224 (refreshUsed-/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data6/current/BP-35161775-127.0.1.1-1517713219102): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:132) java.lang.Thread.run(Thread.java:748) Thread 222 (refreshUsed-/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data5/current/BP-35161775-127.0.1.1-1517713219102): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:132) java.lang.Thread.run(Thread.java:748) Thread 218 (org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl$LazyWriter@2327f0e2): State: TIMED_WAITING Blocked count: 0 Waited count: 5 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl$LazyWriter.run(FsDatasetImpl.java:3071) java.lang.Thread.run(Thread.java:748) Thread 217 (VolumeScannerThread(/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data6)): State: TIMED_WAITING Blocked count: 1 Waited count: 2 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:622) Thread 216 (VolumeScannerThread(/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data5)): State: TIMED_WAITING Blocked count: 1 Waited count: 2 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:622) Thread 213 (IPC Server handler 9 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 297 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 212 (IPC Server handler 8 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 291 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 211 (IPC Server handler 7 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 286 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 210 (IPC Server handler 6 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 308 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 209 (IPC Server handler 5 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 285 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 208 (IPC Server handler 4 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 311 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 207 (IPC Server handler 3 on 43825): State: TIMED_WAITING Blocked count: 1 Waited count: 269 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 206 (IPC Server handler 2 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 308 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 205 (IPC Server handler 1 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 324 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 204 (IPC Server handler 0 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 267 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 199 (IPC Server listener on 43825): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener.run(Server.java:807) Thread 202 (IPC Server Responder): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.ipc.Server$Responder.doRunLoop(Server.java:982) org.apache.hadoop.ipc.Server$Responder.run(Server.java:965) Thread 161 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@3e65225e): State: RUNNABLE Blocked count: 1 Waited count: 0 Stack: sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:100) org.apache.hadoop.hdfs.net.TcpPeerServer.accept(TcpPeerServer.java:141) org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:135) java.lang.Thread.run(Thread.java:748) Thread 203 (DataNode: [[[DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data5/, [DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data6/]] heartbeating to localhost/127.0.0.1:44491): State: TIMED_WAITING Blocked count: 412 Waited count: 964 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.IncrementalBlockReportManager.waitTillNextIBR(IncrementalBlockReportManager.java:130) org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:542) org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:659) java.lang.Thread.run(Thread.java:748) Thread 201 (IPC Server idle connection scanner for port 43825): State: TIMED_WAITING Blocked count: 1 Waited count: 28 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 200 (Socket Reader #1 for port 43825): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:745) org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:724) Thread 198 (org.apache.hadoop.util.JvmPauseMonitor$Monitor@3369a0f5): State: TIMED_WAITING Blocked count: 0 Waited count: 529 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:182) java.lang.Thread.run(Thread.java:748) Thread 182 (nioEventLoopGroup-6-1): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) java.lang.Thread.run(Thread.java:748) Thread 181 (Timer-3): State: TIMED_WAITING Blocked count: 0 Waited count: 9 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 180 (java.util.concurrent.ThreadPoolExecutor$Worker@4231ed0c[State = -1, empty queue]): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 175 (refreshUsed-/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data4/current/BP-35161775-127.0.1.1-1517713219102): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:132) java.lang.Thread.run(Thread.java:748) Thread 173 (refreshUsed-/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data3/current/BP-35161775-127.0.1.1-1517713219102): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:132) java.lang.Thread.run(Thread.java:748) Thread 169 (org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl$LazyWriter@221a6ce4): State: TIMED_WAITING Blocked count: 0 Waited count: 5 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl$LazyWriter.run(FsDatasetImpl.java:3071) java.lang.Thread.run(Thread.java:748) Thread 168 (VolumeScannerThread(/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data4)): State: TIMED_WAITING Blocked count: 1 Waited count: 2 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:622) Thread 167 (VolumeScannerThread(/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data3)): State: TIMED_WAITING Blocked count: 1 Waited count: 2 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:622) Thread 164 (103115058@qtp-229466626-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:41633): State: RUNNABLE Blocked count: 2 Waited count: 1 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498) org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192) org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Thread 163 (498148987@qtp-229466626-0): State: TIMED_WAITING Blocked count: 0 Waited count: 5 Stack: java.lang.Object.wait(Native Method) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626) Thread 162 (pool-12-thread-1): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 156 (IPC Server handler 9 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 311 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 155 (IPC Server handler 8 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 267 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 154 (IPC Server handler 7 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 268 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 153 (IPC Server handler 6 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 310 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 152 (IPC Server handler 5 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 291 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 151 (IPC Server handler 4 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 273 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 150 (IPC Server handler 3 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 297 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 149 (IPC Server handler 2 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 307 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 148 (IPC Server handler 1 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 269 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 147 (IPC Server handler 0 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 266 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 142 (IPC Server listener on 33493): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener.run(Server.java:807) Thread 145 (IPC Server Responder): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.ipc.Server$Responder.doRunLoop(Server.java:982) org.apache.hadoop.ipc.Server$Responder.run(Server.java:965) Thread 102 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@26506cf6): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:100) org.apache.hadoop.hdfs.net.TcpPeerServer.accept(TcpPeerServer.java:141) org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:135) java.lang.Thread.run(Thread.java:748) Thread 146 (DataNode: [[[DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data3/, [DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data4/]] heartbeating to localhost/127.0.0.1:44491): State: TIMED_WAITING Blocked count: 393 Waited count: 959 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.IncrementalBlockReportManager.waitTillNextIBR(IncrementalBlockReportManager.java:130) org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:542) org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:659) java.lang.Thread.run(Thread.java:748) Thread 144 (IPC Server idle connection scanner for port 33493): State: TIMED_WAITING Blocked count: 1 Waited count: 28 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 143 (Socket Reader #1 for port 33493): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:745) org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:724) Thread 141 (org.apache.hadoop.util.JvmPauseMonitor$Monitor@1d004c0c): State: TIMED_WAITING Blocked count: 0 Waited count: 531 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:182) java.lang.Thread.run(Thread.java:748) Thread 125 (nioEventLoopGroup-4-1): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) java.lang.Thread.run(Thread.java:748) Thread 124 (Timer-2): State: TIMED_WAITING Blocked count: 0 Waited count: 9 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 123 (java.util.concurrent.ThreadPoolExecutor$Worker@3adb50a8[State = -1, empty queue]): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 118 (refreshUsed-/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data1/current/BP-35161775-127.0.1.1-1517713219102): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:132) java.lang.Thread.run(Thread.java:748) Thread 117 (refreshUsed-/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data2/current/BP-35161775-127.0.1.1-1517713219102): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:132) java.lang.Thread.run(Thread.java:748) Thread 112 (org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl$LazyWriter@7e498b6b): State: TIMED_WAITING Blocked count: 0 Waited count: 5 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl$LazyWriter.run(FsDatasetImpl.java:3071) java.lang.Thread.run(Thread.java:748) Thread 111 (VolumeScannerThread(/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data2)): State: TIMED_WAITING Blocked count: 11 Waited count: 2 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:622) Thread 110 (VolumeScannerThread(/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data1)): State: TIMED_WAITING Blocked count: 7 Waited count: 2 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:622) Thread 106 (IPC Client (1670261399) connection to localhost/127.0.0.1:44491 from zhangduo): State: TIMED_WAITING Blocked count: 883 Waited count: 884 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:934) org.apache.hadoop.ipc.Client$Connection.run(Client.java:979) Thread 105 (1512126834@qtp-261780187-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:34039): State: RUNNABLE Blocked count: 1 Waited count: 1 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498) org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192) org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Thread 104 (1495196264@qtp-261780187-0): State: TIMED_WAITING Blocked count: 0 Waited count: 5 Stack: java.lang.Object.wait(Native Method) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626) Thread 103 (pool-6-thread-1): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 97 (IPC Server handler 9 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 312 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 96 (IPC Server handler 8 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 309 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 95 (IPC Server handler 7 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 300 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 94 (IPC Server handler 6 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 296 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 93 (IPC Server handler 5 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 298 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 92 (IPC Server handler 4 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 295 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 91 (IPC Server handler 3 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 327 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 90 (IPC Server handler 2 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 308 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 89 (IPC Server handler 1 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 316 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 88 (IPC Server handler 0 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 305 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 83 (IPC Server listener on 45595): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener.run(Server.java:807) Thread 86 (IPC Server Responder): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.ipc.Server$Responder.doRunLoop(Server.java:982) org.apache.hadoop.ipc.Server$Responder.run(Server.java:965) Thread 60 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@5ec0030b): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:100) org.apache.hadoop.hdfs.net.TcpPeerServer.accept(TcpPeerServer.java:141) org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:135) java.lang.Thread.run(Thread.java:748) Thread 87 (DataNode: [[[DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data1/, [DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data2/]] heartbeating to localhost/127.0.0.1:44491): State: TIMED_WAITING Blocked count: 406 Waited count: 961 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.IncrementalBlockReportManager.waitTillNextIBR(IncrementalBlockReportManager.java:130) org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:542) org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:659) java.lang.Thread.run(Thread.java:748) Thread 85 (IPC Server idle connection scanner for port 45595): State: TIMED_WAITING Blocked count: 1 Waited count: 28 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 84 (Socket Reader #1 for port 45595): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:745) org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:724) Thread 82 (org.apache.hadoop.util.JvmPauseMonitor$Monitor@74095914): State: TIMED_WAITING Blocked count: 0 Waited count: 533 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:182) java.lang.Thread.run(Thread.java:748) Thread 65 (nioEventLoopGroup-2-1): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) java.lang.Thread.run(Thread.java:748) Thread 64 (Timer-1): State: TIMED_WAITING Blocked count: 0 Waited count: 9 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 63 (1020827856@qtp-415074966-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:40851): State: RUNNABLE Blocked count: 1 Waited count: 1 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498) org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192) org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Thread 62 (1498522151@qtp-415074966-0): State: TIMED_WAITING Blocked count: 0 Waited count: 5 Stack: java.lang.Object.wait(Native Method) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626) Thread 61 (pool-4-thread-1): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 55 (CacheReplicationMonitor(1454695942)): State: TIMED_WAITING Blocked count: 0 Waited count: 10 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163) org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor.run(CacheReplicationMonitor.java:181) Thread 54 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$LazyPersistFileScrubber@5654e22b): State: TIMED_WAITING Blocked count: 0 Waited count: 2 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.namenode.FSNamesystem$LazyPersistFileScrubber.run(FSNamesystem.java:4725) java.lang.Thread.run(Thread.java:748) Thread 53 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeEditLogRoller@68702a45): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeEditLogRoller.run(FSNamesystem.java:4642) java.lang.Thread.run(Thread.java:748) Thread 52 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor@3d921e48): State: TIMED_WAITING Blocked count: 0 Waited count: 54 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor.run(FSNamesystem.java:4598) java.lang.Thread.run(Thread.java:748) Thread 51 (org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@fe27903): State: TIMED_WAITING Blocked count: 0 Waited count: 135 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:431) java.lang.Thread.run(Thread.java:748) Thread 50 (IPC Server handler 9 on 44491): State: TIMED_WAITING Blocked count: 3 Waited count: 467 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 49 (IPC Server handler 8 on 44491): State: TIMED_WAITING Blocked count: 2 Waited count: 456 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 48 (IPC Server handler 7 on 44491): State: TIMED_WAITING Blocked count: 3 Waited count: 465 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 47 (IPC Server handler 6 on 44491): State: TIMED_WAITING Blocked count: 3 Waited count: 473 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 46 (IPC Server handler 5 on 44491): State: TIMED_WAITING Blocked count: 17 Waited count: 468 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 45 (IPC Server handler 4 on 44491): State: TIMED_WAITING Blocked count: 5 Waited count: 466 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 44 (IPC Server handler 3 on 44491): State: TIMED_WAITING Blocked count: 6 Waited count: 457 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 43 (IPC Server handler 2 on 44491): State: TIMED_WAITING Blocked count: 8 Waited count: 464 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 42 (IPC Server handler 1 on 44491): State: TIMED_WAITING Blocked count: 4 Waited count: 462 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 41 (IPC Server handler 0 on 44491): State: TIMED_WAITING Blocked count: 7 Waited count: 468 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 31 (IPC Server listener on 44491): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener.run(Server.java:807) Thread 34 (IPC Server Responder): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.ipc.Server$Responder.doRunLoop(Server.java:982) org.apache.hadoop.ipc.Server$Responder.run(Server.java:965) Thread 29 (Block report processor): State: WAITING Blocked count: 2 Waited count: 140 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@c5f3502 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403) org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.processQueue(BlockManager.java:3854) org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.run(BlockManager.java:3843) Thread 28 (org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@2748be45): State: TIMED_WAITING Blocked count: 0 Waited count: 91 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3638) java.lang.Thread.run(Thread.java:748) Thread 30 (org.apache.hadoop.hdfs.server.blockmanagement.HeartbeatManager$Monitor@1706a704): State: TIMED_WAITING Blocked count: 0 Waited count: 54 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.blockmanagement.HeartbeatManager$Monitor.run(HeartbeatManager.java:381) java.lang.Thread.run(Thread.java:748) Thread 40 (DecommissionMonitor-0): State: TIMED_WAITING Blocked count: 0 Waited count: 90 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 39 (org.apache.hadoop.hdfs.server.blockmanagement.PendingReplicationBlocks$PendingReplicationMonitor@28cb973d): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.blockmanagement.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:221) java.lang.Thread.run(Thread.java:748) Thread 35 (org.apache.hadoop.util.JvmPauseMonitor$Monitor@7501869b): State: TIMED_WAITING Blocked count: 0 Waited count: 535 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:182) java.lang.Thread.run(Thread.java:748) Thread 33 (IPC Server idle connection scanner for port 44491): State: TIMED_WAITING Blocked count: 1 Waited count: 28 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 32 (Socket Reader #1 for port 44491): State: RUNNABLE Blocked count: 1 Waited count: 1 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:745) org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:724) Thread 27 (Timer-0): State: TIMED_WAITING Blocked count: 1 Waited count: 10 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 25 (1941945463@qtp-2007783481-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:43361): State: RUNNABLE Blocked count: 1 Waited count: 1 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498) org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192) org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Thread 24 (1848959351@qtp-2007783481-0): State: TIMED_WAITING Blocked count: 0 Waited count: 5 Stack: java.lang.Object.wait(Native Method) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626) Thread 23 (pool-2-thread-1): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 14 (org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner): State: WAITING Blocked count: 4 Waited count: 5 Waiting on java.lang.ref.ReferenceQueue$Lock@688d8764 Stack: java.lang.Object.wait(Native Method) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164) org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:3063) java.lang.Thread.run(Thread.java:748) Thread 13 (Time-limited test): State: RUNNABLE Blocked count: 91 Waited count: 230 Stack: sun.management.ThreadImpl.getThreadInfo1(Native Method) sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:178) sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:139) org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:168) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) org.apache.hadoop.hbase.util.Threads$PrintThreadInfoLazyHolder$1.printThreadInfo(Threads.java:294) org.apache.hadoop.hbase.util.Threads.printThreadInfo(Threads.java:341) org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:135) org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:367) org.apache.hadoop.hbase.MiniHBaseCluster.waitUntilShutDown(MiniHBaseCluster.java:865) org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:1130) org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:1105) org.apache.hadoop.hbase.TestFullLogReconstruction.tearDownAfterClass(TestFullLogReconstruction.java:68) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) Thread 9 (ReaderThread): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: java.net.SocketInputStream.socketRead0(Native Method) java.net.SocketInputStream.socketRead(SocketInputStream.java:116) java.net.SocketInputStream.read(SocketInputStream.java:171) java.net.SocketInputStream.read(SocketInputStream.java:141) sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284) sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326) sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178) java.io.InputStreamReader.read(InputStreamReader.java:184) java.io.BufferedReader.fill(BufferedReader.java:161) java.io.BufferedReader.readLine(BufferedReader.java:324) java.io.BufferedReader.readLine(BufferedReader.java:389) org.eclipse.jdt.internal.junit.runner.RemoteTestRunner$ReaderThread.run(RemoteTestRunner.java:148) Thread 4 (Signal Dispatcher): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: Thread 3 (Finalizer): State: WAITING Blocked count: 266 Waited count: 15 Waiting on java.lang.ref.ReferenceQueue$Lock@60cb01b5 Stack: java.lang.Object.wait(Native Method) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164) java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209) Thread 2 (Reference Handler): State: WAITING Blocked count: 21 Waited count: 14 Waiting on java.lang.ref.Reference$Lock@6a276aab Stack: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) java.lang.ref.Reference.tryHandlePending(Reference.java:191) java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153) Thread 1 (main): State: TIMED_WAITING Blocked count: 0 Waited count: 2 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.FutureTask.awaitDone(FutureTask.java:426) java.util.concurrent.FutureTask.get(FutureTask.java:204) org.junit.internal.runners.statements.FailOnTimeout.getResult(FailOnTimeout.java:141) org.junit.internal.runners.statements.FailOnTimeout.evaluate(FailOnTimeout.java:127) org.junit.rules.RunRules.evaluate(RunRules.java:20) org.junit.runners.ParentRunner.run(ParentRunner.java:363) org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86) org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38) org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:538) org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:760) org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:460) org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:206) 2018-02-04 11:05:27,562 DEBUG [M:0;ubuntu:39063-MemStoreChunkPool Statistics] regionserver.ChunkCreator$MemStoreChunkPool$StatisticsThread(339): Stats: current pool size=24,created chunk count=29,reused chunk count=0,reuseRatio=0.00% 2018-02-04 11:05:28,705 DEBUG [ReadOnlyZKClient-localhost:56841@0x143faef3] zookeeper.ReadOnlyZKClient(316): 0x143faef3 to localhost:56841 inactive for 60000ms; closing (Will reconnect when new requests) 2018-02-04 11:05:28,711 DEBUG [ReadOnlyZKClient-localhost:56841@0x46e70b17] zookeeper.ReadOnlyZKClient(316): 0x46e70b17 to localhost:56841 inactive for 60000ms; closing (Will reconnect when new requests) 2018-02-04 11:05:28,714 DEBUG [ReadOnlyZKClient-localhost:56841@0x796736be] zookeeper.ReadOnlyZKClient(316): 0x796736be to localhost:56841 inactive for 60000ms; closing (Will reconnect when new requests) 2018-02-04 11:05:36,159 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2018-02-04 11:05:40,120 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-0] helpers.MarkerIgnoringBase(159): ***** ABORTING region server ubuntu,39267,1517713227316: Replay of WAL required. Forcing server shutdown ***** org.apache.hadoop.hbase.DroppedSnapshotException: region: tabletest,ttt,1517713232695.31b3e09f2db65dd6057a91f45cb8b163. at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2667) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2356) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.hbase.exceptions.TimeoutIOException: Failed to get sync result after 300000 ms for txid=136, WAL system stuck? at org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:136) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:718) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:605) at org.apache.hadoop.hbase.regionserver.wal.WALUtil.doFullAppendTransaction(WALUtil.java:154) at org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeFlushMarker(WALUtil.java:81) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2645) ... 10 more 2018-02-04 11:05:40,120 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-0] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-02-04 11:05:40,133 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegionServer(2350): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 881852416, "init" : 197132288, "max" : 2794455040, "used" : 252389760 }, "NonHeapMemoryUsage" : { "committed" : 123625472, "init" : 2555904, "max" : -1, "used" : 121677040 }, "Verbose" : false, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : -1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 6227078, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 207, "TotalCallTime_min" : 0, "TotalCallTime_max" : 0, "TotalCallTime_mean" : 0, "TotalCallTime_25th_percentile" : 0, "TotalCallTime_median" : 0, "TotalCallTime_75th_percentile" : 0, "TotalCallTime_90th_percentile" : 0, "TotalCallTime_95th_percentile" : 0, "TotalCallTime_98th_percentile" : 0, "TotalCallTime_99th_percentile" : 0, "TotalCallTime_99.9th_percentile" : 0, "TotalCallTime_TimeRangeCount_600000-inf" : 207, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 207, "ResponseSize_min" : 0, "ResponseSize_max" : 0, "ResponseSize_mean" : 0, "ResponseSize_25th_percentile" : 0, "ResponseSize_median" : 0, "ResponseSize_75th_percentile" : 0, "ResponseSize_90th_percentile" : 0, "ResponseSize_95th_percentile" : 0, "ResponseSize_98th_percentile" : 0, "ResponseSize_99th_percentile" : 0, "ResponseSize_99.9th_percentile" : 0, "ResponseSize_SizeRangeCount_100000000-inf" : 207, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 207, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 0, "ProcessCallTime_mean" : 0, "ProcessCallTime_25th_percentile" : 0, "ProcessCallTime_median" : 0, "ProcessCallTime_75th_percentile" : 0, "ProcessCallTime_90th_percentile" : 0, "ProcessCallTime_95th_percentile" : 0, "ProcessCallTime_98th_percentile" : 0, "ProcessCallTime_99th_percentile" : 0, "ProcessCallTime_99.9th_percentile" : 0, "ProcessCallTime_TimeRangeCount_600000-inf" : 207, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 9, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 207, "RequestSize_min" : 0, "RequestSize_max" : 0, "RequestSize_mean" : 0, "RequestSize_25th_percentile" : 0, "RequestSize_median" : 0, "RequestSize_75th_percentile" : 0, "RequestSize_90th_percentile" : 0, "RequestSize_95th_percentile" : 0, "RequestSize_98th_percentile" : 0, "RequestSize_99th_percentile" : 0, "RequestSize_99.9th_percentile" : 0, "RequestSize_SizeRangeCount_100000000-inf" : 207, "sentBytes" : 2283235, "QueueCallTime_num_ops" : 207, "QueueCallTime_min" : 0, "QueueCallTime_max" : 0, "QueueCallTime_mean" : 0, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 0, "QueueCallTime_90th_percentile" : 0, "QueueCallTime_95th_percentile" : 0, "QueueCallTime_98th_percentile" : 0, "QueueCallTime_99th_percentile" : 0, "QueueCallTime_99.9th_percentile" : 0, "QueueCallTime_TimeRangeCount_600000-inf" : 207, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:56841", "tag.serverName" : "ubuntu,39267,1517713227316", "tag.clusterId" : "990c040f-15f4-4bef-a30c-08f39bc273d8", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "regionCount" : 8, "storeCount" : 8, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 2, "memStoreSize" : 534358, "storeFileSize" : 66522, "maxStoreFileAge" : 294103, "minStoreFileAge" : 293655, "avgStoreFileAge" : 293879, "numReferenceFiles" : 0, "regionServerStartTime" : 1517713227316, "averageRegionSize" : 75110, "totalRequestCount" : 32, "totalRowActionRequestCount" : 34451, "readRequestCount" : 9844, "filteredReadRequestCount" : 0, "writeRequestCount" : 24607, "rpcGetRequestCount" : 4, "rpcScanRequestCount" : 19, "rpcMultiRequestCount" : 5, "rpcMutateRequestCount" : 2, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 656, "staticIndexSize" : 80, "staticBloomSize" : 2048, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1116962296, "blockCacheCount" : 0, "blockCacheSize" : 819720, "blockCacheHitCount" : 0, "blockCacheHitCountPrimary" : 0, "blockCacheMissCount" : 0, "blockCacheMissCountPrimary" : 0, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 0.0, "blockCacheExpressHitPercent" : 0.0, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 0, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 0, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 0, "l1CacheMissCount" : 0, "l1CacheHitRatio" : NaN, "l1CacheMissRatio" : NaN, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 10545, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 400710, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 19, "ScanTime_min" : 0, "ScanTime_max" : 0, "ScanTime_mean" : 0, "ScanTime_25th_percentile" : 0, "ScanTime_median" : 0, "ScanTime_75th_percentile" : 0, "ScanTime_90th_percentile" : 0, "ScanTime_95th_percentile" : 0, "ScanTime_98th_percentile" : 0, "ScanTime_99th_percentile" : 0, "ScanTime_99.9th_percentile" : 0, "ScanTime_TimeRangeCount_600000-inf" : 19, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 2, "Put_min" : 0, "Put_max" : 0, "Put_mean" : 0, "Put_25th_percentile" : 0, "Put_median" : 0, "Put_75th_percentile" : 0, "Put_90th_percentile" : 0, "Put_95th_percentile" : 0, "Put_98th_percentile" : 0, "Put_99th_percentile" : 0, "Put_99.9th_percentile" : 0, "Put_TimeRangeCount_600000-inf" : 2, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 2, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "FlushMemstoreSize_SizeRangeCount_100000000-inf" : 2, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 45, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "PutBatch_TimeRangeCount_600000-inf" : 45, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 4, "Get_min" : 0, "Get_max" : 0, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 0, "Get_75th_percentile" : 0, "Get_90th_percentile" : 0, "Get_95th_percentile" : 0, "Get_98th_percentile" : 0, "Get_99th_percentile" : 0, "Get_99.9th_percentile" : 0, "Get_TimeRangeCount_600000-inf" : 4, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 66522, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 2, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "FlushTime_TimeRangeCount_600000-inf" : 2, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 19, "ScanSize_min" : 0, "ScanSize_max" : 0, "ScanSize_mean" : 0, "ScanSize_25th_percentile" : 0, "ScanSize_median" : 0, "ScanSize_75th_percentile" : 0, "ScanSize_90th_percentile" : 0, "ScanSize_95th_percentile" : 0, "ScanSize_98th_percentile" : 0, "ScanSize_99th_percentile" : 0, "ScanSize_99.9th_percentile" : 0, "ScanSize_SizeRangeCount_100000000-inf" : 19, "slowGetCount" : 0, "flushedMemstoreBytes" : 267140, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 2, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "FlushOutputSize_SizeRangeCount_100000000-inf" : 2, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-02-04 11:05:40,135 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] ipc.NettyRpcConnection(254): Connecting to ubuntu/127.0.1.1:39063 2018-02-04 11:05:40,143 WARN [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegionServer(2371): Unable to report fatal error to master org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: java.net.ConnectException: Call to ubuntu/127.0.1.1:39063 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: syscall:getsockopt(..) failed: Connection refused: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$0(AbstractRpcClient.java:324) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:571) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.reportRSFatalError(RegionServerStatusProtos.java:13568) at org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:2368) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.abortRegionServer(MiniHBaseCluster.java:208) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$1(MiniHBaseCluster.java:207) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$2.run(MiniHBaseCluster.java:201) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.abort(MiniHBaseCluster.java:198) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2680) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2356) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.net.ConnectException: Call to ubuntu/127.0.1.1:39063 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: syscall:getsockopt(..) failed: Connection refused: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:165) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$3(AbstractRpcClient.java:375) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:1) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.BufferCallBeforeInitHandler.userEventTriggered(BufferCallBeforeInitHandler.java:92) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.userEventTriggered(DefaultChannelPipeline.java:1377) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireUserEventTriggered(DefaultChannelPipeline.java:929) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.failInit(NettyRpcConnection.java:179) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.access$5(NettyRpcConnection.java:176) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:267) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:1) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:122) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:659) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:678) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:552) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:304) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) ... 1 more Caused by: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: syscall:getsockopt(..) failed: Connection refused: ubuntu/127.0.1.1:39063 at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(..)(Unknown Source) Caused by: org.apache.hbase.thirdparty.io.netty.channel.unix.Errors$NativeConnectException: syscall:getsockopt(..) failed: Connection refused ... 1 more 2018-02-04 11:05:40,143 DEBUG [RS-EventLoopGroup-5-7] ipc.FailedServers(56): Added failed server with address ubuntu/127.0.1.1:39063 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: syscall:getsockopt(..) failed: Connection refused: ubuntu/127.0.1.1:39063 2018-02-04 11:05:40,149 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-0] helpers.MarkerIgnoringBase(159): ***** ABORTING region server ubuntu,39267,1517713227316: Unrecoverable exception while closing region tabletest,ttt,1517713232695.31b3e09f2db65dd6057a91f45cb8b163., still finishing close ***** org.apache.hadoop.hbase.DroppedSnapshotException: region: tabletest,ttt,1517713232695.31b3e09f2db65dd6057a91f45cb8b163. at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2667) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2356) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.hbase.exceptions.TimeoutIOException: Failed to get sync result after 300000 ms for txid=136, WAL system stuck? at org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:136) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:718) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:605) at org.apache.hadoop.hbase.regionserver.wal.WALUtil.doFullAppendTransaction(WALUtil.java:154) at org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeFlushMarker(WALUtil.java:81) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2645) ... 10 more 2018-02-04 11:05:40,149 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-0] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-02-04 11:05:40,153 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegionServer(2350): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 881852416, "init" : 197132288, "max" : 2794455040, "used" : 253652312 }, "NonHeapMemoryUsage" : { "committed" : 123625472, "init" : 2555904, "max" : -1, "used" : 121699104 }, "Verbose" : false, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : -1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 6227078, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 207, "TotalCallTime_min" : 0, "TotalCallTime_max" : 0, "TotalCallTime_mean" : 0, "TotalCallTime_25th_percentile" : 0, "TotalCallTime_median" : 0, "TotalCallTime_75th_percentile" : 0, "TotalCallTime_90th_percentile" : 0, "TotalCallTime_95th_percentile" : 0, "TotalCallTime_98th_percentile" : 0, "TotalCallTime_99th_percentile" : 0, "TotalCallTime_99.9th_percentile" : 0, "TotalCallTime_TimeRangeCount_600000-inf" : 207, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 207, "ResponseSize_min" : 0, "ResponseSize_max" : 0, "ResponseSize_mean" : 0, "ResponseSize_25th_percentile" : 0, "ResponseSize_median" : 0, "ResponseSize_75th_percentile" : 0, "ResponseSize_90th_percentile" : 0, "ResponseSize_95th_percentile" : 0, "ResponseSize_98th_percentile" : 0, "ResponseSize_99th_percentile" : 0, "ResponseSize_99.9th_percentile" : 0, "ResponseSize_SizeRangeCount_100000000-inf" : 207, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 207, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 0, "ProcessCallTime_mean" : 0, "ProcessCallTime_25th_percentile" : 0, "ProcessCallTime_median" : 0, "ProcessCallTime_75th_percentile" : 0, "ProcessCallTime_90th_percentile" : 0, "ProcessCallTime_95th_percentile" : 0, "ProcessCallTime_98th_percentile" : 0, "ProcessCallTime_99th_percentile" : 0, "ProcessCallTime_99.9th_percentile" : 0, "ProcessCallTime_TimeRangeCount_600000-inf" : 207, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 9, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 207, "RequestSize_min" : 0, "RequestSize_max" : 0, "RequestSize_mean" : 0, "RequestSize_25th_percentile" : 0, "RequestSize_median" : 0, "RequestSize_75th_percentile" : 0, "RequestSize_90th_percentile" : 0, "RequestSize_95th_percentile" : 0, "RequestSize_98th_percentile" : 0, "RequestSize_99th_percentile" : 0, "RequestSize_99.9th_percentile" : 0, "RequestSize_SizeRangeCount_100000000-inf" : 207, "sentBytes" : 2283235, "QueueCallTime_num_ops" : 207, "QueueCallTime_min" : 0, "QueueCallTime_max" : 0, "QueueCallTime_mean" : 0, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 0, "QueueCallTime_90th_percentile" : 0, "QueueCallTime_95th_percentile" : 0, "QueueCallTime_98th_percentile" : 0, "QueueCallTime_99th_percentile" : 0, "QueueCallTime_99.9th_percentile" : 0, "QueueCallTime_TimeRangeCount_600000-inf" : 207, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:56841", "tag.serverName" : "ubuntu,39267,1517713227316", "tag.clusterId" : "990c040f-15f4-4bef-a30c-08f39bc273d8", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "regionCount" : 8, "storeCount" : 8, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 2, "memStoreSize" : 534358, "storeFileSize" : 66522, "maxStoreFileAge" : 294103, "minStoreFileAge" : 293655, "avgStoreFileAge" : 293879, "numReferenceFiles" : 0, "regionServerStartTime" : 1517713227316, "averageRegionSize" : 75110, "totalRequestCount" : 32, "totalRowActionRequestCount" : 34451, "readRequestCount" : 9844, "filteredReadRequestCount" : 0, "writeRequestCount" : 24607, "rpcGetRequestCount" : 4, "rpcScanRequestCount" : 19, "rpcMultiRequestCount" : 5, "rpcMutateRequestCount" : 2, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 656, "staticIndexSize" : 80, "staticBloomSize" : 2048, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1116962296, "blockCacheCount" : 0, "blockCacheSize" : 819720, "blockCacheHitCount" : 0, "blockCacheHitCountPrimary" : 0, "blockCacheMissCount" : 0, "blockCacheMissCountPrimary" : 0, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 0.0, "blockCacheExpressHitPercent" : 0.0, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 0, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 0, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 0, "l1CacheMissCount" : 0, "l1CacheHitRatio" : NaN, "l1CacheMissRatio" : NaN, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 10545, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 400710, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 19, "ScanTime_min" : 0, "ScanTime_max" : 0, "ScanTime_mean" : 0, "ScanTime_25th_percentile" : 0, "ScanTime_median" : 0, "ScanTime_75th_percentile" : 0, "ScanTime_90th_percentile" : 0, "ScanTime_95th_percentile" : 0, "ScanTime_98th_percentile" : 0, "ScanTime_99th_percentile" : 0, "ScanTime_99.9th_percentile" : 0, "ScanTime_TimeRangeCount_600000-inf" : 19, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 2, "Put_min" : 0, "Put_max" : 0, "Put_mean" : 0, "Put_25th_percentile" : 0, "Put_median" : 0, "Put_75th_percentile" : 0, "Put_90th_percentile" : 0, "Put_95th_percentile" : 0, "Put_98th_percentile" : 0, "Put_99th_percentile" : 0, "Put_99.9th_percentile" : 0, "Put_TimeRangeCount_600000-inf" : 2, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 2, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "FlushMemstoreSize_SizeRangeCount_100000000-inf" : 2, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 45, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "PutBatch_TimeRangeCount_600000-inf" : 45, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 4, "Get_min" : 0, "Get_max" : 0, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 0, "Get_75th_percentile" : 0, "Get_90th_percentile" : 0, "Get_95th_percentile" : 0, "Get_98th_percentile" : 0, "Get_99th_percentile" : 0, "Get_99.9th_percentile" : 0, "Get_TimeRangeCount_600000-inf" : 4, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 66522, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 2, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "FlushTime_TimeRangeCount_600000-inf" : 2, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 19, "ScanSize_min" : 0, "ScanSize_max" : 0, "ScanSize_mean" : 0, "ScanSize_25th_percentile" : 0, "ScanSize_median" : 0, "ScanSize_75th_percentile" : 0, "ScanSize_90th_percentile" : 0, "ScanSize_95th_percentile" : 0, "ScanSize_98th_percentile" : 0, "ScanSize_99th_percentile" : 0, "ScanSize_99.9th_percentile" : 0, "ScanSize_SizeRangeCount_100000000-inf" : 19, "slowGetCount" : 0, "flushedMemstoreBytes" : 267140, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 2, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "FlushOutputSize_SizeRangeCount_100000000-inf" : 2, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-02-04 11:05:40,155 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] ipc.AbstractRpcClient(349): Not trying to connect to ubuntu/127.0.1.1:39063 this server is in the failed servers list 2018-02-04 11:05:40,155 WARN [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegionServer(2371): Unable to report fatal error to master org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: java.io.IOException: Call to ubuntu/127.0.1.1:39063 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$0(AbstractRpcClient.java:324) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:571) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.reportRSFatalError(RegionServerStatusProtos.java:13568) at org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:2368) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.abortRegionServer(MiniHBaseCluster.java:208) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$1(MiniHBaseCluster.java:207) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$2.run(MiniHBaseCluster.java:201) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.abort(MiniHBaseCluster.java:198) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:115) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Call to ubuntu/127.0.1.1:39063 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$3(AbstractRpcClient.java:375) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:1) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:423) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:328) ... 17 more Caused by: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.getConnection(AbstractRpcClient.java:352) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:420) ... 18 more 2018-02-04 11:05:40,156 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-0] executor.EventHandler(186): Caught throwable while processing event M_RS_CLOSE_REGION java.lang.RuntimeException: org.apache.hadoop.hbase.DroppedSnapshotException: region: tabletest,ttt,1517713232695.31b3e09f2db65dd6057a91f45cb8b163. at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:117) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.hbase.DroppedSnapshotException: region: tabletest,ttt,1517713232695.31b3e09f2db65dd6057a91f45cb8b163. at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2667) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2356) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) ... 4 more Caused by: org.apache.hadoop.hbase.exceptions.TimeoutIOException: Failed to get sync result after 300000 ms for txid=136, WAL system stuck? at org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:136) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:718) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:605) at org.apache.hadoop.hbase.regionserver.wal.WALUtil.doFullAppendTransaction(WALUtil.java:154) at org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeFlushMarker(WALUtil.java:81) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2645) ... 10 more 2018-02-04 11:05:40,157 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-0] helpers.MarkerIgnoringBase(159): ***** ABORTING region server ubuntu,39267,1517713227316: Caught throwable while processing event M_RS_CLOSE_REGION ***** java.lang.RuntimeException: org.apache.hadoop.hbase.DroppedSnapshotException: region: tabletest,ttt,1517713232695.31b3e09f2db65dd6057a91f45cb8b163. at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:117) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.hbase.DroppedSnapshotException: region: tabletest,ttt,1517713232695.31b3e09f2db65dd6057a91f45cb8b163. at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2667) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2356) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) ... 4 more Caused by: org.apache.hadoop.hbase.exceptions.TimeoutIOException: Failed to get sync result after 300000 ms for txid=136, WAL system stuck? at org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:136) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:718) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:605) at org.apache.hadoop.hbase.regionserver.wal.WALUtil.doFullAppendTransaction(WALUtil.java:154) at org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeFlushMarker(WALUtil.java:81) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2645) ... 10 more 2018-02-04 11:05:40,157 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-0] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-02-04 11:05:40,160 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegionServer(2350): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 881852416, "init" : 197132288, "max" : 2794455040, "used" : 254797168 }, "NonHeapMemoryUsage" : { "committed" : 123625472, "init" : 2555904, "max" : -1, "used" : 121700144 }, "Verbose" : false, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : -1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 6227078, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 207, "TotalCallTime_min" : 0, "TotalCallTime_max" : 0, "TotalCallTime_mean" : 0, "TotalCallTime_25th_percentile" : 0, "TotalCallTime_median" : 0, "TotalCallTime_75th_percentile" : 0, "TotalCallTime_90th_percentile" : 0, "TotalCallTime_95th_percentile" : 0, "TotalCallTime_98th_percentile" : 0, "TotalCallTime_99th_percentile" : 0, "TotalCallTime_99.9th_percentile" : 0, "TotalCallTime_TimeRangeCount_600000-inf" : 207, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 207, "ResponseSize_min" : 0, "ResponseSize_max" : 0, "ResponseSize_mean" : 0, "ResponseSize_25th_percentile" : 0, "ResponseSize_median" : 0, "ResponseSize_75th_percentile" : 0, "ResponseSize_90th_percentile" : 0, "ResponseSize_95th_percentile" : 0, "ResponseSize_98th_percentile" : 0, "ResponseSize_99th_percentile" : 0, "ResponseSize_99.9th_percentile" : 0, "ResponseSize_SizeRangeCount_100000000-inf" : 207, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 207, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 0, "ProcessCallTime_mean" : 0, "ProcessCallTime_25th_percentile" : 0, "ProcessCallTime_median" : 0, "ProcessCallTime_75th_percentile" : 0, "ProcessCallTime_90th_percentile" : 0, "ProcessCallTime_95th_percentile" : 0, "ProcessCallTime_98th_percentile" : 0, "ProcessCallTime_99th_percentile" : 0, "ProcessCallTime_99.9th_percentile" : 0, "ProcessCallTime_TimeRangeCount_600000-inf" : 207, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 9, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 207, "RequestSize_min" : 0, "RequestSize_max" : 0, "RequestSize_mean" : 0, "RequestSize_25th_percentile" : 0, "RequestSize_median" : 0, "RequestSize_75th_percentile" : 0, "RequestSize_90th_percentile" : 0, "RequestSize_95th_percentile" : 0, "RequestSize_98th_percentile" : 0, "RequestSize_99th_percentile" : 0, "RequestSize_99.9th_percentile" : 0, "RequestSize_SizeRangeCount_100000000-inf" : 207, "sentBytes" : 2283235, "QueueCallTime_num_ops" : 207, "QueueCallTime_min" : 0, "QueueCallTime_max" : 0, "QueueCallTime_mean" : 0, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 0, "QueueCallTime_90th_percentile" : 0, "QueueCallTime_95th_percentile" : 0, "QueueCallTime_98th_percentile" : 0, "QueueCallTime_99th_percentile" : 0, "QueueCallTime_99.9th_percentile" : 0, "QueueCallTime_TimeRangeCount_600000-inf" : 207, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:56841", "tag.serverName" : "ubuntu,39267,1517713227316", "tag.clusterId" : "990c040f-15f4-4bef-a30c-08f39bc273d8", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "regionCount" : 8, "storeCount" : 8, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 2, "memStoreSize" : 534358, "storeFileSize" : 66522, "maxStoreFileAge" : 294103, "minStoreFileAge" : 293655, "avgStoreFileAge" : 293879, "numReferenceFiles" : 0, "regionServerStartTime" : 1517713227316, "averageRegionSize" : 75110, "totalRequestCount" : 32, "totalRowActionRequestCount" : 34451, "readRequestCount" : 9844, "filteredReadRequestCount" : 0, "writeRequestCount" : 24607, "rpcGetRequestCount" : 4, "rpcScanRequestCount" : 19, "rpcMultiRequestCount" : 5, "rpcMutateRequestCount" : 2, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 656, "staticIndexSize" : 80, "staticBloomSize" : 2048, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1116962296, "blockCacheCount" : 0, "blockCacheSize" : 819720, "blockCacheHitCount" : 0, "blockCacheHitCountPrimary" : 0, "blockCacheMissCount" : 0, "blockCacheMissCountPrimary" : 0, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 0.0, "blockCacheExpressHitPercent" : 0.0, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 0, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 0, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 0, "l1CacheMissCount" : 0, "l1CacheHitRatio" : NaN, "l1CacheMissRatio" : NaN, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 10545, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 400710, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 19, "ScanTime_min" : 0, "ScanTime_max" : 0, "ScanTime_mean" : 0, "ScanTime_25th_percentile" : 0, "ScanTime_median" : 0, "ScanTime_75th_percentile" : 0, "ScanTime_90th_percentile" : 0, "ScanTime_95th_percentile" : 0, "ScanTime_98th_percentile" : 0, "ScanTime_99th_percentile" : 0, "ScanTime_99.9th_percentile" : 0, "ScanTime_TimeRangeCount_600000-inf" : 19, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 2, "Put_min" : 0, "Put_max" : 0, "Put_mean" : 0, "Put_25th_percentile" : 0, "Put_median" : 0, "Put_75th_percentile" : 0, "Put_90th_percentile" : 0, "Put_95th_percentile" : 0, "Put_98th_percentile" : 0, "Put_99th_percentile" : 0, "Put_99.9th_percentile" : 0, "Put_TimeRangeCount_600000-inf" : 2, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 2, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "FlushMemstoreSize_SizeRangeCount_100000000-inf" : 2, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 45, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "PutBatch_TimeRangeCount_600000-inf" : 45, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 4, "Get_min" : 0, "Get_max" : 0, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 0, "Get_75th_percentile" : 0, "Get_90th_percentile" : 0, "Get_95th_percentile" : 0, "Get_98th_percentile" : 0, "Get_99th_percentile" : 0, "Get_99.9th_percentile" : 0, "Get_TimeRangeCount_600000-inf" : 4, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 66522, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 2, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "FlushTime_TimeRangeCount_600000-inf" : 2, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 19, "ScanSize_min" : 0, "ScanSize_max" : 0, "ScanSize_mean" : 0, "ScanSize_25th_percentile" : 0, "ScanSize_median" : 0, "ScanSize_75th_percentile" : 0, "ScanSize_90th_percentile" : 0, "ScanSize_95th_percentile" : 0, "ScanSize_98th_percentile" : 0, "ScanSize_99th_percentile" : 0, "ScanSize_99.9th_percentile" : 0, "ScanSize_SizeRangeCount_100000000-inf" : 19, "slowGetCount" : 0, "flushedMemstoreBytes" : 267140, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 2, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "FlushOutputSize_SizeRangeCount_100000000-inf" : 2, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-02-04 11:05:40,162 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] ipc.AbstractRpcClient(349): Not trying to connect to ubuntu/127.0.1.1:39063 this server is in the failed servers list 2018-02-04 11:05:40,162 WARN [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegionServer(2371): Unable to report fatal error to master org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: java.io.IOException: Call to ubuntu/127.0.1.1:39063 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$0(AbstractRpcClient.java:324) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:571) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.reportRSFatalError(RegionServerStatusProtos.java:13568) at org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:2368) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.abortRegionServer(MiniHBaseCluster.java:208) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$1(MiniHBaseCluster.java:207) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$2.run(MiniHBaseCluster.java:201) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.abort(MiniHBaseCluster.java:198) at org.apache.hadoop.hbase.executor.EventHandler.handleException(EventHandler.java:188) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:106) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Call to ubuntu/127.0.1.1:39063 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$3(AbstractRpcClient.java:375) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:1) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:423) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:328) ... 17 more Caused by: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.getConnection(AbstractRpcClient.java:352) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:420) ... 18 more 2018-02-04 11:05:40,163 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] handler.CloseRegionHandler(92): Processing close of hbase:namespace,,1517713230960.cf37e425ee5be11cb09b53c3671e1599. 2018-02-04 11:05:40,163 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(1483): Closing hbase:namespace,,1517713230960.cf37e425ee5be11cb09b53c3671e1599.: disabling compactions & flushes 2018-02-04 11:05:40,163 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(1522): Updates disabled for region hbase:namespace,,1517713230960.cf37e425ee5be11cb09b53c3671e1599. 2018-02-04 11:05:40,163 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-0] helpers.MarkerIgnoringBase(159): ***** ABORTING region server ubuntu,39267,1517713227316: Unrecoverable exception while closing region hbase:namespace,,1517713230960.cf37e425ee5be11cb09b53c3671e1599., still finishing close ***** java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2369) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2354) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-02-04 11:05:40,164 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-0] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-02-04 11:05:40,165 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegionServer(2350): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 881852416, "init" : 197132288, "max" : 2794455040, "used" : 255820696 }, "NonHeapMemoryUsage" : { "committed" : 123625472, "init" : 2555904, "max" : -1, "used" : 121705928 }, "Verbose" : false, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : -1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 6227078, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 207, "TotalCallTime_min" : 0, "TotalCallTime_max" : 0, "TotalCallTime_mean" : 0, "TotalCallTime_25th_percentile" : 0, "TotalCallTime_median" : 0, "TotalCallTime_75th_percentile" : 0, "TotalCallTime_90th_percentile" : 0, "TotalCallTime_95th_percentile" : 0, "TotalCallTime_98th_percentile" : 0, "TotalCallTime_99th_percentile" : 0, "TotalCallTime_99.9th_percentile" : 0, "TotalCallTime_TimeRangeCount_600000-inf" : 207, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 207, "ResponseSize_min" : 0, "ResponseSize_max" : 0, "ResponseSize_mean" : 0, "ResponseSize_25th_percentile" : 0, "ResponseSize_median" : 0, "ResponseSize_75th_percentile" : 0, "ResponseSize_90th_percentile" : 0, "ResponseSize_95th_percentile" : 0, "ResponseSize_98th_percentile" : 0, "ResponseSize_99th_percentile" : 0, "ResponseSize_99.9th_percentile" : 0, "ResponseSize_SizeRangeCount_100000000-inf" : 207, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 207, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 0, "ProcessCallTime_mean" : 0, "ProcessCallTime_25th_percentile" : 0, "ProcessCallTime_median" : 0, "ProcessCallTime_75th_percentile" : 0, "ProcessCallTime_90th_percentile" : 0, "ProcessCallTime_95th_percentile" : 0, "ProcessCallTime_98th_percentile" : 0, "ProcessCallTime_99th_percentile" : 0, "ProcessCallTime_99.9th_percentile" : 0, "ProcessCallTime_TimeRangeCount_600000-inf" : 207, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 9, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 207, "RequestSize_min" : 0, "RequestSize_max" : 0, "RequestSize_mean" : 0, "RequestSize_25th_percentile" : 0, "RequestSize_median" : 0, "RequestSize_75th_percentile" : 0, "RequestSize_90th_percentile" : 0, "RequestSize_95th_percentile" : 0, "RequestSize_98th_percentile" : 0, "RequestSize_99th_percentile" : 0, "RequestSize_99.9th_percentile" : 0, "RequestSize_SizeRangeCount_100000000-inf" : 207, "sentBytes" : 2283235, "QueueCallTime_num_ops" : 207, "QueueCallTime_min" : 0, "QueueCallTime_max" : 0, "QueueCallTime_mean" : 0, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 0, "QueueCallTime_90th_percentile" : 0, "QueueCallTime_95th_percentile" : 0, "QueueCallTime_98th_percentile" : 0, "QueueCallTime_99th_percentile" : 0, "QueueCallTime_99.9th_percentile" : 0, "QueueCallTime_TimeRangeCount_600000-inf" : 207, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:56841", "tag.serverName" : "ubuntu,39267,1517713227316", "tag.clusterId" : "990c040f-15f4-4bef-a30c-08f39bc273d8", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "regionCount" : 8, "storeCount" : 8, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 2, "memStoreSize" : 534358, "storeFileSize" : 66522, "maxStoreFileAge" : 294103, "minStoreFileAge" : 293655, "avgStoreFileAge" : 293879, "numReferenceFiles" : 0, "regionServerStartTime" : 1517713227316, "averageRegionSize" : 75110, "totalRequestCount" : 32, "totalRowActionRequestCount" : 34451, "readRequestCount" : 9844, "filteredReadRequestCount" : 0, "writeRequestCount" : 24607, "rpcGetRequestCount" : 4, "rpcScanRequestCount" : 19, "rpcMultiRequestCount" : 5, "rpcMutateRequestCount" : 2, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 656, "staticIndexSize" : 80, "staticBloomSize" : 2048, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1116962296, "blockCacheCount" : 0, "blockCacheSize" : 819720, "blockCacheHitCount" : 0, "blockCacheHitCountPrimary" : 0, "blockCacheMissCount" : 0, "blockCacheMissCountPrimary" : 0, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 0.0, "blockCacheExpressHitPercent" : 0.0, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 0, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 0, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 0, "l1CacheMissCount" : 0, "l1CacheHitRatio" : NaN, "l1CacheMissRatio" : NaN, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 10545, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 400710, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 19, "ScanTime_min" : 0, "ScanTime_max" : 0, "ScanTime_mean" : 0, "ScanTime_25th_percentile" : 0, "ScanTime_median" : 0, "ScanTime_75th_percentile" : 0, "ScanTime_90th_percentile" : 0, "ScanTime_95th_percentile" : 0, "ScanTime_98th_percentile" : 0, "ScanTime_99th_percentile" : 0, "ScanTime_99.9th_percentile" : 0, "ScanTime_TimeRangeCount_600000-inf" : 19, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 2, "Put_min" : 0, "Put_max" : 0, "Put_mean" : 0, "Put_25th_percentile" : 0, "Put_median" : 0, "Put_75th_percentile" : 0, "Put_90th_percentile" : 0, "Put_95th_percentile" : 0, "Put_98th_percentile" : 0, "Put_99th_percentile" : 0, "Put_99.9th_percentile" : 0, "Put_TimeRangeCount_600000-inf" : 2, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 2, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "FlushMemstoreSize_SizeRangeCount_100000000-inf" : 2, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 45, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "PutBatch_TimeRangeCount_600000-inf" : 45, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 4, "Get_min" : 0, "Get_max" : 0, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 0, "Get_75th_percentile" : 0, "Get_90th_percentile" : 0, "Get_95th_percentile" : 0, "Get_98th_percentile" : 0, "Get_99th_percentile" : 0, "Get_99.9th_percentile" : 0, "Get_TimeRangeCount_600000-inf" : 4, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 66522, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 2, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "FlushTime_TimeRangeCount_600000-inf" : 2, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 19, "ScanSize_min" : 0, "ScanSize_max" : 0, "ScanSize_mean" : 0, "ScanSize_25th_percentile" : 0, "ScanSize_median" : 0, "ScanSize_75th_percentile" : 0, "ScanSize_90th_percentile" : 0, "ScanSize_95th_percentile" : 0, "ScanSize_98th_percentile" : 0, "ScanSize_99th_percentile" : 0, "ScanSize_99.9th_percentile" : 0, "ScanSize_SizeRangeCount_100000000-inf" : 19, "slowGetCount" : 0, "flushedMemstoreBytes" : 267140, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 2, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "FlushOutputSize_SizeRangeCount_100000000-inf" : 2, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-02-04 11:05:40,165 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] ipc.AbstractRpcClient(349): Not trying to connect to ubuntu/127.0.1.1:39063 this server is in the failed servers list 2018-02-04 11:05:40,166 WARN [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegionServer(2371): Unable to report fatal error to master org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: java.io.IOException: Call to ubuntu/127.0.1.1:39063 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$0(AbstractRpcClient.java:324) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:571) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.reportRSFatalError(RegionServerStatusProtos.java:13568) at org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:2368) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.abortRegionServer(MiniHBaseCluster.java:208) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$1(MiniHBaseCluster.java:207) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$2.run(MiniHBaseCluster.java:201) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.abort(MiniHBaseCluster.java:198) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:115) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Call to ubuntu/127.0.1.1:39063 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$3(AbstractRpcClient.java:375) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:1) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:423) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:328) ... 17 more Caused by: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.getConnection(AbstractRpcClient.java:352) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:420) ... 18 more 2018-02-04 11:05:40,166 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-0] executor.EventHandler(186): Caught throwable while processing event M_RS_CLOSE_REGION java.lang.RuntimeException: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:117) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2369) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2354) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) ... 4 more 2018-02-04 11:05:40,166 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-0] helpers.MarkerIgnoringBase(159): ***** ABORTING region server ubuntu,39267,1517713227316: Caught throwable while processing event M_RS_CLOSE_REGION ***** java.lang.RuntimeException: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:117) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2369) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2354) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) ... 4 more 2018-02-04 11:05:40,166 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-0] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-02-04 11:05:40,168 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegionServer(2350): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 881852416, "init" : 197132288, "max" : 2794455040, "used" : 256872432 }, "NonHeapMemoryUsage" : { "committed" : 123625472, "init" : 2555904, "max" : -1, "used" : 121749736 }, "Verbose" : false, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : -1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 6227078, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 207, "TotalCallTime_min" : 0, "TotalCallTime_max" : 0, "TotalCallTime_mean" : 0, "TotalCallTime_25th_percentile" : 0, "TotalCallTime_median" : 0, "TotalCallTime_75th_percentile" : 0, "TotalCallTime_90th_percentile" : 0, "TotalCallTime_95th_percentile" : 0, "TotalCallTime_98th_percentile" : 0, "TotalCallTime_99th_percentile" : 0, "TotalCallTime_99.9th_percentile" : 0, "TotalCallTime_TimeRangeCount_600000-inf" : 207, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 207, "ResponseSize_min" : 0, "ResponseSize_max" : 0, "ResponseSize_mean" : 0, "ResponseSize_25th_percentile" : 0, "ResponseSize_median" : 0, "ResponseSize_75th_percentile" : 0, "ResponseSize_90th_percentile" : 0, "ResponseSize_95th_percentile" : 0, "ResponseSize_98th_percentile" : 0, "ResponseSize_99th_percentile" : 0, "ResponseSize_99.9th_percentile" : 0, "ResponseSize_SizeRangeCount_100000000-inf" : 207, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 207, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 0, "ProcessCallTime_mean" : 0, "ProcessCallTime_25th_percentile" : 0, "ProcessCallTime_median" : 0, "ProcessCallTime_75th_percentile" : 0, "ProcessCallTime_90th_percentile" : 0, "ProcessCallTime_95th_percentile" : 0, "ProcessCallTime_98th_percentile" : 0, "ProcessCallTime_99th_percentile" : 0, "ProcessCallTime_99.9th_percentile" : 0, "ProcessCallTime_TimeRangeCount_600000-inf" : 207, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 9, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 207, "RequestSize_min" : 0, "RequestSize_max" : 0, "RequestSize_mean" : 0, "RequestSize_25th_percentile" : 0, "RequestSize_median" : 0, "RequestSize_75th_percentile" : 0, "RequestSize_90th_percentile" : 0, "RequestSize_95th_percentile" : 0, "RequestSize_98th_percentile" : 0, "RequestSize_99th_percentile" : 0, "RequestSize_99.9th_percentile" : 0, "RequestSize_SizeRangeCount_100000000-inf" : 207, "sentBytes" : 2283235, "QueueCallTime_num_ops" : 207, "QueueCallTime_min" : 0, "QueueCallTime_max" : 0, "QueueCallTime_mean" : 0, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 0, "QueueCallTime_90th_percentile" : 0, "QueueCallTime_95th_percentile" : 0, "QueueCallTime_98th_percentile" : 0, "QueueCallTime_99th_percentile" : 0, "QueueCallTime_99.9th_percentile" : 0, "QueueCallTime_TimeRangeCount_600000-inf" : 207, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:56841", "tag.serverName" : "ubuntu,39267,1517713227316", "tag.clusterId" : "990c040f-15f4-4bef-a30c-08f39bc273d8", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "regionCount" : 8, "storeCount" : 8, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 2, "memStoreSize" : 534358, "storeFileSize" : 66522, "maxStoreFileAge" : 294103, "minStoreFileAge" : 293655, "avgStoreFileAge" : 293879, "numReferenceFiles" : 0, "regionServerStartTime" : 1517713227316, "averageRegionSize" : 75110, "totalRequestCount" : 32, "totalRowActionRequestCount" : 34451, "readRequestCount" : 9844, "filteredReadRequestCount" : 0, "writeRequestCount" : 24607, "rpcGetRequestCount" : 4, "rpcScanRequestCount" : 19, "rpcMultiRequestCount" : 5, "rpcMutateRequestCount" : 2, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 656, "staticIndexSize" : 80, "staticBloomSize" : 2048, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1116962296, "blockCacheCount" : 0, "blockCacheSize" : 819720, "blockCacheHitCount" : 0, "blockCacheHitCountPrimary" : 0, "blockCacheMissCount" : 0, "blockCacheMissCountPrimary" : 0, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 0.0, "blockCacheExpressHitPercent" : 0.0, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 0, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 0, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 0, "l1CacheMissCount" : 0, "l1CacheHitRatio" : NaN, "l1CacheMissRatio" : NaN, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 10545, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 400710, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 19, "ScanTime_min" : 0, "ScanTime_max" : 0, "ScanTime_mean" : 0, "ScanTime_25th_percentile" : 0, "ScanTime_median" : 0, "ScanTime_75th_percentile" : 0, "ScanTime_90th_percentile" : 0, "ScanTime_95th_percentile" : 0, "ScanTime_98th_percentile" : 0, "ScanTime_99th_percentile" : 0, "ScanTime_99.9th_percentile" : 0, "ScanTime_TimeRangeCount_600000-inf" : 19, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 2, "Put_min" : 0, "Put_max" : 0, "Put_mean" : 0, "Put_25th_percentile" : 0, "Put_median" : 0, "Put_75th_percentile" : 0, "Put_90th_percentile" : 0, "Put_95th_percentile" : 0, "Put_98th_percentile" : 0, "Put_99th_percentile" : 0, "Put_99.9th_percentile" : 0, "Put_TimeRangeCount_600000-inf" : 2, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 2, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "FlushMemstoreSize_SizeRangeCount_100000000-inf" : 2, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 45, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "PutBatch_TimeRangeCount_600000-inf" : 45, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 4, "Get_min" : 0, "Get_max" : 0, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 0, "Get_75th_percentile" : 0, "Get_90th_percentile" : 0, "Get_95th_percentile" : 0, "Get_98th_percentile" : 0, "Get_99th_percentile" : 0, "Get_99.9th_percentile" : 0, "Get_TimeRangeCount_600000-inf" : 4, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 66522, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 2, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "FlushTime_TimeRangeCount_600000-inf" : 2, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 19, "ScanSize_min" : 0, "ScanSize_max" : 0, "ScanSize_mean" : 0, "ScanSize_25th_percentile" : 0, "ScanSize_median" : 0, "ScanSize_75th_percentile" : 0, "ScanSize_90th_percentile" : 0, "ScanSize_95th_percentile" : 0, "ScanSize_98th_percentile" : 0, "ScanSize_99th_percentile" : 0, "ScanSize_99.9th_percentile" : 0, "ScanSize_SizeRangeCount_100000000-inf" : 19, "slowGetCount" : 0, "flushedMemstoreBytes" : 267140, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 2, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "FlushOutputSize_SizeRangeCount_100000000-inf" : 2, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-02-04 11:05:40,168 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] ipc.AbstractRpcClient(349): Not trying to connect to ubuntu/127.0.1.1:39063 this server is in the failed servers list 2018-02-04 11:05:40,168 WARN [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegionServer(2371): Unable to report fatal error to master org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: java.io.IOException: Call to ubuntu/127.0.1.1:39063 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$0(AbstractRpcClient.java:324) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:571) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.reportRSFatalError(RegionServerStatusProtos.java:13568) at org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:2368) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.abortRegionServer(MiniHBaseCluster.java:208) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$1(MiniHBaseCluster.java:207) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$2.run(MiniHBaseCluster.java:201) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.abort(MiniHBaseCluster.java:198) at org.apache.hadoop.hbase.executor.EventHandler.handleException(EventHandler.java:188) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:106) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Call to ubuntu/127.0.1.1:39063 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$3(AbstractRpcClient.java:375) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:1) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:423) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:328) ... 17 more Caused by: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.getConnection(AbstractRpcClient.java:352) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:420) ... 18 more 2018-02-04 11:05:40,169 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] handler.CloseRegionHandler(92): Processing close of tabletest,iii,1517713232695.f1f365dfd0648c0fb5d9848c1db452cc. 2018-02-04 11:05:40,170 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(1483): Closing tabletest,iii,1517713232695.f1f365dfd0648c0fb5d9848c1db452cc.: disabling compactions & flushes 2018-02-04 11:05:40,170 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(1522): Updates disabled for region tabletest,iii,1517713232695.f1f365dfd0648c0fb5d9848c1db452cc. 2018-02-04 11:05:40,170 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-0] helpers.MarkerIgnoringBase(159): ***** ABORTING region server ubuntu,39267,1517713227316: Unrecoverable exception while closing region tabletest,iii,1517713232695.f1f365dfd0648c0fb5d9848c1db452cc., still finishing close ***** java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2369) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2354) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-02-04 11:05:40,170 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-0] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-02-04 11:05:40,171 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegionServer(2350): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 881852416, "init" : 197132288, "max" : 2794455040, "used" : 257873368 }, "NonHeapMemoryUsage" : { "committed" : 123625472, "init" : 2555904, "max" : -1, "used" : 121745552 }, "Verbose" : false, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : -1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 6227078, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 207, "TotalCallTime_min" : 0, "TotalCallTime_max" : 0, "TotalCallTime_mean" : 0, "TotalCallTime_25th_percentile" : 0, "TotalCallTime_median" : 0, "TotalCallTime_75th_percentile" : 0, "TotalCallTime_90th_percentile" : 0, "TotalCallTime_95th_percentile" : 0, "TotalCallTime_98th_percentile" : 0, "TotalCallTime_99th_percentile" : 0, "TotalCallTime_99.9th_percentile" : 0, "TotalCallTime_TimeRangeCount_600000-inf" : 207, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 207, "ResponseSize_min" : 0, "ResponseSize_max" : 0, "ResponseSize_mean" : 0, "ResponseSize_25th_percentile" : 0, "ResponseSize_median" : 0, "ResponseSize_75th_percentile" : 0, "ResponseSize_90th_percentile" : 0, "ResponseSize_95th_percentile" : 0, "ResponseSize_98th_percentile" : 0, "ResponseSize_99th_percentile" : 0, "ResponseSize_99.9th_percentile" : 0, "ResponseSize_SizeRangeCount_100000000-inf" : 207, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 207, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 0, "ProcessCallTime_mean" : 0, "ProcessCallTime_25th_percentile" : 0, "ProcessCallTime_median" : 0, "ProcessCallTime_75th_percentile" : 0, "ProcessCallTime_90th_percentile" : 0, "ProcessCallTime_95th_percentile" : 0, "ProcessCallTime_98th_percentile" : 0, "ProcessCallTime_99th_percentile" : 0, "ProcessCallTime_99.9th_percentile" : 0, "ProcessCallTime_TimeRangeCount_600000-inf" : 207, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 9, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 207, "RequestSize_min" : 0, "RequestSize_max" : 0, "RequestSize_mean" : 0, "RequestSize_25th_percentile" : 0, "RequestSize_median" : 0, "RequestSize_75th_percentile" : 0, "RequestSize_90th_percentile" : 0, "RequestSize_95th_percentile" : 0, "RequestSize_98th_percentile" : 0, "RequestSize_99th_percentile" : 0, "RequestSize_99.9th_percentile" : 0, "RequestSize_SizeRangeCount_100000000-inf" : 207, "sentBytes" : 2283235, "QueueCallTime_num_ops" : 207, "QueueCallTime_min" : 0, "QueueCallTime_max" : 0, "QueueCallTime_mean" : 0, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 0, "QueueCallTime_90th_percentile" : 0, "QueueCallTime_95th_percentile" : 0, "QueueCallTime_98th_percentile" : 0, "QueueCallTime_99th_percentile" : 0, "QueueCallTime_99.9th_percentile" : 0, "QueueCallTime_TimeRangeCount_600000-inf" : 207, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:56841", "tag.serverName" : "ubuntu,39267,1517713227316", "tag.clusterId" : "990c040f-15f4-4bef-a30c-08f39bc273d8", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "regionCount" : 8, "storeCount" : 8, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 2, "memStoreSize" : 534358, "storeFileSize" : 66522, "maxStoreFileAge" : 294103, "minStoreFileAge" : 293655, "avgStoreFileAge" : 293879, "numReferenceFiles" : 0, "regionServerStartTime" : 1517713227316, "averageRegionSize" : 75110, "totalRequestCount" : 32, "totalRowActionRequestCount" : 34451, "readRequestCount" : 9844, "filteredReadRequestCount" : 0, "writeRequestCount" : 24607, "rpcGetRequestCount" : 4, "rpcScanRequestCount" : 19, "rpcMultiRequestCount" : 5, "rpcMutateRequestCount" : 2, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 656, "staticIndexSize" : 80, "staticBloomSize" : 2048, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1116962296, "blockCacheCount" : 0, "blockCacheSize" : 819720, "blockCacheHitCount" : 0, "blockCacheHitCountPrimary" : 0, "blockCacheMissCount" : 0, "blockCacheMissCountPrimary" : 0, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 0.0, "blockCacheExpressHitPercent" : 0.0, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 0, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 0, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 0, "l1CacheMissCount" : 0, "l1CacheHitRatio" : NaN, "l1CacheMissRatio" : NaN, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 10545, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 400710, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 19, "ScanTime_min" : 0, "ScanTime_max" : 0, "ScanTime_mean" : 0, "ScanTime_25th_percentile" : 0, "ScanTime_median" : 0, "ScanTime_75th_percentile" : 0, "ScanTime_90th_percentile" : 0, "ScanTime_95th_percentile" : 0, "ScanTime_98th_percentile" : 0, "ScanTime_99th_percentile" : 0, "ScanTime_99.9th_percentile" : 0, "ScanTime_TimeRangeCount_600000-inf" : 19, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 2, "Put_min" : 0, "Put_max" : 0, "Put_mean" : 0, "Put_25th_percentile" : 0, "Put_median" : 0, "Put_75th_percentile" : 0, "Put_90th_percentile" : 0, "Put_95th_percentile" : 0, "Put_98th_percentile" : 0, "Put_99th_percentile" : 0, "Put_99.9th_percentile" : 0, "Put_TimeRangeCount_600000-inf" : 2, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 2, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "FlushMemstoreSize_SizeRangeCount_100000000-inf" : 2, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 45, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "PutBatch_TimeRangeCount_600000-inf" : 45, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 4, "Get_min" : 0, "Get_max" : 0, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 0, "Get_75th_percentile" : 0, "Get_90th_percentile" : 0, "Get_95th_percentile" : 0, "Get_98th_percentile" : 0, "Get_99th_percentile" : 0, "Get_99.9th_percentile" : 0, "Get_TimeRangeCount_600000-inf" : 4, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 66522, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 2, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "FlushTime_TimeRangeCount_600000-inf" : 2, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 19, "ScanSize_min" : 0, "ScanSize_max" : 0, "ScanSize_mean" : 0, "ScanSize_25th_percentile" : 0, "ScanSize_median" : 0, "ScanSize_75th_percentile" : 0, "ScanSize_90th_percentile" : 0, "ScanSize_95th_percentile" : 0, "ScanSize_98th_percentile" : 0, "ScanSize_99th_percentile" : 0, "ScanSize_99.9th_percentile" : 0, "ScanSize_SizeRangeCount_100000000-inf" : 19, "slowGetCount" : 0, "flushedMemstoreBytes" : 267140, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 2, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "FlushOutputSize_SizeRangeCount_100000000-inf" : 2, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-02-04 11:05:40,172 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] ipc.AbstractRpcClient(349): Not trying to connect to ubuntu/127.0.1.1:39063 this server is in the failed servers list 2018-02-04 11:05:40,172 WARN [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegionServer(2371): Unable to report fatal error to master org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: java.io.IOException: Call to ubuntu/127.0.1.1:39063 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$0(AbstractRpcClient.java:324) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:571) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.reportRSFatalError(RegionServerStatusProtos.java:13568) at org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:2368) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.abortRegionServer(MiniHBaseCluster.java:208) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$1(MiniHBaseCluster.java:207) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$2.run(MiniHBaseCluster.java:201) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.abort(MiniHBaseCluster.java:198) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:115) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Call to ubuntu/127.0.1.1:39063 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$3(AbstractRpcClient.java:375) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:1) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:423) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:328) ... 17 more Caused by: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.getConnection(AbstractRpcClient.java:352) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:420) ... 18 more 2018-02-04 11:05:40,172 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-0] executor.EventHandler(186): Caught throwable while processing event M_RS_CLOSE_REGION java.lang.RuntimeException: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:117) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2369) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2354) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) ... 4 more 2018-02-04 11:05:40,172 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-0] helpers.MarkerIgnoringBase(159): ***** ABORTING region server ubuntu,39267,1517713227316: Caught throwable while processing event M_RS_CLOSE_REGION ***** java.lang.RuntimeException: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:117) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2369) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2354) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) ... 4 more 2018-02-04 11:05:40,173 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-0] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-02-04 11:05:40,174 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegionServer(2350): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 881852416, "init" : 197132288, "max" : 2794455040, "used" : 258923240 }, "NonHeapMemoryUsage" : { "committed" : 123625472, "init" : 2555904, "max" : -1, "used" : 121746888 }, "Verbose" : false, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : -1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 6227078, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 207, "TotalCallTime_min" : 0, "TotalCallTime_max" : 0, "TotalCallTime_mean" : 0, "TotalCallTime_25th_percentile" : 0, "TotalCallTime_median" : 0, "TotalCallTime_75th_percentile" : 0, "TotalCallTime_90th_percentile" : 0, "TotalCallTime_95th_percentile" : 0, "TotalCallTime_98th_percentile" : 0, "TotalCallTime_99th_percentile" : 0, "TotalCallTime_99.9th_percentile" : 0, "TotalCallTime_TimeRangeCount_600000-inf" : 207, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 207, "ResponseSize_min" : 0, "ResponseSize_max" : 0, "ResponseSize_mean" : 0, "ResponseSize_25th_percentile" : 0, "ResponseSize_median" : 0, "ResponseSize_75th_percentile" : 0, "ResponseSize_90th_percentile" : 0, "ResponseSize_95th_percentile" : 0, "ResponseSize_98th_percentile" : 0, "ResponseSize_99th_percentile" : 0, "ResponseSize_99.9th_percentile" : 0, "ResponseSize_SizeRangeCount_100000000-inf" : 207, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 207, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 0, "ProcessCallTime_mean" : 0, "ProcessCallTime_25th_percentile" : 0, "ProcessCallTime_median" : 0, "ProcessCallTime_75th_percentile" : 0, "ProcessCallTime_90th_percentile" : 0, "ProcessCallTime_95th_percentile" : 0, "ProcessCallTime_98th_percentile" : 0, "ProcessCallTime_99th_percentile" : 0, "ProcessCallTime_99.9th_percentile" : 0, "ProcessCallTime_TimeRangeCount_600000-inf" : 207, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 9, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 207, "RequestSize_min" : 0, "RequestSize_max" : 0, "RequestSize_mean" : 0, "RequestSize_25th_percentile" : 0, "RequestSize_median" : 0, "RequestSize_75th_percentile" : 0, "RequestSize_90th_percentile" : 0, "RequestSize_95th_percentile" : 0, "RequestSize_98th_percentile" : 0, "RequestSize_99th_percentile" : 0, "RequestSize_99.9th_percentile" : 0, "RequestSize_SizeRangeCount_100000000-inf" : 207, "sentBytes" : 2283235, "QueueCallTime_num_ops" : 207, "QueueCallTime_min" : 0, "QueueCallTime_max" : 0, "QueueCallTime_mean" : 0, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 0, "QueueCallTime_90th_percentile" : 0, "QueueCallTime_95th_percentile" : 0, "QueueCallTime_98th_percentile" : 0, "QueueCallTime_99th_percentile" : 0, "QueueCallTime_99.9th_percentile" : 0, "QueueCallTime_TimeRangeCount_600000-inf" : 207, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:56841", "tag.serverName" : "ubuntu,39267,1517713227316", "tag.clusterId" : "990c040f-15f4-4bef-a30c-08f39bc273d8", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "regionCount" : 8, "storeCount" : 8, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 2, "memStoreSize" : 534358, "storeFileSize" : 66522, "maxStoreFileAge" : 294103, "minStoreFileAge" : 293655, "avgStoreFileAge" : 293879, "numReferenceFiles" : 0, "regionServerStartTime" : 1517713227316, "averageRegionSize" : 75110, "totalRequestCount" : 32, "totalRowActionRequestCount" : 34451, "readRequestCount" : 9844, "filteredReadRequestCount" : 0, "writeRequestCount" : 24607, "rpcGetRequestCount" : 4, "rpcScanRequestCount" : 19, "rpcMultiRequestCount" : 5, "rpcMutateRequestCount" : 2, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 656, "staticIndexSize" : 80, "staticBloomSize" : 2048, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1116962296, "blockCacheCount" : 0, "blockCacheSize" : 819720, "blockCacheHitCount" : 0, "blockCacheHitCountPrimary" : 0, "blockCacheMissCount" : 0, "blockCacheMissCountPrimary" : 0, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 0.0, "blockCacheExpressHitPercent" : 0.0, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 0, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 0, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 0, "l1CacheMissCount" : 0, "l1CacheHitRatio" : NaN, "l1CacheMissRatio" : NaN, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 10545, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 400710, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 19, "ScanTime_min" : 0, "ScanTime_max" : 0, "ScanTime_mean" : 0, "ScanTime_25th_percentile" : 0, "ScanTime_median" : 0, "ScanTime_75th_percentile" : 0, "ScanTime_90th_percentile" : 0, "ScanTime_95th_percentile" : 0, "ScanTime_98th_percentile" : 0, "ScanTime_99th_percentile" : 0, "ScanTime_99.9th_percentile" : 0, "ScanTime_TimeRangeCount_600000-inf" : 19, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 2, "Put_min" : 0, "Put_max" : 0, "Put_mean" : 0, "Put_25th_percentile" : 0, "Put_median" : 0, "Put_75th_percentile" : 0, "Put_90th_percentile" : 0, "Put_95th_percentile" : 0, "Put_98th_percentile" : 0, "Put_99th_percentile" : 0, "Put_99.9th_percentile" : 0, "Put_TimeRangeCount_600000-inf" : 2, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 2, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "FlushMemstoreSize_SizeRangeCount_100000000-inf" : 2, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 45, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "PutBatch_TimeRangeCount_600000-inf" : 45, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 4, "Get_min" : 0, "Get_max" : 0, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 0, "Get_75th_percentile" : 0, "Get_90th_percentile" : 0, "Get_95th_percentile" : 0, "Get_98th_percentile" : 0, "Get_99th_percentile" : 0, "Get_99.9th_percentile" : 0, "Get_TimeRangeCount_600000-inf" : 4, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 66522, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 2, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "FlushTime_TimeRangeCount_600000-inf" : 2, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 19, "ScanSize_min" : 0, "ScanSize_max" : 0, "ScanSize_mean" : 0, "ScanSize_25th_percentile" : 0, "ScanSize_median" : 0, "ScanSize_75th_percentile" : 0, "ScanSize_90th_percentile" : 0, "ScanSize_95th_percentile" : 0, "ScanSize_98th_percentile" : 0, "ScanSize_99th_percentile" : 0, "ScanSize_99.9th_percentile" : 0, "ScanSize_SizeRangeCount_100000000-inf" : 19, "slowGetCount" : 0, "flushedMemstoreBytes" : 267140, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 2, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "FlushOutputSize_SizeRangeCount_100000000-inf" : 2, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-02-04 11:05:40,174 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] ipc.AbstractRpcClient(349): Not trying to connect to ubuntu/127.0.1.1:39063 this server is in the failed servers list 2018-02-04 11:05:40,174 WARN [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegionServer(2371): Unable to report fatal error to master org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: java.io.IOException: Call to ubuntu/127.0.1.1:39063 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$0(AbstractRpcClient.java:324) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:571) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.reportRSFatalError(RegionServerStatusProtos.java:13568) at org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:2368) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.abortRegionServer(MiniHBaseCluster.java:208) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$1(MiniHBaseCluster.java:207) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$2.run(MiniHBaseCluster.java:201) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.abort(MiniHBaseCluster.java:198) at org.apache.hadoop.hbase.executor.EventHandler.handleException(EventHandler.java:188) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:106) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Call to ubuntu/127.0.1.1:39063 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$3(AbstractRpcClient.java:375) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:1) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:423) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:328) ... 17 more Caused by: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.getConnection(AbstractRpcClient.java:352) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:420) ... 18 more 2018-02-04 11:05:40,175 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] handler.CloseRegionHandler(92): Processing close of tabletest,sss,1517713232695.e063e6b2c2640eb6eafc8737a71fb5f5. 2018-02-04 11:05:40,176 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(1483): Closing tabletest,sss,1517713232695.e063e6b2c2640eb6eafc8737a71fb5f5.: disabling compactions & flushes 2018-02-04 11:05:40,176 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(1522): Updates disabled for region tabletest,sss,1517713232695.e063e6b2c2640eb6eafc8737a71fb5f5. 2018-02-04 11:05:40,176 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-0] helpers.MarkerIgnoringBase(159): ***** ABORTING region server ubuntu,39267,1517713227316: Unrecoverable exception while closing region tabletest,sss,1517713232695.e063e6b2c2640eb6eafc8737a71fb5f5., still finishing close ***** java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2369) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2354) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-02-04 11:05:40,176 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-0] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-02-04 11:05:40,178 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegionServer(2350): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 881852416, "init" : 197132288, "max" : 2794455040, "used" : 259924304 }, "NonHeapMemoryUsage" : { "committed" : 123625472, "init" : 2555904, "max" : -1, "used" : 121755880 }, "Verbose" : false, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : -1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 6227078, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 207, "TotalCallTime_min" : 0, "TotalCallTime_max" : 0, "TotalCallTime_mean" : 0, "TotalCallTime_25th_percentile" : 0, "TotalCallTime_median" : 0, "TotalCallTime_75th_percentile" : 0, "TotalCallTime_90th_percentile" : 0, "TotalCallTime_95th_percentile" : 0, "TotalCallTime_98th_percentile" : 0, "TotalCallTime_99th_percentile" : 0, "TotalCallTime_99.9th_percentile" : 0, "TotalCallTime_TimeRangeCount_600000-inf" : 207, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 207, "ResponseSize_min" : 0, "ResponseSize_max" : 0, "ResponseSize_mean" : 0, "ResponseSize_25th_percentile" : 0, "ResponseSize_median" : 0, "ResponseSize_75th_percentile" : 0, "ResponseSize_90th_percentile" : 0, "ResponseSize_95th_percentile" : 0, "ResponseSize_98th_percentile" : 0, "ResponseSize_99th_percentile" : 0, "ResponseSize_99.9th_percentile" : 0, "ResponseSize_SizeRangeCount_100000000-inf" : 207, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 207, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 0, "ProcessCallTime_mean" : 0, "ProcessCallTime_25th_percentile" : 0, "ProcessCallTime_median" : 0, "ProcessCallTime_75th_percentile" : 0, "ProcessCallTime_90th_percentile" : 0, "ProcessCallTime_95th_percentile" : 0, "ProcessCallTime_98th_percentile" : 0, "ProcessCallTime_99th_percentile" : 0, "ProcessCallTime_99.9th_percentile" : 0, "ProcessCallTime_TimeRangeCount_600000-inf" : 207, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 9, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 207, "RequestSize_min" : 0, "RequestSize_max" : 0, "RequestSize_mean" : 0, "RequestSize_25th_percentile" : 0, "RequestSize_median" : 0, "RequestSize_75th_percentile" : 0, "RequestSize_90th_percentile" : 0, "RequestSize_95th_percentile" : 0, "RequestSize_98th_percentile" : 0, "RequestSize_99th_percentile" : 0, "RequestSize_99.9th_percentile" : 0, "RequestSize_SizeRangeCount_100000000-inf" : 207, "sentBytes" : 2283235, "QueueCallTime_num_ops" : 207, "QueueCallTime_min" : 0, "QueueCallTime_max" : 0, "QueueCallTime_mean" : 0, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 0, "QueueCallTime_90th_percentile" : 0, "QueueCallTime_95th_percentile" : 0, "QueueCallTime_98th_percentile" : 0, "QueueCallTime_99th_percentile" : 0, "QueueCallTime_99.9th_percentile" : 0, "QueueCallTime_TimeRangeCount_600000-inf" : 207, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:56841", "tag.serverName" : "ubuntu,39267,1517713227316", "tag.clusterId" : "990c040f-15f4-4bef-a30c-08f39bc273d8", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "regionCount" : 8, "storeCount" : 8, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 2, "memStoreSize" : 534358, "storeFileSize" : 66522, "maxStoreFileAge" : 294103, "minStoreFileAge" : 293655, "avgStoreFileAge" : 293879, "numReferenceFiles" : 0, "regionServerStartTime" : 1517713227316, "averageRegionSize" : 75110, "totalRequestCount" : 32, "totalRowActionRequestCount" : 34451, "readRequestCount" : 9844, "filteredReadRequestCount" : 0, "writeRequestCount" : 24607, "rpcGetRequestCount" : 4, "rpcScanRequestCount" : 19, "rpcMultiRequestCount" : 5, "rpcMutateRequestCount" : 2, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 656, "staticIndexSize" : 80, "staticBloomSize" : 2048, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1116962296, "blockCacheCount" : 0, "blockCacheSize" : 819720, "blockCacheHitCount" : 0, "blockCacheHitCountPrimary" : 0, "blockCacheMissCount" : 0, "blockCacheMissCountPrimary" : 0, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 0.0, "blockCacheExpressHitPercent" : 0.0, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 0, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 0, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 0, "l1CacheMissCount" : 0, "l1CacheHitRatio" : NaN, "l1CacheMissRatio" : NaN, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 10545, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 400710, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 19, "ScanTime_min" : 0, "ScanTime_max" : 0, "ScanTime_mean" : 0, "ScanTime_25th_percentile" : 0, "ScanTime_median" : 0, "ScanTime_75th_percentile" : 0, "ScanTime_90th_percentile" : 0, "ScanTime_95th_percentile" : 0, "ScanTime_98th_percentile" : 0, "ScanTime_99th_percentile" : 0, "ScanTime_99.9th_percentile" : 0, "ScanTime_TimeRangeCount_600000-inf" : 19, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 2, "Put_min" : 0, "Put_max" : 0, "Put_mean" : 0, "Put_25th_percentile" : 0, "Put_median" : 0, "Put_75th_percentile" : 0, "Put_90th_percentile" : 0, "Put_95th_percentile" : 0, "Put_98th_percentile" : 0, "Put_99th_percentile" : 0, "Put_99.9th_percentile" : 0, "Put_TimeRangeCount_600000-inf" : 2, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 2, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "FlushMemstoreSize_SizeRangeCount_100000000-inf" : 2, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 45, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "PutBatch_TimeRangeCount_600000-inf" : 45, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 4, "Get_min" : 0, "Get_max" : 0, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 0, "Get_75th_percentile" : 0, "Get_90th_percentile" : 0, "Get_95th_percentile" : 0, "Get_98th_percentile" : 0, "Get_99th_percentile" : 0, "Get_99.9th_percentile" : 0, "Get_TimeRangeCount_600000-inf" : 4, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 66522, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 2, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "FlushTime_TimeRangeCount_600000-inf" : 2, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 19, "ScanSize_min" : 0, "ScanSize_max" : 0, "ScanSize_mean" : 0, "ScanSize_25th_percentile" : 0, "ScanSize_median" : 0, "ScanSize_75th_percentile" : 0, "ScanSize_90th_percentile" : 0, "ScanSize_95th_percentile" : 0, "ScanSize_98th_percentile" : 0, "ScanSize_99th_percentile" : 0, "ScanSize_99.9th_percentile" : 0, "ScanSize_SizeRangeCount_100000000-inf" : 19, "slowGetCount" : 0, "flushedMemstoreBytes" : 267140, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 2, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "FlushOutputSize_SizeRangeCount_100000000-inf" : 2, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-02-04 11:05:40,178 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] ipc.AbstractRpcClient(349): Not trying to connect to ubuntu/127.0.1.1:39063 this server is in the failed servers list 2018-02-04 11:05:40,179 WARN [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegionServer(2371): Unable to report fatal error to master org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: java.io.IOException: Call to ubuntu/127.0.1.1:39063 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$0(AbstractRpcClient.java:324) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:571) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.reportRSFatalError(RegionServerStatusProtos.java:13568) at org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:2368) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.abortRegionServer(MiniHBaseCluster.java:208) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$1(MiniHBaseCluster.java:207) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$2.run(MiniHBaseCluster.java:201) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.abort(MiniHBaseCluster.java:198) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:115) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Call to ubuntu/127.0.1.1:39063 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$3(AbstractRpcClient.java:375) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:1) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:423) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:328) ... 17 more Caused by: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.getConnection(AbstractRpcClient.java:352) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:420) ... 18 more 2018-02-04 11:05:40,179 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-0] executor.EventHandler(186): Caught throwable while processing event M_RS_CLOSE_REGION java.lang.RuntimeException: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:117) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2369) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2354) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) ... 4 more 2018-02-04 11:05:40,179 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-0] helpers.MarkerIgnoringBase(159): ***** ABORTING region server ubuntu,39267,1517713227316: Caught throwable while processing event M_RS_CLOSE_REGION ***** java.lang.RuntimeException: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:117) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2369) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2354) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) ... 4 more 2018-02-04 11:05:40,180 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-0] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-02-04 11:05:40,183 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegionServer(2350): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 881852416, "init" : 197132288, "max" : 2794455040, "used" : 260973768 }, "NonHeapMemoryUsage" : { "committed" : 123625472, "init" : 2555904, "max" : -1, "used" : 121760144 }, "Verbose" : false, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : -1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 6227078, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 207, "TotalCallTime_min" : 0, "TotalCallTime_max" : 0, "TotalCallTime_mean" : 0, "TotalCallTime_25th_percentile" : 0, "TotalCallTime_median" : 0, "TotalCallTime_75th_percentile" : 0, "TotalCallTime_90th_percentile" : 0, "TotalCallTime_95th_percentile" : 0, "TotalCallTime_98th_percentile" : 0, "TotalCallTime_99th_percentile" : 0, "TotalCallTime_99.9th_percentile" : 0, "TotalCallTime_TimeRangeCount_600000-inf" : 207, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 207, "ResponseSize_min" : 0, "ResponseSize_max" : 0, "ResponseSize_mean" : 0, "ResponseSize_25th_percentile" : 0, "ResponseSize_median" : 0, "ResponseSize_75th_percentile" : 0, "ResponseSize_90th_percentile" : 0, "ResponseSize_95th_percentile" : 0, "ResponseSize_98th_percentile" : 0, "ResponseSize_99th_percentile" : 0, "ResponseSize_99.9th_percentile" : 0, "ResponseSize_SizeRangeCount_100000000-inf" : 207, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 207, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 0, "ProcessCallTime_mean" : 0, "ProcessCallTime_25th_percentile" : 0, "ProcessCallTime_median" : 0, "ProcessCallTime_75th_percentile" : 0, "ProcessCallTime_90th_percentile" : 0, "ProcessCallTime_95th_percentile" : 0, "ProcessCallTime_98th_percentile" : 0, "ProcessCallTime_99th_percentile" : 0, "ProcessCallTime_99.9th_percentile" : 0, "ProcessCallTime_TimeRangeCount_600000-inf" : 207, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 9, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 207, "RequestSize_min" : 0, "RequestSize_max" : 0, "RequestSize_mean" : 0, "RequestSize_25th_percentile" : 0, "RequestSize_median" : 0, "RequestSize_75th_percentile" : 0, "RequestSize_90th_percentile" : 0, "RequestSize_95th_percentile" : 0, "RequestSize_98th_percentile" : 0, "RequestSize_99th_percentile" : 0, "RequestSize_99.9th_percentile" : 0, "RequestSize_SizeRangeCount_100000000-inf" : 207, "sentBytes" : 2283235, "QueueCallTime_num_ops" : 207, "QueueCallTime_min" : 0, "QueueCallTime_max" : 0, "QueueCallTime_mean" : 0, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 0, "QueueCallTime_90th_percentile" : 0, "QueueCallTime_95th_percentile" : 0, "QueueCallTime_98th_percentile" : 0, "QueueCallTime_99th_percentile" : 0, "QueueCallTime_99.9th_percentile" : 0, "QueueCallTime_TimeRangeCount_600000-inf" : 207, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:56841", "tag.serverName" : "ubuntu,39267,1517713227316", "tag.clusterId" : "990c040f-15f4-4bef-a30c-08f39bc273d8", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "regionCount" : 8, "storeCount" : 8, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 2, "memStoreSize" : 534358, "storeFileSize" : 66522, "maxStoreFileAge" : 294103, "minStoreFileAge" : 293655, "avgStoreFileAge" : 293879, "numReferenceFiles" : 0, "regionServerStartTime" : 1517713227316, "averageRegionSize" : 75110, "totalRequestCount" : 32, "totalRowActionRequestCount" : 34451, "readRequestCount" : 9844, "filteredReadRequestCount" : 0, "writeRequestCount" : 24607, "rpcGetRequestCount" : 4, "rpcScanRequestCount" : 19, "rpcMultiRequestCount" : 5, "rpcMutateRequestCount" : 2, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 656, "staticIndexSize" : 80, "staticBloomSize" : 2048, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1116962296, "blockCacheCount" : 0, "blockCacheSize" : 819720, "blockCacheHitCount" : 0, "blockCacheHitCountPrimary" : 0, "blockCacheMissCount" : 0, "blockCacheMissCountPrimary" : 0, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 0.0, "blockCacheExpressHitPercent" : 0.0, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 0, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 0, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 0, "l1CacheMissCount" : 0, "l1CacheHitRatio" : NaN, "l1CacheMissRatio" : NaN, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 10545, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 400710, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 19, "ScanTime_min" : 0, "ScanTime_max" : 0, "ScanTime_mean" : 0, "ScanTime_25th_percentile" : 0, "ScanTime_median" : 0, "ScanTime_75th_percentile" : 0, "ScanTime_90th_percentile" : 0, "ScanTime_95th_percentile" : 0, "ScanTime_98th_percentile" : 0, "ScanTime_99th_percentile" : 0, "ScanTime_99.9th_percentile" : 0, "ScanTime_TimeRangeCount_600000-inf" : 19, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 2, "Put_min" : 0, "Put_max" : 0, "Put_mean" : 0, "Put_25th_percentile" : 0, "Put_median" : 0, "Put_75th_percentile" : 0, "Put_90th_percentile" : 0, "Put_95th_percentile" : 0, "Put_98th_percentile" : 0, "Put_99th_percentile" : 0, "Put_99.9th_percentile" : 0, "Put_TimeRangeCount_600000-inf" : 2, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 2, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "FlushMemstoreSize_SizeRangeCount_100000000-inf" : 2, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 45, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "PutBatch_TimeRangeCount_600000-inf" : 45, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 4, "Get_min" : 0, "Get_max" : 0, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 0, "Get_75th_percentile" : 0, "Get_90th_percentile" : 0, "Get_95th_percentile" : 0, "Get_98th_percentile" : 0, "Get_99th_percentile" : 0, "Get_99.9th_percentile" : 0, "Get_TimeRangeCount_600000-inf" : 4, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 66522, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 2, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "FlushTime_TimeRangeCount_600000-inf" : 2, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 19, "ScanSize_min" : 0, "ScanSize_max" : 0, "ScanSize_mean" : 0, "ScanSize_25th_percentile" : 0, "ScanSize_median" : 0, "ScanSize_75th_percentile" : 0, "ScanSize_90th_percentile" : 0, "ScanSize_95th_percentile" : 0, "ScanSize_98th_percentile" : 0, "ScanSize_99th_percentile" : 0, "ScanSize_99.9th_percentile" : 0, "ScanSize_SizeRangeCount_100000000-inf" : 19, "slowGetCount" : 0, "flushedMemstoreBytes" : 267140, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 2, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "FlushOutputSize_SizeRangeCount_100000000-inf" : 2, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-02-04 11:05:40,194 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] ipc.AbstractRpcClient(349): Not trying to connect to ubuntu/127.0.1.1:39063 this server is in the failed servers list 2018-02-04 11:05:40,195 WARN [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegionServer(2371): Unable to report fatal error to master org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: java.io.IOException: Call to ubuntu/127.0.1.1:39063 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$0(AbstractRpcClient.java:324) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:571) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.reportRSFatalError(RegionServerStatusProtos.java:13568) at org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:2368) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.abortRegionServer(MiniHBaseCluster.java:208) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$1(MiniHBaseCluster.java:207) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$2.run(MiniHBaseCluster.java:201) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.abort(MiniHBaseCluster.java:198) at org.apache.hadoop.hbase.executor.EventHandler.handleException(EventHandler.java:188) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:106) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Call to ubuntu/127.0.1.1:39063 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$3(AbstractRpcClient.java:375) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:1) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:423) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:328) ... 17 more Caused by: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.getConnection(AbstractRpcClient.java:352) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:420) ... 18 more 2018-02-04 11:05:40,197 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] handler.CloseRegionHandler(92): Processing close of tabletest,eee,1517713232695.14343105008c3cdd39a78990296fbe94. 2018-02-04 11:05:40,198 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(1483): Closing tabletest,eee,1517713232695.14343105008c3cdd39a78990296fbe94.: disabling compactions & flushes 2018-02-04 11:05:40,198 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(1522): Updates disabled for region tabletest,eee,1517713232695.14343105008c3cdd39a78990296fbe94. 2018-02-04 11:05:40,198 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-0] helpers.MarkerIgnoringBase(159): ***** ABORTING region server ubuntu,39267,1517713227316: Unrecoverable exception while closing region tabletest,eee,1517713232695.14343105008c3cdd39a78990296fbe94., still finishing close ***** java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2369) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2354) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-02-04 11:05:40,199 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-0] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-02-04 11:05:40,201 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegionServer(2350): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 881852416, "init" : 197132288, "max" : 2794455040, "used" : 261974752 }, "NonHeapMemoryUsage" : { "committed" : 123625472, "init" : 2555904, "max" : -1, "used" : 121766072 }, "Verbose" : false, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : -1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 6227078, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 207, "TotalCallTime_min" : 0, "TotalCallTime_max" : 0, "TotalCallTime_mean" : 0, "TotalCallTime_25th_percentile" : 0, "TotalCallTime_median" : 0, "TotalCallTime_75th_percentile" : 0, "TotalCallTime_90th_percentile" : 0, "TotalCallTime_95th_percentile" : 0, "TotalCallTime_98th_percentile" : 0, "TotalCallTime_99th_percentile" : 0, "TotalCallTime_99.9th_percentile" : 0, "TotalCallTime_TimeRangeCount_600000-inf" : 207, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 207, "ResponseSize_min" : 0, "ResponseSize_max" : 0, "ResponseSize_mean" : 0, "ResponseSize_25th_percentile" : 0, "ResponseSize_median" : 0, "ResponseSize_75th_percentile" : 0, "ResponseSize_90th_percentile" : 0, "ResponseSize_95th_percentile" : 0, "ResponseSize_98th_percentile" : 0, "ResponseSize_99th_percentile" : 0, "ResponseSize_99.9th_percentile" : 0, "ResponseSize_SizeRangeCount_100000000-inf" : 207, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 207, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 0, "ProcessCallTime_mean" : 0, "ProcessCallTime_25th_percentile" : 0, "ProcessCallTime_median" : 0, "ProcessCallTime_75th_percentile" : 0, "ProcessCallTime_90th_percentile" : 0, "ProcessCallTime_95th_percentile" : 0, "ProcessCallTime_98th_percentile" : 0, "ProcessCallTime_99th_percentile" : 0, "ProcessCallTime_99.9th_percentile" : 0, "ProcessCallTime_TimeRangeCount_600000-inf" : 207, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 9, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 207, "RequestSize_min" : 0, "RequestSize_max" : 0, "RequestSize_mean" : 0, "RequestSize_25th_percentile" : 0, "RequestSize_median" : 0, "RequestSize_75th_percentile" : 0, "RequestSize_90th_percentile" : 0, "RequestSize_95th_percentile" : 0, "RequestSize_98th_percentile" : 0, "RequestSize_99th_percentile" : 0, "RequestSize_99.9th_percentile" : 0, "RequestSize_SizeRangeCount_100000000-inf" : 207, "sentBytes" : 2283235, "QueueCallTime_num_ops" : 207, "QueueCallTime_min" : 0, "QueueCallTime_max" : 0, "QueueCallTime_mean" : 0, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 0, "QueueCallTime_90th_percentile" : 0, "QueueCallTime_95th_percentile" : 0, "QueueCallTime_98th_percentile" : 0, "QueueCallTime_99th_percentile" : 0, "QueueCallTime_99.9th_percentile" : 0, "QueueCallTime_TimeRangeCount_600000-inf" : 207, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:56841", "tag.serverName" : "ubuntu,39267,1517713227316", "tag.clusterId" : "990c040f-15f4-4bef-a30c-08f39bc273d8", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "regionCount" : 8, "storeCount" : 8, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 2, "memStoreSize" : 534358, "storeFileSize" : 66522, "maxStoreFileAge" : 294103, "minStoreFileAge" : 293655, "avgStoreFileAge" : 293879, "numReferenceFiles" : 0, "regionServerStartTime" : 1517713227316, "averageRegionSize" : 75110, "totalRequestCount" : 32, "totalRowActionRequestCount" : 34451, "readRequestCount" : 9844, "filteredReadRequestCount" : 0, "writeRequestCount" : 24607, "rpcGetRequestCount" : 4, "rpcScanRequestCount" : 19, "rpcMultiRequestCount" : 5, "rpcMutateRequestCount" : 2, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 656, "staticIndexSize" : 80, "staticBloomSize" : 2048, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1116962296, "blockCacheCount" : 0, "blockCacheSize" : 819720, "blockCacheHitCount" : 0, "blockCacheHitCountPrimary" : 0, "blockCacheMissCount" : 0, "blockCacheMissCountPrimary" : 0, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 0.0, "blockCacheExpressHitPercent" : 0.0, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 0, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 0, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 0, "l1CacheMissCount" : 0, "l1CacheHitRatio" : NaN, "l1CacheMissRatio" : NaN, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 10545, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 400710, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 19, "ScanTime_min" : 0, "ScanTime_max" : 0, "ScanTime_mean" : 0, "ScanTime_25th_percentile" : 0, "ScanTime_median" : 0, "ScanTime_75th_percentile" : 0, "ScanTime_90th_percentile" : 0, "ScanTime_95th_percentile" : 0, "ScanTime_98th_percentile" : 0, "ScanTime_99th_percentile" : 0, "ScanTime_99.9th_percentile" : 0, "ScanTime_TimeRangeCount_600000-inf" : 19, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 2, "Put_min" : 0, "Put_max" : 0, "Put_mean" : 0, "Put_25th_percentile" : 0, "Put_median" : 0, "Put_75th_percentile" : 0, "Put_90th_percentile" : 0, "Put_95th_percentile" : 0, "Put_98th_percentile" : 0, "Put_99th_percentile" : 0, "Put_99.9th_percentile" : 0, "Put_TimeRangeCount_600000-inf" : 2, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 2, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "FlushMemstoreSize_SizeRangeCount_100000000-inf" : 2, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 45, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "PutBatch_TimeRangeCount_600000-inf" : 45, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 4, "Get_min" : 0, "Get_max" : 0, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 0, "Get_75th_percentile" : 0, "Get_90th_percentile" : 0, "Get_95th_percentile" : 0, "Get_98th_percentile" : 0, "Get_99th_percentile" : 0, "Get_99.9th_percentile" : 0, "Get_TimeRangeCount_600000-inf" : 4, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 66522, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 2, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "FlushTime_TimeRangeCount_600000-inf" : 2, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 19, "ScanSize_min" : 0, "ScanSize_max" : 0, "ScanSize_mean" : 0, "ScanSize_25th_percentile" : 0, "ScanSize_median" : 0, "ScanSize_75th_percentile" : 0, "ScanSize_90th_percentile" : 0, "ScanSize_95th_percentile" : 0, "ScanSize_98th_percentile" : 0, "ScanSize_99th_percentile" : 0, "ScanSize_99.9th_percentile" : 0, "ScanSize_SizeRangeCount_100000000-inf" : 19, "slowGetCount" : 0, "flushedMemstoreBytes" : 267140, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 2, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "FlushOutputSize_SizeRangeCount_100000000-inf" : 2, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-02-04 11:05:40,201 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] ipc.AbstractRpcClient(349): Not trying to connect to ubuntu/127.0.1.1:39063 this server is in the failed servers list 2018-02-04 11:05:40,202 WARN [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegionServer(2371): Unable to report fatal error to master org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: java.io.IOException: Call to ubuntu/127.0.1.1:39063 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$0(AbstractRpcClient.java:324) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:571) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.reportRSFatalError(RegionServerStatusProtos.java:13568) at org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:2368) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.abortRegionServer(MiniHBaseCluster.java:208) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$1(MiniHBaseCluster.java:207) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$2.run(MiniHBaseCluster.java:201) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.abort(MiniHBaseCluster.java:198) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:115) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Call to ubuntu/127.0.1.1:39063 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$3(AbstractRpcClient.java:375) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:1) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:423) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:328) ... 17 more Caused by: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.getConnection(AbstractRpcClient.java:352) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:420) ... 18 more 2018-02-04 11:05:40,203 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-0] executor.EventHandler(186): Caught throwable while processing event M_RS_CLOSE_REGION java.lang.RuntimeException: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:117) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2369) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2354) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) ... 4 more 2018-02-04 11:05:40,203 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-0] helpers.MarkerIgnoringBase(159): ***** ABORTING region server ubuntu,39267,1517713227316: Caught throwable while processing event M_RS_CLOSE_REGION ***** java.lang.RuntimeException: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:117) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2369) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2354) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) ... 4 more 2018-02-04 11:05:40,204 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-0] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-02-04 11:05:40,206 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegionServer(2350): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 881852416, "init" : 197132288, "max" : 2794455040, "used" : 263024216 }, "NonHeapMemoryUsage" : { "committed" : 123625472, "init" : 2555904, "max" : -1, "used" : 121768816 }, "Verbose" : false, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : -1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 6227078, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 207, "TotalCallTime_min" : 0, "TotalCallTime_max" : 0, "TotalCallTime_mean" : 0, "TotalCallTime_25th_percentile" : 0, "TotalCallTime_median" : 0, "TotalCallTime_75th_percentile" : 0, "TotalCallTime_90th_percentile" : 0, "TotalCallTime_95th_percentile" : 0, "TotalCallTime_98th_percentile" : 0, "TotalCallTime_99th_percentile" : 0, "TotalCallTime_99.9th_percentile" : 0, "TotalCallTime_TimeRangeCount_600000-inf" : 207, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 207, "ResponseSize_min" : 0, "ResponseSize_max" : 0, "ResponseSize_mean" : 0, "ResponseSize_25th_percentile" : 0, "ResponseSize_median" : 0, "ResponseSize_75th_percentile" : 0, "ResponseSize_90th_percentile" : 0, "ResponseSize_95th_percentile" : 0, "ResponseSize_98th_percentile" : 0, "ResponseSize_99th_percentile" : 0, "ResponseSize_99.9th_percentile" : 0, "ResponseSize_SizeRangeCount_100000000-inf" : 207, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 207, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 0, "ProcessCallTime_mean" : 0, "ProcessCallTime_25th_percentile" : 0, "ProcessCallTime_median" : 0, "ProcessCallTime_75th_percentile" : 0, "ProcessCallTime_90th_percentile" : 0, "ProcessCallTime_95th_percentile" : 0, "ProcessCallTime_98th_percentile" : 0, "ProcessCallTime_99th_percentile" : 0, "ProcessCallTime_99.9th_percentile" : 0, "ProcessCallTime_TimeRangeCount_600000-inf" : 207, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 9, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 207, "RequestSize_min" : 0, "RequestSize_max" : 0, "RequestSize_mean" : 0, "RequestSize_25th_percentile" : 0, "RequestSize_median" : 0, "RequestSize_75th_percentile" : 0, "RequestSize_90th_percentile" : 0, "RequestSize_95th_percentile" : 0, "RequestSize_98th_percentile" : 0, "RequestSize_99th_percentile" : 0, "RequestSize_99.9th_percentile" : 0, "RequestSize_SizeRangeCount_100000000-inf" : 207, "sentBytes" : 2283235, "QueueCallTime_num_ops" : 207, "QueueCallTime_min" : 0, "QueueCallTime_max" : 0, "QueueCallTime_mean" : 0, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 0, "QueueCallTime_90th_percentile" : 0, "QueueCallTime_95th_percentile" : 0, "QueueCallTime_98th_percentile" : 0, "QueueCallTime_99th_percentile" : 0, "QueueCallTime_99.9th_percentile" : 0, "QueueCallTime_TimeRangeCount_600000-inf" : 207, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:56841", "tag.serverName" : "ubuntu,39267,1517713227316", "tag.clusterId" : "990c040f-15f4-4bef-a30c-08f39bc273d8", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "regionCount" : 8, "storeCount" : 8, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 2, "memStoreSize" : 534358, "storeFileSize" : 66522, "maxStoreFileAge" : 294103, "minStoreFileAge" : 293655, "avgStoreFileAge" : 293879, "numReferenceFiles" : 0, "regionServerStartTime" : 1517713227316, "averageRegionSize" : 75110, "totalRequestCount" : 32, "totalRowActionRequestCount" : 34451, "readRequestCount" : 9844, "filteredReadRequestCount" : 0, "writeRequestCount" : 24607, "rpcGetRequestCount" : 4, "rpcScanRequestCount" : 19, "rpcMultiRequestCount" : 5, "rpcMutateRequestCount" : 2, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 656, "staticIndexSize" : 80, "staticBloomSize" : 2048, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1116962296, "blockCacheCount" : 0, "blockCacheSize" : 819720, "blockCacheHitCount" : 0, "blockCacheHitCountPrimary" : 0, "blockCacheMissCount" : 0, "blockCacheMissCountPrimary" : 0, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 0.0, "blockCacheExpressHitPercent" : 0.0, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 0, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 0, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 0, "l1CacheMissCount" : 0, "l1CacheHitRatio" : NaN, "l1CacheMissRatio" : NaN, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 10545, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 400710, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 19, "ScanTime_min" : 0, "ScanTime_max" : 0, "ScanTime_mean" : 0, "ScanTime_25th_percentile" : 0, "ScanTime_median" : 0, "ScanTime_75th_percentile" : 0, "ScanTime_90th_percentile" : 0, "ScanTime_95th_percentile" : 0, "ScanTime_98th_percentile" : 0, "ScanTime_99th_percentile" : 0, "ScanTime_99.9th_percentile" : 0, "ScanTime_TimeRangeCount_600000-inf" : 19, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 2, "Put_min" : 0, "Put_max" : 0, "Put_mean" : 0, "Put_25th_percentile" : 0, "Put_median" : 0, "Put_75th_percentile" : 0, "Put_90th_percentile" : 0, "Put_95th_percentile" : 0, "Put_98th_percentile" : 0, "Put_99th_percentile" : 0, "Put_99.9th_percentile" : 0, "Put_TimeRangeCount_600000-inf" : 2, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 2, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "FlushMemstoreSize_SizeRangeCount_100000000-inf" : 2, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 45, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "PutBatch_TimeRangeCount_600000-inf" : 45, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 4, "Get_min" : 0, "Get_max" : 0, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 0, "Get_75th_percentile" : 0, "Get_90th_percentile" : 0, "Get_95th_percentile" : 0, "Get_98th_percentile" : 0, "Get_99th_percentile" : 0, "Get_99.9th_percentile" : 0, "Get_TimeRangeCount_600000-inf" : 4, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 66522, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 2, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "FlushTime_TimeRangeCount_600000-inf" : 2, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 19, "ScanSize_min" : 0, "ScanSize_max" : 0, "ScanSize_mean" : 0, "ScanSize_25th_percentile" : 0, "ScanSize_median" : 0, "ScanSize_75th_percentile" : 0, "ScanSize_90th_percentile" : 0, "ScanSize_95th_percentile" : 0, "ScanSize_98th_percentile" : 0, "ScanSize_99th_percentile" : 0, "ScanSize_99.9th_percentile" : 0, "ScanSize_SizeRangeCount_100000000-inf" : 19, "slowGetCount" : 0, "flushedMemstoreBytes" : 267140, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 2, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "FlushOutputSize_SizeRangeCount_100000000-inf" : 2, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-02-04 11:05:40,207 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] ipc.AbstractRpcClient(349): Not trying to connect to ubuntu/127.0.1.1:39063 this server is in the failed servers list 2018-02-04 11:05:40,208 WARN [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegionServer(2371): Unable to report fatal error to master org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: java.io.IOException: Call to ubuntu/127.0.1.1:39063 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$0(AbstractRpcClient.java:324) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:571) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.reportRSFatalError(RegionServerStatusProtos.java:13568) at org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:2368) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.abortRegionServer(MiniHBaseCluster.java:208) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$1(MiniHBaseCluster.java:207) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$2.run(MiniHBaseCluster.java:201) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.abort(MiniHBaseCluster.java:198) at org.apache.hadoop.hbase.executor.EventHandler.handleException(EventHandler.java:188) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:106) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Call to ubuntu/127.0.1.1:39063 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$3(AbstractRpcClient.java:375) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:1) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:423) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:328) ... 17 more Caused by: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.getConnection(AbstractRpcClient.java:352) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:420) ... 18 more 2018-02-04 11:05:40,209 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] handler.CloseRegionHandler(92): Processing close of tabletest,fff,1517713232695.7b96ff3277efd9751a2175b9c2b4d481. 2018-02-04 11:05:40,218 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(1483): Closing tabletest,fff,1517713232695.7b96ff3277efd9751a2175b9c2b4d481.: disabling compactions & flushes 2018-02-04 11:05:40,218 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(1522): Updates disabled for region tabletest,fff,1517713232695.7b96ff3277efd9751a2175b9c2b4d481. 2018-02-04 11:05:40,218 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-0] helpers.MarkerIgnoringBase(159): ***** ABORTING region server ubuntu,39267,1517713227316: Unrecoverable exception while closing region tabletest,fff,1517713232695.7b96ff3277efd9751a2175b9c2b4d481., still finishing close ***** java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2369) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2354) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-02-04 11:05:40,218 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-0] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-02-04 11:05:40,220 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegionServer(2350): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 881852416, "init" : 197132288, "max" : 2794455040, "used" : 264027888 }, "NonHeapMemoryUsage" : { "committed" : 123691008, "init" : 2555904, "max" : -1, "used" : 121809640 }, "Verbose" : false, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : -1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 6227078, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 207, "TotalCallTime_min" : 0, "TotalCallTime_max" : 0, "TotalCallTime_mean" : 0, "TotalCallTime_25th_percentile" : 0, "TotalCallTime_median" : 0, "TotalCallTime_75th_percentile" : 0, "TotalCallTime_90th_percentile" : 0, "TotalCallTime_95th_percentile" : 0, "TotalCallTime_98th_percentile" : 0, "TotalCallTime_99th_percentile" : 0, "TotalCallTime_99.9th_percentile" : 0, "TotalCallTime_TimeRangeCount_600000-inf" : 207, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 207, "ResponseSize_min" : 0, "ResponseSize_max" : 0, "ResponseSize_mean" : 0, "ResponseSize_25th_percentile" : 0, "ResponseSize_median" : 0, "ResponseSize_75th_percentile" : 0, "ResponseSize_90th_percentile" : 0, "ResponseSize_95th_percentile" : 0, "ResponseSize_98th_percentile" : 0, "ResponseSize_99th_percentile" : 0, "ResponseSize_99.9th_percentile" : 0, "ResponseSize_SizeRangeCount_100000000-inf" : 207, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 207, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 0, "ProcessCallTime_mean" : 0, "ProcessCallTime_25th_percentile" : 0, "ProcessCallTime_median" : 0, "ProcessCallTime_75th_percentile" : 0, "ProcessCallTime_90th_percentile" : 0, "ProcessCallTime_95th_percentile" : 0, "ProcessCallTime_98th_percentile" : 0, "ProcessCallTime_99th_percentile" : 0, "ProcessCallTime_99.9th_percentile" : 0, "ProcessCallTime_TimeRangeCount_600000-inf" : 207, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 9, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 207, "RequestSize_min" : 0, "RequestSize_max" : 0, "RequestSize_mean" : 0, "RequestSize_25th_percentile" : 0, "RequestSize_median" : 0, "RequestSize_75th_percentile" : 0, "RequestSize_90th_percentile" : 0, "RequestSize_95th_percentile" : 0, "RequestSize_98th_percentile" : 0, "RequestSize_99th_percentile" : 0, "RequestSize_99.9th_percentile" : 0, "RequestSize_SizeRangeCount_100000000-inf" : 207, "sentBytes" : 2283235, "QueueCallTime_num_ops" : 207, "QueueCallTime_min" : 0, "QueueCallTime_max" : 0, "QueueCallTime_mean" : 0, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 0, "QueueCallTime_90th_percentile" : 0, "QueueCallTime_95th_percentile" : 0, "QueueCallTime_98th_percentile" : 0, "QueueCallTime_99th_percentile" : 0, "QueueCallTime_99.9th_percentile" : 0, "QueueCallTime_TimeRangeCount_600000-inf" : 207, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:56841", "tag.serverName" : "ubuntu,39267,1517713227316", "tag.clusterId" : "990c040f-15f4-4bef-a30c-08f39bc273d8", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "regionCount" : 8, "storeCount" : 8, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 2, "memStoreSize" : 534358, "storeFileSize" : 66522, "maxStoreFileAge" : 294103, "minStoreFileAge" : 293655, "avgStoreFileAge" : 293879, "numReferenceFiles" : 0, "regionServerStartTime" : 1517713227316, "averageRegionSize" : 75110, "totalRequestCount" : 32, "totalRowActionRequestCount" : 34451, "readRequestCount" : 9844, "filteredReadRequestCount" : 0, "writeRequestCount" : 24607, "rpcGetRequestCount" : 4, "rpcScanRequestCount" : 19, "rpcMultiRequestCount" : 5, "rpcMutateRequestCount" : 2, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 656, "staticIndexSize" : 80, "staticBloomSize" : 2048, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1116962296, "blockCacheCount" : 0, "blockCacheSize" : 819720, "blockCacheHitCount" : 0, "blockCacheHitCountPrimary" : 0, "blockCacheMissCount" : 0, "blockCacheMissCountPrimary" : 0, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 0.0, "blockCacheExpressHitPercent" : 0.0, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 0, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 0, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 0, "l1CacheMissCount" : 0, "l1CacheHitRatio" : NaN, "l1CacheMissRatio" : NaN, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 10545, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 400710, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 19, "ScanTime_min" : 0, "ScanTime_max" : 0, "ScanTime_mean" : 0, "ScanTime_25th_percentile" : 0, "ScanTime_median" : 0, "ScanTime_75th_percentile" : 0, "ScanTime_90th_percentile" : 0, "ScanTime_95th_percentile" : 0, "ScanTime_98th_percentile" : 0, "ScanTime_99th_percentile" : 0, "ScanTime_99.9th_percentile" : 0, "ScanTime_TimeRangeCount_600000-inf" : 19, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 2, "Put_min" : 0, "Put_max" : 0, "Put_mean" : 0, "Put_25th_percentile" : 0, "Put_median" : 0, "Put_75th_percentile" : 0, "Put_90th_percentile" : 0, "Put_95th_percentile" : 0, "Put_98th_percentile" : 0, "Put_99th_percentile" : 0, "Put_99.9th_percentile" : 0, "Put_TimeRangeCount_600000-inf" : 2, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 2, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "FlushMemstoreSize_SizeRangeCount_100000000-inf" : 2, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 45, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "PutBatch_TimeRangeCount_600000-inf" : 45, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 4, "Get_min" : 0, "Get_max" : 0, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 0, "Get_75th_percentile" : 0, "Get_90th_percentile" : 0, "Get_95th_percentile" : 0, "Get_98th_percentile" : 0, "Get_99th_percentile" : 0, "Get_99.9th_percentile" : 0, "Get_TimeRangeCount_600000-inf" : 4, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 66522, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 2, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "FlushTime_TimeRangeCount_600000-inf" : 2, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 19, "ScanSize_min" : 0, "ScanSize_max" : 0, "ScanSize_mean" : 0, "ScanSize_25th_percentile" : 0, "ScanSize_median" : 0, "ScanSize_75th_percentile" : 0, "ScanSize_90th_percentile" : 0, "ScanSize_95th_percentile" : 0, "ScanSize_98th_percentile" : 0, "ScanSize_99th_percentile" : 0, "ScanSize_99.9th_percentile" : 0, "ScanSize_SizeRangeCount_100000000-inf" : 19, "slowGetCount" : 0, "flushedMemstoreBytes" : 267140, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 2, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "FlushOutputSize_SizeRangeCount_100000000-inf" : 2, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-02-04 11:05:40,220 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] ipc.AbstractRpcClient(349): Not trying to connect to ubuntu/127.0.1.1:39063 this server is in the failed servers list 2018-02-04 11:05:40,220 WARN [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegionServer(2371): Unable to report fatal error to master org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: java.io.IOException: Call to ubuntu/127.0.1.1:39063 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$0(AbstractRpcClient.java:324) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:571) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.reportRSFatalError(RegionServerStatusProtos.java:13568) at org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:2368) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.abortRegionServer(MiniHBaseCluster.java:208) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$1(MiniHBaseCluster.java:207) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$2.run(MiniHBaseCluster.java:201) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.abort(MiniHBaseCluster.java:198) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:115) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Call to ubuntu/127.0.1.1:39063 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$3(AbstractRpcClient.java:375) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:1) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:423) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:328) ... 17 more Caused by: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.getConnection(AbstractRpcClient.java:352) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:420) ... 18 more 2018-02-04 11:05:40,220 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-0] executor.EventHandler(186): Caught throwable while processing event M_RS_CLOSE_REGION java.lang.RuntimeException: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:117) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2369) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2354) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) ... 4 more 2018-02-04 11:05:40,220 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-0] helpers.MarkerIgnoringBase(159): ***** ABORTING region server ubuntu,39267,1517713227316: Caught throwable while processing event M_RS_CLOSE_REGION ***** java.lang.RuntimeException: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:117) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2369) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2354) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) ... 4 more 2018-02-04 11:05:40,221 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-0] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-02-04 11:05:40,221 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegionServer(2350): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 881852416, "init" : 197132288, "max" : 2794455040, "used" : 265079432 }, "NonHeapMemoryUsage" : { "committed" : 123691008, "init" : 2555904, "max" : -1, "used" : 121809640 }, "Verbose" : false, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : -1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 6227078, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 207, "TotalCallTime_min" : 0, "TotalCallTime_max" : 0, "TotalCallTime_mean" : 0, "TotalCallTime_25th_percentile" : 0, "TotalCallTime_median" : 0, "TotalCallTime_75th_percentile" : 0, "TotalCallTime_90th_percentile" : 0, "TotalCallTime_95th_percentile" : 0, "TotalCallTime_98th_percentile" : 0, "TotalCallTime_99th_percentile" : 0, "TotalCallTime_99.9th_percentile" : 0, "TotalCallTime_TimeRangeCount_600000-inf" : 207, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 207, "ResponseSize_min" : 0, "ResponseSize_max" : 0, "ResponseSize_mean" : 0, "ResponseSize_25th_percentile" : 0, "ResponseSize_median" : 0, "ResponseSize_75th_percentile" : 0, "ResponseSize_90th_percentile" : 0, "ResponseSize_95th_percentile" : 0, "ResponseSize_98th_percentile" : 0, "ResponseSize_99th_percentile" : 0, "ResponseSize_99.9th_percentile" : 0, "ResponseSize_SizeRangeCount_100000000-inf" : 207, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 207, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 0, "ProcessCallTime_mean" : 0, "ProcessCallTime_25th_percentile" : 0, "ProcessCallTime_median" : 0, "ProcessCallTime_75th_percentile" : 0, "ProcessCallTime_90th_percentile" : 0, "ProcessCallTime_95th_percentile" : 0, "ProcessCallTime_98th_percentile" : 0, "ProcessCallTime_99th_percentile" : 0, "ProcessCallTime_99.9th_percentile" : 0, "ProcessCallTime_TimeRangeCount_600000-inf" : 207, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 9, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 207, "RequestSize_min" : 0, "RequestSize_max" : 0, "RequestSize_mean" : 0, "RequestSize_25th_percentile" : 0, "RequestSize_median" : 0, "RequestSize_75th_percentile" : 0, "RequestSize_90th_percentile" : 0, "RequestSize_95th_percentile" : 0, "RequestSize_98th_percentile" : 0, "RequestSize_99th_percentile" : 0, "RequestSize_99.9th_percentile" : 0, "RequestSize_SizeRangeCount_100000000-inf" : 207, "sentBytes" : 2283235, "QueueCallTime_num_ops" : 207, "QueueCallTime_min" : 0, "QueueCallTime_max" : 0, "QueueCallTime_mean" : 0, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 0, "QueueCallTime_90th_percentile" : 0, "QueueCallTime_95th_percentile" : 0, "QueueCallTime_98th_percentile" : 0, "QueueCallTime_99th_percentile" : 0, "QueueCallTime_99.9th_percentile" : 0, "QueueCallTime_TimeRangeCount_600000-inf" : 207, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:56841", "tag.serverName" : "ubuntu,39267,1517713227316", "tag.clusterId" : "990c040f-15f4-4bef-a30c-08f39bc273d8", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "regionCount" : 8, "storeCount" : 8, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 2, "memStoreSize" : 534358, "storeFileSize" : 66522, "maxStoreFileAge" : 294103, "minStoreFileAge" : 293655, "avgStoreFileAge" : 293879, "numReferenceFiles" : 0, "regionServerStartTime" : 1517713227316, "averageRegionSize" : 75110, "totalRequestCount" : 32, "totalRowActionRequestCount" : 34451, "readRequestCount" : 9844, "filteredReadRequestCount" : 0, "writeRequestCount" : 24607, "rpcGetRequestCount" : 4, "rpcScanRequestCount" : 19, "rpcMultiRequestCount" : 5, "rpcMutateRequestCount" : 2, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 656, "staticIndexSize" : 80, "staticBloomSize" : 2048, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1116962296, "blockCacheCount" : 0, "blockCacheSize" : 819720, "blockCacheHitCount" : 0, "blockCacheHitCountPrimary" : 0, "blockCacheMissCount" : 0, "blockCacheMissCountPrimary" : 0, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 0.0, "blockCacheExpressHitPercent" : 0.0, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 0, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 0, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 0, "l1CacheMissCount" : 0, "l1CacheHitRatio" : NaN, "l1CacheMissRatio" : NaN, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 10545, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 400710, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 19, "ScanTime_min" : 0, "ScanTime_max" : 0, "ScanTime_mean" : 0, "ScanTime_25th_percentile" : 0, "ScanTime_median" : 0, "ScanTime_75th_percentile" : 0, "ScanTime_90th_percentile" : 0, "ScanTime_95th_percentile" : 0, "ScanTime_98th_percentile" : 0, "ScanTime_99th_percentile" : 0, "ScanTime_99.9th_percentile" : 0, "ScanTime_TimeRangeCount_600000-inf" : 19, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 2, "Put_min" : 0, "Put_max" : 0, "Put_mean" : 0, "Put_25th_percentile" : 0, "Put_median" : 0, "Put_75th_percentile" : 0, "Put_90th_percentile" : 0, "Put_95th_percentile" : 0, "Put_98th_percentile" : 0, "Put_99th_percentile" : 0, "Put_99.9th_percentile" : 0, "Put_TimeRangeCount_600000-inf" : 2, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 2, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "FlushMemstoreSize_SizeRangeCount_100000000-inf" : 2, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 45, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "PutBatch_TimeRangeCount_600000-inf" : 45, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 4, "Get_min" : 0, "Get_max" : 0, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 0, "Get_75th_percentile" : 0, "Get_90th_percentile" : 0, "Get_95th_percentile" : 0, "Get_98th_percentile" : 0, "Get_99th_percentile" : 0, "Get_99.9th_percentile" : 0, "Get_TimeRangeCount_600000-inf" : 4, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 66522, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 2, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "FlushTime_TimeRangeCount_600000-inf" : 2, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 19, "ScanSize_min" : 0, "ScanSize_max" : 0, "ScanSize_mean" : 0, "ScanSize_25th_percentile" : 0, "ScanSize_median" : 0, "ScanSize_75th_percentile" : 0, "ScanSize_90th_percentile" : 0, "ScanSize_95th_percentile" : 0, "ScanSize_98th_percentile" : 0, "ScanSize_99th_percentile" : 0, "ScanSize_99.9th_percentile" : 0, "ScanSize_SizeRangeCount_100000000-inf" : 19, "slowGetCount" : 0, "flushedMemstoreBytes" : 267140, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 2, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "FlushOutputSize_SizeRangeCount_100000000-inf" : 2, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-02-04 11:05:40,222 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] ipc.AbstractRpcClient(349): Not trying to connect to ubuntu/127.0.1.1:39063 this server is in the failed servers list 2018-02-04 11:05:40,222 WARN [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegionServer(2371): Unable to report fatal error to master org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: java.io.IOException: Call to ubuntu/127.0.1.1:39063 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$0(AbstractRpcClient.java:324) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:571) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.reportRSFatalError(RegionServerStatusProtos.java:13568) at org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:2368) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.abortRegionServer(MiniHBaseCluster.java:208) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$1(MiniHBaseCluster.java:207) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$2.run(MiniHBaseCluster.java:201) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.abort(MiniHBaseCluster.java:198) at org.apache.hadoop.hbase.executor.EventHandler.handleException(EventHandler.java:188) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:106) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Call to ubuntu/127.0.1.1:39063 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$3(AbstractRpcClient.java:375) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:1) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:423) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:328) ... 17 more Caused by: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.getConnection(AbstractRpcClient.java:352) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:420) ... 18 more 2018-02-04 11:05:40,266 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] handler.CloseRegionHandler(92): Processing close of tabletest,ttt,1517713232695.31b3e09f2db65dd6057a91f45cb8b163. 2018-02-04 11:05:40,267 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(1483): Closing tabletest,ttt,1517713232695.31b3e09f2db65dd6057a91f45cb8b163.: disabling compactions & flushes 2018-02-04 11:05:40,267 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegion(1522): Updates disabled for region tabletest,ttt,1517713232695.31b3e09f2db65dd6057a91f45cb8b163. 2018-02-04 11:05:40,271 INFO [StoreCloserThread-tabletest,ttt,1517713232695.31b3e09f2db65dd6057a91f45cb8b163.-1] regionserver.HStore(932): Closed family 2018-02-04 11:05:40,637 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-2] helpers.MarkerIgnoringBase(159): ***** ABORTING region server ubuntu,39267,1517713227316: Replay of WAL required. Forcing server shutdown ***** org.apache.hadoop.hbase.DroppedSnapshotException: region: tabletest,ddd,1517713232695.68c9676b485038d63fd2cf8a435420e5. at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2667) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2356) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.hbase.exceptions.TimeoutIOException: Failed to get sync result after 300000 ms for txid=138, WAL system stuck? at org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:136) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:718) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:605) at org.apache.hadoop.hbase.regionserver.wal.WALUtil.doFullAppendTransaction(WALUtil.java:154) at org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeFlushMarker(WALUtil.java:81) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2645) ... 10 more 2018-02-04 11:05:40,638 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-2] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-02-04 11:05:40,644 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegionServer(2350): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 881852416, "init" : 197132288, "max" : 2794455040, "used" : 266319336 }, "NonHeapMemoryUsage" : { "committed" : 123756544, "init" : 2555904, "max" : -1, "used" : 121876136 }, "Verbose" : false, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : -1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 6227078, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 207, "TotalCallTime_min" : 0, "TotalCallTime_max" : 0, "TotalCallTime_mean" : 0, "TotalCallTime_25th_percentile" : 0, "TotalCallTime_median" : 0, "TotalCallTime_75th_percentile" : 0, "TotalCallTime_90th_percentile" : 0, "TotalCallTime_95th_percentile" : 0, "TotalCallTime_98th_percentile" : 0, "TotalCallTime_99th_percentile" : 0, "TotalCallTime_99.9th_percentile" : 0, "TotalCallTime_TimeRangeCount_600000-inf" : 207, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 207, "ResponseSize_min" : 0, "ResponseSize_max" : 0, "ResponseSize_mean" : 0, "ResponseSize_25th_percentile" : 0, "ResponseSize_median" : 0, "ResponseSize_75th_percentile" : 0, "ResponseSize_90th_percentile" : 0, "ResponseSize_95th_percentile" : 0, "ResponseSize_98th_percentile" : 0, "ResponseSize_99th_percentile" : 0, "ResponseSize_99.9th_percentile" : 0, "ResponseSize_SizeRangeCount_100000000-inf" : 207, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 207, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 0, "ProcessCallTime_mean" : 0, "ProcessCallTime_25th_percentile" : 0, "ProcessCallTime_median" : 0, "ProcessCallTime_75th_percentile" : 0, "ProcessCallTime_90th_percentile" : 0, "ProcessCallTime_95th_percentile" : 0, "ProcessCallTime_98th_percentile" : 0, "ProcessCallTime_99th_percentile" : 0, "ProcessCallTime_99.9th_percentile" : 0, "ProcessCallTime_TimeRangeCount_600000-inf" : 207, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 9, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 207, "RequestSize_min" : 0, "RequestSize_max" : 0, "RequestSize_mean" : 0, "RequestSize_25th_percentile" : 0, "RequestSize_median" : 0, "RequestSize_75th_percentile" : 0, "RequestSize_90th_percentile" : 0, "RequestSize_95th_percentile" : 0, "RequestSize_98th_percentile" : 0, "RequestSize_99th_percentile" : 0, "RequestSize_99.9th_percentile" : 0, "RequestSize_SizeRangeCount_100000000-inf" : 207, "sentBytes" : 2283235, "QueueCallTime_num_ops" : 207, "QueueCallTime_min" : 0, "QueueCallTime_max" : 0, "QueueCallTime_mean" : 0, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 0, "QueueCallTime_90th_percentile" : 0, "QueueCallTime_95th_percentile" : 0, "QueueCallTime_98th_percentile" : 0, "QueueCallTime_99th_percentile" : 0, "QueueCallTime_99.9th_percentile" : 0, "QueueCallTime_TimeRangeCount_600000-inf" : 207, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:56841", "tag.serverName" : "ubuntu,39267,1517713227316", "tag.clusterId" : "990c040f-15f4-4bef-a30c-08f39bc273d8", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "regionCount" : 8, "storeCount" : 8, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 2, "memStoreSize" : 534358, "storeFileSize" : 66522, "maxStoreFileAge" : 294103, "minStoreFileAge" : 293655, "avgStoreFileAge" : 293879, "numReferenceFiles" : 0, "regionServerStartTime" : 1517713227316, "averageRegionSize" : 75110, "totalRequestCount" : 32, "totalRowActionRequestCount" : 34451, "readRequestCount" : 9844, "filteredReadRequestCount" : 0, "writeRequestCount" : 24607, "rpcGetRequestCount" : 4, "rpcScanRequestCount" : 19, "rpcMultiRequestCount" : 5, "rpcMutateRequestCount" : 2, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 656, "staticIndexSize" : 80, "staticBloomSize" : 2048, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1116962296, "blockCacheCount" : 0, "blockCacheSize" : 819720, "blockCacheHitCount" : 0, "blockCacheHitCountPrimary" : 0, "blockCacheMissCount" : 0, "blockCacheMissCountPrimary" : 0, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 0.0, "blockCacheExpressHitPercent" : 0.0, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 0, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 0, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 0, "l1CacheMissCount" : 0, "l1CacheHitRatio" : NaN, "l1CacheMissRatio" : NaN, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 10545, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 400710, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 19, "ScanTime_min" : 0, "ScanTime_max" : 0, "ScanTime_mean" : 0, "ScanTime_25th_percentile" : 0, "ScanTime_median" : 0, "ScanTime_75th_percentile" : 0, "ScanTime_90th_percentile" : 0, "ScanTime_95th_percentile" : 0, "ScanTime_98th_percentile" : 0, "ScanTime_99th_percentile" : 0, "ScanTime_99.9th_percentile" : 0, "ScanTime_TimeRangeCount_600000-inf" : 19, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 2, "Put_min" : 0, "Put_max" : 0, "Put_mean" : 0, "Put_25th_percentile" : 0, "Put_median" : 0, "Put_75th_percentile" : 0, "Put_90th_percentile" : 0, "Put_95th_percentile" : 0, "Put_98th_percentile" : 0, "Put_99th_percentile" : 0, "Put_99.9th_percentile" : 0, "Put_TimeRangeCount_600000-inf" : 2, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 2, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "FlushMemstoreSize_SizeRangeCount_100000000-inf" : 2, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 45, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "PutBatch_TimeRangeCount_600000-inf" : 45, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 4, "Get_min" : 0, "Get_max" : 0, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 0, "Get_75th_percentile" : 0, "Get_90th_percentile" : 0, "Get_95th_percentile" : 0, "Get_98th_percentile" : 0, "Get_99th_percentile" : 0, "Get_99.9th_percentile" : 0, "Get_TimeRangeCount_600000-inf" : 4, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 66522, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 2, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "FlushTime_TimeRangeCount_600000-inf" : 2, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 19, "ScanSize_min" : 0, "ScanSize_max" : 0, "ScanSize_mean" : 0, "ScanSize_25th_percentile" : 0, "ScanSize_median" : 0, "ScanSize_75th_percentile" : 0, "ScanSize_90th_percentile" : 0, "ScanSize_95th_percentile" : 0, "ScanSize_98th_percentile" : 0, "ScanSize_99th_percentile" : 0, "ScanSize_99.9th_percentile" : 0, "ScanSize_SizeRangeCount_100000000-inf" : 19, "slowGetCount" : 0, "flushedMemstoreBytes" : 267140, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 2, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "FlushOutputSize_SizeRangeCount_100000000-inf" : 2, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-02-04 11:05:40,645 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] ipc.AbstractRpcClient(349): Not trying to connect to ubuntu/127.0.1.1:39063 this server is in the failed servers list 2018-02-04 11:05:40,645 WARN [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegionServer(2371): Unable to report fatal error to master org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: java.io.IOException: Call to ubuntu/127.0.1.1:39063 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$0(AbstractRpcClient.java:324) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:571) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.reportRSFatalError(RegionServerStatusProtos.java:13568) at org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:2368) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.abortRegionServer(MiniHBaseCluster.java:208) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$1(MiniHBaseCluster.java:207) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$2.run(MiniHBaseCluster.java:201) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.abort(MiniHBaseCluster.java:198) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2680) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2356) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Call to ubuntu/127.0.1.1:39063 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$3(AbstractRpcClient.java:375) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:1) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:423) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:328) ... 23 more Caused by: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.getConnection(AbstractRpcClient.java:352) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:420) ... 24 more 2018-02-04 11:05:40,647 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-2] helpers.MarkerIgnoringBase(159): ***** ABORTING region server ubuntu,39267,1517713227316: Unrecoverable exception while closing region tabletest,ddd,1517713232695.68c9676b485038d63fd2cf8a435420e5., still finishing close ***** org.apache.hadoop.hbase.DroppedSnapshotException: region: tabletest,ddd,1517713232695.68c9676b485038d63fd2cf8a435420e5. at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2667) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2356) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.hbase.exceptions.TimeoutIOException: Failed to get sync result after 300000 ms for txid=138, WAL system stuck? at org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:136) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:718) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:605) at org.apache.hadoop.hbase.regionserver.wal.WALUtil.doFullAppendTransaction(WALUtil.java:154) at org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeFlushMarker(WALUtil.java:81) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2645) ... 10 more 2018-02-04 11:05:40,647 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-2] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-02-04 11:05:40,650 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegionServer(2350): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 881852416, "init" : 197132288, "max" : 2794455040, "used" : 267411184 }, "NonHeapMemoryUsage" : { "committed" : 123756544, "init" : 2555904, "max" : -1, "used" : 121864456 }, "Verbose" : false, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : -1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 6227078, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 207, "TotalCallTime_min" : 0, "TotalCallTime_max" : 0, "TotalCallTime_mean" : 0, "TotalCallTime_25th_percentile" : 0, "TotalCallTime_median" : 0, "TotalCallTime_75th_percentile" : 0, "TotalCallTime_90th_percentile" : 0, "TotalCallTime_95th_percentile" : 0, "TotalCallTime_98th_percentile" : 0, "TotalCallTime_99th_percentile" : 0, "TotalCallTime_99.9th_percentile" : 0, "TotalCallTime_TimeRangeCount_600000-inf" : 207, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 207, "ResponseSize_min" : 0, "ResponseSize_max" : 0, "ResponseSize_mean" : 0, "ResponseSize_25th_percentile" : 0, "ResponseSize_median" : 0, "ResponseSize_75th_percentile" : 0, "ResponseSize_90th_percentile" : 0, "ResponseSize_95th_percentile" : 0, "ResponseSize_98th_percentile" : 0, "ResponseSize_99th_percentile" : 0, "ResponseSize_99.9th_percentile" : 0, "ResponseSize_SizeRangeCount_100000000-inf" : 207, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 207, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 0, "ProcessCallTime_mean" : 0, "ProcessCallTime_25th_percentile" : 0, "ProcessCallTime_median" : 0, "ProcessCallTime_75th_percentile" : 0, "ProcessCallTime_90th_percentile" : 0, "ProcessCallTime_95th_percentile" : 0, "ProcessCallTime_98th_percentile" : 0, "ProcessCallTime_99th_percentile" : 0, "ProcessCallTime_99.9th_percentile" : 0, "ProcessCallTime_TimeRangeCount_600000-inf" : 207, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 9, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 207, "RequestSize_min" : 0, "RequestSize_max" : 0, "RequestSize_mean" : 0, "RequestSize_25th_percentile" : 0, "RequestSize_median" : 0, "RequestSize_75th_percentile" : 0, "RequestSize_90th_percentile" : 0, "RequestSize_95th_percentile" : 0, "RequestSize_98th_percentile" : 0, "RequestSize_99th_percentile" : 0, "RequestSize_99.9th_percentile" : 0, "RequestSize_SizeRangeCount_100000000-inf" : 207, "sentBytes" : 2283235, "QueueCallTime_num_ops" : 207, "QueueCallTime_min" : 0, "QueueCallTime_max" : 0, "QueueCallTime_mean" : 0, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 0, "QueueCallTime_90th_percentile" : 0, "QueueCallTime_95th_percentile" : 0, "QueueCallTime_98th_percentile" : 0, "QueueCallTime_99th_percentile" : 0, "QueueCallTime_99.9th_percentile" : 0, "QueueCallTime_TimeRangeCount_600000-inf" : 207, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:56841", "tag.serverName" : "ubuntu,39267,1517713227316", "tag.clusterId" : "990c040f-15f4-4bef-a30c-08f39bc273d8", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "regionCount" : 8, "storeCount" : 8, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 2, "memStoreSize" : 534358, "storeFileSize" : 66522, "maxStoreFileAge" : 294103, "minStoreFileAge" : 293655, "avgStoreFileAge" : 293879, "numReferenceFiles" : 0, "regionServerStartTime" : 1517713227316, "averageRegionSize" : 75110, "totalRequestCount" : 32, "totalRowActionRequestCount" : 34451, "readRequestCount" : 9844, "filteredReadRequestCount" : 0, "writeRequestCount" : 24607, "rpcGetRequestCount" : 4, "rpcScanRequestCount" : 19, "rpcMultiRequestCount" : 5, "rpcMutateRequestCount" : 2, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 656, "staticIndexSize" : 80, "staticBloomSize" : 2048, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1116962296, "blockCacheCount" : 0, "blockCacheSize" : 819720, "blockCacheHitCount" : 0, "blockCacheHitCountPrimary" : 0, "blockCacheMissCount" : 0, "blockCacheMissCountPrimary" : 0, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 0.0, "blockCacheExpressHitPercent" : 0.0, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 0, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 0, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 0, "l1CacheMissCount" : 0, "l1CacheHitRatio" : NaN, "l1CacheMissRatio" : NaN, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 10545, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 400710, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 19, "ScanTime_min" : 0, "ScanTime_max" : 0, "ScanTime_mean" : 0, "ScanTime_25th_percentile" : 0, "ScanTime_median" : 0, "ScanTime_75th_percentile" : 0, "ScanTime_90th_percentile" : 0, "ScanTime_95th_percentile" : 0, "ScanTime_98th_percentile" : 0, "ScanTime_99th_percentile" : 0, "ScanTime_99.9th_percentile" : 0, "ScanTime_TimeRangeCount_600000-inf" : 19, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 2, "Put_min" : 0, "Put_max" : 0, "Put_mean" : 0, "Put_25th_percentile" : 0, "Put_median" : 0, "Put_75th_percentile" : 0, "Put_90th_percentile" : 0, "Put_95th_percentile" : 0, "Put_98th_percentile" : 0, "Put_99th_percentile" : 0, "Put_99.9th_percentile" : 0, "Put_TimeRangeCount_600000-inf" : 2, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 2, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "FlushMemstoreSize_SizeRangeCount_100000000-inf" : 2, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 45, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "PutBatch_TimeRangeCount_600000-inf" : 45, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 4, "Get_min" : 0, "Get_max" : 0, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 0, "Get_75th_percentile" : 0, "Get_90th_percentile" : 0, "Get_95th_percentile" : 0, "Get_98th_percentile" : 0, "Get_99th_percentile" : 0, "Get_99.9th_percentile" : 0, "Get_TimeRangeCount_600000-inf" : 4, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 66522, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 2, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "FlushTime_TimeRangeCount_600000-inf" : 2, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 19, "ScanSize_min" : 0, "ScanSize_max" : 0, "ScanSize_mean" : 0, "ScanSize_25th_percentile" : 0, "ScanSize_median" : 0, "ScanSize_75th_percentile" : 0, "ScanSize_90th_percentile" : 0, "ScanSize_95th_percentile" : 0, "ScanSize_98th_percentile" : 0, "ScanSize_99th_percentile" : 0, "ScanSize_99.9th_percentile" : 0, "ScanSize_SizeRangeCount_100000000-inf" : 19, "slowGetCount" : 0, "flushedMemstoreBytes" : 267140, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 2, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "FlushOutputSize_SizeRangeCount_100000000-inf" : 2, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-02-04 11:05:40,651 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] ipc.AbstractRpcClient(349): Not trying to connect to ubuntu/127.0.1.1:39063 this server is in the failed servers list 2018-02-04 11:05:40,651 WARN [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegionServer(2371): Unable to report fatal error to master org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: java.io.IOException: Call to ubuntu/127.0.1.1:39063 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$0(AbstractRpcClient.java:324) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:571) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.reportRSFatalError(RegionServerStatusProtos.java:13568) at org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:2368) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.abortRegionServer(MiniHBaseCluster.java:208) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$1(MiniHBaseCluster.java:207) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$2.run(MiniHBaseCluster.java:201) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.abort(MiniHBaseCluster.java:198) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:115) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Call to ubuntu/127.0.1.1:39063 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$3(AbstractRpcClient.java:375) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:1) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:423) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:328) ... 17 more Caused by: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.getConnection(AbstractRpcClient.java:352) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:420) ... 18 more 2018-02-04 11:05:40,652 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-2] executor.EventHandler(186): Caught throwable while processing event M_RS_CLOSE_REGION java.lang.RuntimeException: org.apache.hadoop.hbase.DroppedSnapshotException: region: tabletest,ddd,1517713232695.68c9676b485038d63fd2cf8a435420e5. at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:117) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.hbase.DroppedSnapshotException: region: tabletest,ddd,1517713232695.68c9676b485038d63fd2cf8a435420e5. at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2667) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2356) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) ... 4 more Caused by: org.apache.hadoop.hbase.exceptions.TimeoutIOException: Failed to get sync result after 300000 ms for txid=138, WAL system stuck? at org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:136) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:718) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:605) at org.apache.hadoop.hbase.regionserver.wal.WALUtil.doFullAppendTransaction(WALUtil.java:154) at org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeFlushMarker(WALUtil.java:81) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2645) ... 10 more 2018-02-04 11:05:40,653 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-2] helpers.MarkerIgnoringBase(159): ***** ABORTING region server ubuntu,39267,1517713227316: Caught throwable while processing event M_RS_CLOSE_REGION ***** java.lang.RuntimeException: org.apache.hadoop.hbase.DroppedSnapshotException: region: tabletest,ddd,1517713232695.68c9676b485038d63fd2cf8a435420e5. at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:117) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.hbase.DroppedSnapshotException: region: tabletest,ddd,1517713232695.68c9676b485038d63fd2cf8a435420e5. at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2667) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2356) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) ... 4 more Caused by: org.apache.hadoop.hbase.exceptions.TimeoutIOException: Failed to get sync result after 300000 ms for txid=138, WAL system stuck? at org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:136) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:718) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:605) at org.apache.hadoop.hbase.regionserver.wal.WALUtil.doFullAppendTransaction(WALUtil.java:154) at org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeFlushMarker(WALUtil.java:81) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2645) ... 10 more 2018-02-04 11:05:40,653 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-2] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-02-04 11:05:40,656 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegionServer(2350): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 881852416, "init" : 197132288, "max" : 2794455040, "used" : 268509368 }, "NonHeapMemoryUsage" : { "committed" : 123756544, "init" : 2555904, "max" : -1, "used" : 121856480 }, "Verbose" : false, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : -1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 6227078, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 207, "TotalCallTime_min" : 0, "TotalCallTime_max" : 0, "TotalCallTime_mean" : 0, "TotalCallTime_25th_percentile" : 0, "TotalCallTime_median" : 0, "TotalCallTime_75th_percentile" : 0, "TotalCallTime_90th_percentile" : 0, "TotalCallTime_95th_percentile" : 0, "TotalCallTime_98th_percentile" : 0, "TotalCallTime_99th_percentile" : 0, "TotalCallTime_99.9th_percentile" : 0, "TotalCallTime_TimeRangeCount_600000-inf" : 207, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 207, "ResponseSize_min" : 0, "ResponseSize_max" : 0, "ResponseSize_mean" : 0, "ResponseSize_25th_percentile" : 0, "ResponseSize_median" : 0, "ResponseSize_75th_percentile" : 0, "ResponseSize_90th_percentile" : 0, "ResponseSize_95th_percentile" : 0, "ResponseSize_98th_percentile" : 0, "ResponseSize_99th_percentile" : 0, "ResponseSize_99.9th_percentile" : 0, "ResponseSize_SizeRangeCount_100000000-inf" : 207, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 207, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 0, "ProcessCallTime_mean" : 0, "ProcessCallTime_25th_percentile" : 0, "ProcessCallTime_median" : 0, "ProcessCallTime_75th_percentile" : 0, "ProcessCallTime_90th_percentile" : 0, "ProcessCallTime_95th_percentile" : 0, "ProcessCallTime_98th_percentile" : 0, "ProcessCallTime_99th_percentile" : 0, "ProcessCallTime_99.9th_percentile" : 0, "ProcessCallTime_TimeRangeCount_600000-inf" : 207, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 9, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 207, "RequestSize_min" : 0, "RequestSize_max" : 0, "RequestSize_mean" : 0, "RequestSize_25th_percentile" : 0, "RequestSize_median" : 0, "RequestSize_75th_percentile" : 0, "RequestSize_90th_percentile" : 0, "RequestSize_95th_percentile" : 0, "RequestSize_98th_percentile" : 0, "RequestSize_99th_percentile" : 0, "RequestSize_99.9th_percentile" : 0, "RequestSize_SizeRangeCount_100000000-inf" : 207, "sentBytes" : 2283235, "QueueCallTime_num_ops" : 207, "QueueCallTime_min" : 0, "QueueCallTime_max" : 0, "QueueCallTime_mean" : 0, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 0, "QueueCallTime_90th_percentile" : 0, "QueueCallTime_95th_percentile" : 0, "QueueCallTime_98th_percentile" : 0, "QueueCallTime_99th_percentile" : 0, "QueueCallTime_99.9th_percentile" : 0, "QueueCallTime_TimeRangeCount_600000-inf" : 207, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:56841", "tag.serverName" : "ubuntu,39267,1517713227316", "tag.clusterId" : "990c040f-15f4-4bef-a30c-08f39bc273d8", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "regionCount" : 8, "storeCount" : 8, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 2, "memStoreSize" : 534358, "storeFileSize" : 66522, "maxStoreFileAge" : 294103, "minStoreFileAge" : 293655, "avgStoreFileAge" : 293879, "numReferenceFiles" : 0, "regionServerStartTime" : 1517713227316, "averageRegionSize" : 75110, "totalRequestCount" : 32, "totalRowActionRequestCount" : 34451, "readRequestCount" : 9844, "filteredReadRequestCount" : 0, "writeRequestCount" : 24607, "rpcGetRequestCount" : 4, "rpcScanRequestCount" : 19, "rpcMultiRequestCount" : 5, "rpcMutateRequestCount" : 2, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 656, "staticIndexSize" : 80, "staticBloomSize" : 2048, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1116962296, "blockCacheCount" : 0, "blockCacheSize" : 819720, "blockCacheHitCount" : 0, "blockCacheHitCountPrimary" : 0, "blockCacheMissCount" : 0, "blockCacheMissCountPrimary" : 0, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 0.0, "blockCacheExpressHitPercent" : 0.0, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 0, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 0, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 0, "l1CacheMissCount" : 0, "l1CacheHitRatio" : NaN, "l1CacheMissRatio" : NaN, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 10545, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 400710, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 19, "ScanTime_min" : 0, "ScanTime_max" : 0, "ScanTime_mean" : 0, "ScanTime_25th_percentile" : 0, "ScanTime_median" : 0, "ScanTime_75th_percentile" : 0, "ScanTime_90th_percentile" : 0, "ScanTime_95th_percentile" : 0, "ScanTime_98th_percentile" : 0, "ScanTime_99th_percentile" : 0, "ScanTime_99.9th_percentile" : 0, "ScanTime_TimeRangeCount_600000-inf" : 19, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 2, "Put_min" : 0, "Put_max" : 0, "Put_mean" : 0, "Put_25th_percentile" : 0, "Put_median" : 0, "Put_75th_percentile" : 0, "Put_90th_percentile" : 0, "Put_95th_percentile" : 0, "Put_98th_percentile" : 0, "Put_99th_percentile" : 0, "Put_99.9th_percentile" : 0, "Put_TimeRangeCount_600000-inf" : 2, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 2, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "FlushMemstoreSize_SizeRangeCount_100000000-inf" : 2, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 45, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "PutBatch_TimeRangeCount_600000-inf" : 45, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 4, "Get_min" : 0, "Get_max" : 0, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 0, "Get_75th_percentile" : 0, "Get_90th_percentile" : 0, "Get_95th_percentile" : 0, "Get_98th_percentile" : 0, "Get_99th_percentile" : 0, "Get_99.9th_percentile" : 0, "Get_TimeRangeCount_600000-inf" : 4, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 66522, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 2, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "FlushTime_TimeRangeCount_600000-inf" : 2, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 19, "ScanSize_min" : 0, "ScanSize_max" : 0, "ScanSize_mean" : 0, "ScanSize_25th_percentile" : 0, "ScanSize_median" : 0, "ScanSize_75th_percentile" : 0, "ScanSize_90th_percentile" : 0, "ScanSize_95th_percentile" : 0, "ScanSize_98th_percentile" : 0, "ScanSize_99th_percentile" : 0, "ScanSize_99.9th_percentile" : 0, "ScanSize_SizeRangeCount_100000000-inf" : 19, "slowGetCount" : 0, "flushedMemstoreBytes" : 267140, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 2, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "FlushOutputSize_SizeRangeCount_100000000-inf" : 2, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-02-04 11:05:40,657 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] ipc.AbstractRpcClient(349): Not trying to connect to ubuntu/127.0.1.1:39063 this server is in the failed servers list 2018-02-04 11:05:40,658 WARN [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegionServer(2371): Unable to report fatal error to master org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: java.io.IOException: Call to ubuntu/127.0.1.1:39063 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$0(AbstractRpcClient.java:324) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:571) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.reportRSFatalError(RegionServerStatusProtos.java:13568) at org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:2368) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.abortRegionServer(MiniHBaseCluster.java:208) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$1(MiniHBaseCluster.java:207) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$2.run(MiniHBaseCluster.java:201) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.abort(MiniHBaseCluster.java:198) at org.apache.hadoop.hbase.executor.EventHandler.handleException(EventHandler.java:188) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:106) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Call to ubuntu/127.0.1.1:39063 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$3(AbstractRpcClient.java:375) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:1) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:423) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:328) ... 17 more Caused by: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.getConnection(AbstractRpcClient.java:352) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:420) ... 18 more 2018-02-04 11:05:40,668 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] handler.CloseRegionHandler(92): Processing close of tabletest,ddd,1517713232695.68c9676b485038d63fd2cf8a435420e5. 2018-02-04 11:05:40,668 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(1483): Closing tabletest,ddd,1517713232695.68c9676b485038d63fd2cf8a435420e5.: disabling compactions & flushes 2018-02-04 11:05:40,668 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegion(1522): Updates disabled for region tabletest,ddd,1517713232695.68c9676b485038d63fd2cf8a435420e5. 2018-02-04 11:05:40,670 INFO [StoreCloserThread-tabletest,ddd,1517713232695.68c9676b485038d63fd2cf8a435420e5.-1] regionserver.HStore(932): Closed family 2018-02-04 11:05:40,737 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-1] helpers.MarkerIgnoringBase(159): ***** ABORTING region server ubuntu,39267,1517713227316: Unrecoverable exception while closing region tabletest,hhh,1517713232695.ad4f7bcab0a803ddae20b2866466d222., still finishing close ***** org.apache.hadoop.hbase.exceptions.TimeoutIOException: Failed to get sync result after 300000 ms for txid=140, WAL system stuck? at org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:136) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:718) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:605) at org.apache.hadoop.hbase.regionserver.wal.WALUtil.doFullAppendTransaction(WALUtil.java:154) at org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeMarker(WALUtil.java:132) at org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeRegionEventMarker(WALUtil.java:97) at org.apache.hadoop.hbase.regionserver.HRegion.writeRegionCloseMarker(HRegion.java:1103) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1615) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-02-04 11:05:40,737 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-1] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-02-04 11:05:40,739 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegionServer(2350): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 881852416, "init" : 197132288, "max" : 2794455040, "used" : 269860656 }, "NonHeapMemoryUsage" : { "committed" : 123756544, "init" : 2555904, "max" : -1, "used" : 121869496 }, "Verbose" : false, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : -1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 6227078, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 207, "TotalCallTime_min" : 0, "TotalCallTime_max" : 0, "TotalCallTime_mean" : 0, "TotalCallTime_25th_percentile" : 0, "TotalCallTime_median" : 0, "TotalCallTime_75th_percentile" : 0, "TotalCallTime_90th_percentile" : 0, "TotalCallTime_95th_percentile" : 0, "TotalCallTime_98th_percentile" : 0, "TotalCallTime_99th_percentile" : 0, "TotalCallTime_99.9th_percentile" : 0, "TotalCallTime_TimeRangeCount_600000-inf" : 207, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 207, "ResponseSize_min" : 0, "ResponseSize_max" : 0, "ResponseSize_mean" : 0, "ResponseSize_25th_percentile" : 0, "ResponseSize_median" : 0, "ResponseSize_75th_percentile" : 0, "ResponseSize_90th_percentile" : 0, "ResponseSize_95th_percentile" : 0, "ResponseSize_98th_percentile" : 0, "ResponseSize_99th_percentile" : 0, "ResponseSize_99.9th_percentile" : 0, "ResponseSize_SizeRangeCount_100000000-inf" : 207, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 207, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 0, "ProcessCallTime_mean" : 0, "ProcessCallTime_25th_percentile" : 0, "ProcessCallTime_median" : 0, "ProcessCallTime_75th_percentile" : 0, "ProcessCallTime_90th_percentile" : 0, "ProcessCallTime_95th_percentile" : 0, "ProcessCallTime_98th_percentile" : 0, "ProcessCallTime_99th_percentile" : 0, "ProcessCallTime_99.9th_percentile" : 0, "ProcessCallTime_TimeRangeCount_600000-inf" : 207, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 9, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 207, "RequestSize_min" : 0, "RequestSize_max" : 0, "RequestSize_mean" : 0, "RequestSize_25th_percentile" : 0, "RequestSize_median" : 0, "RequestSize_75th_percentile" : 0, "RequestSize_90th_percentile" : 0, "RequestSize_95th_percentile" : 0, "RequestSize_98th_percentile" : 0, "RequestSize_99th_percentile" : 0, "RequestSize_99.9th_percentile" : 0, "RequestSize_SizeRangeCount_100000000-inf" : 207, "sentBytes" : 2283235, "QueueCallTime_num_ops" : 207, "QueueCallTime_min" : 0, "QueueCallTime_max" : 0, "QueueCallTime_mean" : 0, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 0, "QueueCallTime_90th_percentile" : 0, "QueueCallTime_95th_percentile" : 0, "QueueCallTime_98th_percentile" : 0, "QueueCallTime_99th_percentile" : 0, "QueueCallTime_99.9th_percentile" : 0, "QueueCallTime_TimeRangeCount_600000-inf" : 207, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:56841", "tag.serverName" : "ubuntu,39267,1517713227316", "tag.clusterId" : "990c040f-15f4-4bef-a30c-08f39bc273d8", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "regionCount" : 8, "storeCount" : 8, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 2, "memStoreSize" : 534358, "storeFileSize" : 66522, "maxStoreFileAge" : 294103, "minStoreFileAge" : 293655, "avgStoreFileAge" : 293879, "numReferenceFiles" : 0, "regionServerStartTime" : 1517713227316, "averageRegionSize" : 75110, "totalRequestCount" : 32, "totalRowActionRequestCount" : 34451, "readRequestCount" : 9844, "filteredReadRequestCount" : 0, "writeRequestCount" : 24607, "rpcGetRequestCount" : 4, "rpcScanRequestCount" : 19, "rpcMultiRequestCount" : 5, "rpcMutateRequestCount" : 2, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 656, "staticIndexSize" : 80, "staticBloomSize" : 2048, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1116962296, "blockCacheCount" : 0, "blockCacheSize" : 819720, "blockCacheHitCount" : 0, "blockCacheHitCountPrimary" : 0, "blockCacheMissCount" : 0, "blockCacheMissCountPrimary" : 0, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 0.0, "blockCacheExpressHitPercent" : 0.0, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 0, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 0, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 0, "l1CacheMissCount" : 0, "l1CacheHitRatio" : NaN, "l1CacheMissRatio" : NaN, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 10545, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 400710, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 19, "ScanTime_min" : 0, "ScanTime_max" : 0, "ScanTime_mean" : 0, "ScanTime_25th_percentile" : 0, "ScanTime_median" : 0, "ScanTime_75th_percentile" : 0, "ScanTime_90th_percentile" : 0, "ScanTime_95th_percentile" : 0, "ScanTime_98th_percentile" : 0, "ScanTime_99th_percentile" : 0, "ScanTime_99.9th_percentile" : 0, "ScanTime_TimeRangeCount_600000-inf" : 19, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 2, "Put_min" : 0, "Put_max" : 0, "Put_mean" : 0, "Put_25th_percentile" : 0, "Put_median" : 0, "Put_75th_percentile" : 0, "Put_90th_percentile" : 0, "Put_95th_percentile" : 0, "Put_98th_percentile" : 0, "Put_99th_percentile" : 0, "Put_99.9th_percentile" : 0, "Put_TimeRangeCount_600000-inf" : 2, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 2, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "FlushMemstoreSize_SizeRangeCount_100000000-inf" : 2, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 45, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "PutBatch_TimeRangeCount_600000-inf" : 45, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 4, "Get_min" : 0, "Get_max" : 0, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 0, "Get_75th_percentile" : 0, "Get_90th_percentile" : 0, "Get_95th_percentile" : 0, "Get_98th_percentile" : 0, "Get_99th_percentile" : 0, "Get_99.9th_percentile" : 0, "Get_TimeRangeCount_600000-inf" : 4, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 66522, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 2, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "FlushTime_TimeRangeCount_600000-inf" : 2, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 19, "ScanSize_min" : 0, "ScanSize_max" : 0, "ScanSize_mean" : 0, "ScanSize_25th_percentile" : 0, "ScanSize_median" : 0, "ScanSize_75th_percentile" : 0, "ScanSize_90th_percentile" : 0, "ScanSize_95th_percentile" : 0, "ScanSize_98th_percentile" : 0, "ScanSize_99th_percentile" : 0, "ScanSize_99.9th_percentile" : 0, "ScanSize_SizeRangeCount_100000000-inf" : 19, "slowGetCount" : 0, "flushedMemstoreBytes" : 267140, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 2, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "FlushOutputSize_SizeRangeCount_100000000-inf" : 2, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-02-04 11:05:40,742 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] ipc.AbstractRpcClient(349): Not trying to connect to ubuntu/127.0.1.1:39063 this server is in the failed servers list 2018-02-04 11:05:40,743 WARN [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegionServer(2371): Unable to report fatal error to master org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: java.io.IOException: Call to ubuntu/127.0.1.1:39063 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$0(AbstractRpcClient.java:324) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:571) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.reportRSFatalError(RegionServerStatusProtos.java:13568) at org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:2368) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.abortRegionServer(MiniHBaseCluster.java:208) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$1(MiniHBaseCluster.java:207) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$2.run(MiniHBaseCluster.java:201) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.abort(MiniHBaseCluster.java:198) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:115) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Call to ubuntu/127.0.1.1:39063 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$3(AbstractRpcClient.java:375) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:1) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:423) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:328) ... 17 more Caused by: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.getConnection(AbstractRpcClient.java:352) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:420) ... 18 more 2018-02-04 11:05:40,744 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-1] executor.EventHandler(186): Caught throwable while processing event M_RS_CLOSE_REGION java.lang.RuntimeException: org.apache.hadoop.hbase.exceptions.TimeoutIOException: Failed to get sync result after 300000 ms for txid=140, WAL system stuck? at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:117) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.hbase.exceptions.TimeoutIOException: Failed to get sync result after 300000 ms for txid=140, WAL system stuck? at org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:136) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:718) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:605) at org.apache.hadoop.hbase.regionserver.wal.WALUtil.doFullAppendTransaction(WALUtil.java:154) at org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeMarker(WALUtil.java:132) at org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeRegionEventMarker(WALUtil.java:97) at org.apache.hadoop.hbase.regionserver.HRegion.writeRegionCloseMarker(HRegion.java:1103) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1615) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) ... 4 more 2018-02-04 11:05:40,744 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-1] helpers.MarkerIgnoringBase(159): ***** ABORTING region server ubuntu,39267,1517713227316: Caught throwable while processing event M_RS_CLOSE_REGION ***** java.lang.RuntimeException: org.apache.hadoop.hbase.exceptions.TimeoutIOException: Failed to get sync result after 300000 ms for txid=140, WAL system stuck? at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:117) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.hbase.exceptions.TimeoutIOException: Failed to get sync result after 300000 ms for txid=140, WAL system stuck? at org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:136) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:718) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:605) at org.apache.hadoop.hbase.regionserver.wal.WALUtil.doFullAppendTransaction(WALUtil.java:154) at org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeMarker(WALUtil.java:132) at org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeRegionEventMarker(WALUtil.java:97) at org.apache.hadoop.hbase.regionserver.HRegion.writeRegionCloseMarker(HRegion.java:1103) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1615) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) ... 4 more 2018-02-04 11:05:40,744 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-1] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-02-04 11:05:40,745 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegionServer(2350): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 881852416, "init" : 197132288, "max" : 2794455040, "used" : 270941888 }, "NonHeapMemoryUsage" : { "committed" : 123756544, "init" : 2555904, "max" : -1, "used" : 121874976 }, "Verbose" : false, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : -1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 6227078, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 207, "TotalCallTime_min" : 0, "TotalCallTime_max" : 0, "TotalCallTime_mean" : 0, "TotalCallTime_25th_percentile" : 0, "TotalCallTime_median" : 0, "TotalCallTime_75th_percentile" : 0, "TotalCallTime_90th_percentile" : 0, "TotalCallTime_95th_percentile" : 0, "TotalCallTime_98th_percentile" : 0, "TotalCallTime_99th_percentile" : 0, "TotalCallTime_99.9th_percentile" : 0, "TotalCallTime_TimeRangeCount_600000-inf" : 207, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 207, "ResponseSize_min" : 0, "ResponseSize_max" : 0, "ResponseSize_mean" : 0, "ResponseSize_25th_percentile" : 0, "ResponseSize_median" : 0, "ResponseSize_75th_percentile" : 0, "ResponseSize_90th_percentile" : 0, "ResponseSize_95th_percentile" : 0, "ResponseSize_98th_percentile" : 0, "ResponseSize_99th_percentile" : 0, "ResponseSize_99.9th_percentile" : 0, "ResponseSize_SizeRangeCount_100000000-inf" : 207, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 207, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 0, "ProcessCallTime_mean" : 0, "ProcessCallTime_25th_percentile" : 0, "ProcessCallTime_median" : 0, "ProcessCallTime_75th_percentile" : 0, "ProcessCallTime_90th_percentile" : 0, "ProcessCallTime_95th_percentile" : 0, "ProcessCallTime_98th_percentile" : 0, "ProcessCallTime_99th_percentile" : 0, "ProcessCallTime_99.9th_percentile" : 0, "ProcessCallTime_TimeRangeCount_600000-inf" : 207, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 9, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 207, "RequestSize_min" : 0, "RequestSize_max" : 0, "RequestSize_mean" : 0, "RequestSize_25th_percentile" : 0, "RequestSize_median" : 0, "RequestSize_75th_percentile" : 0, "RequestSize_90th_percentile" : 0, "RequestSize_95th_percentile" : 0, "RequestSize_98th_percentile" : 0, "RequestSize_99th_percentile" : 0, "RequestSize_99.9th_percentile" : 0, "RequestSize_SizeRangeCount_100000000-inf" : 207, "sentBytes" : 2283235, "QueueCallTime_num_ops" : 207, "QueueCallTime_min" : 0, "QueueCallTime_max" : 0, "QueueCallTime_mean" : 0, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 0, "QueueCallTime_90th_percentile" : 0, "QueueCallTime_95th_percentile" : 0, "QueueCallTime_98th_percentile" : 0, "QueueCallTime_99th_percentile" : 0, "QueueCallTime_99.9th_percentile" : 0, "QueueCallTime_TimeRangeCount_600000-inf" : 207, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:56841", "tag.serverName" : "ubuntu,39267,1517713227316", "tag.clusterId" : "990c040f-15f4-4bef-a30c-08f39bc273d8", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "regionCount" : 8, "storeCount" : 8, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 2, "memStoreSize" : 534358, "storeFileSize" : 66522, "maxStoreFileAge" : 294103, "minStoreFileAge" : 293655, "avgStoreFileAge" : 293879, "numReferenceFiles" : 0, "regionServerStartTime" : 1517713227316, "averageRegionSize" : 75110, "totalRequestCount" : 32, "totalRowActionRequestCount" : 34451, "readRequestCount" : 9844, "filteredReadRequestCount" : 0, "writeRequestCount" : 24607, "rpcGetRequestCount" : 4, "rpcScanRequestCount" : 19, "rpcMultiRequestCount" : 5, "rpcMutateRequestCount" : 2, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 656, "staticIndexSize" : 80, "staticBloomSize" : 2048, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1116962296, "blockCacheCount" : 0, "blockCacheSize" : 819720, "blockCacheHitCount" : 0, "blockCacheHitCountPrimary" : 0, "blockCacheMissCount" : 0, "blockCacheMissCountPrimary" : 0, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 0.0, "blockCacheExpressHitPercent" : 0.0, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 0, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 0, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 0, "l1CacheMissCount" : 0, "l1CacheHitRatio" : NaN, "l1CacheMissRatio" : NaN, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 10545, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 400710, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 19, "ScanTime_min" : 0, "ScanTime_max" : 0, "ScanTime_mean" : 0, "ScanTime_25th_percentile" : 0, "ScanTime_median" : 0, "ScanTime_75th_percentile" : 0, "ScanTime_90th_percentile" : 0, "ScanTime_95th_percentile" : 0, "ScanTime_98th_percentile" : 0, "ScanTime_99th_percentile" : 0, "ScanTime_99.9th_percentile" : 0, "ScanTime_TimeRangeCount_600000-inf" : 19, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 2, "Put_min" : 0, "Put_max" : 0, "Put_mean" : 0, "Put_25th_percentile" : 0, "Put_median" : 0, "Put_75th_percentile" : 0, "Put_90th_percentile" : 0, "Put_95th_percentile" : 0, "Put_98th_percentile" : 0, "Put_99th_percentile" : 0, "Put_99.9th_percentile" : 0, "Put_TimeRangeCount_600000-inf" : 2, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 2, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "FlushMemstoreSize_SizeRangeCount_100000000-inf" : 2, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 45, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "PutBatch_TimeRangeCount_600000-inf" : 45, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 4, "Get_min" : 0, "Get_max" : 0, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 0, "Get_75th_percentile" : 0, "Get_90th_percentile" : 0, "Get_95th_percentile" : 0, "Get_98th_percentile" : 0, "Get_99th_percentile" : 0, "Get_99.9th_percentile" : 0, "Get_TimeRangeCount_600000-inf" : 4, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 66522, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 2, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "FlushTime_TimeRangeCount_600000-inf" : 2, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 19, "ScanSize_min" : 0, "ScanSize_max" : 0, "ScanSize_mean" : 0, "ScanSize_25th_percentile" : 0, "ScanSize_median" : 0, "ScanSize_75th_percentile" : 0, "ScanSize_90th_percentile" : 0, "ScanSize_95th_percentile" : 0, "ScanSize_98th_percentile" : 0, "ScanSize_99th_percentile" : 0, "ScanSize_99.9th_percentile" : 0, "ScanSize_SizeRangeCount_100000000-inf" : 19, "slowGetCount" : 0, "flushedMemstoreBytes" : 267140, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 2, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "FlushOutputSize_SizeRangeCount_100000000-inf" : 2, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-02-04 11:05:40,746 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-1] ipc.AbstractRpcClient(349): Not trying to connect to ubuntu/127.0.1.1:39063 this server is in the failed servers list 2018-02-04 11:05:40,746 WARN [RS_CLOSE_REGION-regionserver/ubuntu:0-1] regionserver.HRegionServer(2371): Unable to report fatal error to master org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: java.io.IOException: Call to ubuntu/127.0.1.1:39063 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$0(AbstractRpcClient.java:324) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:571) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.reportRSFatalError(RegionServerStatusProtos.java:13568) at org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:2368) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.abortRegionServer(MiniHBaseCluster.java:208) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$1(MiniHBaseCluster.java:207) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$2.run(MiniHBaseCluster.java:201) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.abort(MiniHBaseCluster.java:198) at org.apache.hadoop.hbase.executor.EventHandler.handleException(EventHandler.java:188) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:106) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Call to ubuntu/127.0.1.1:39063 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$3(AbstractRpcClient.java:375) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:1) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:423) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:328) ... 17 more Caused by: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.getConnection(AbstractRpcClient.java:352) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:420) ... 18 more Process Thread Dump: Automatic Stack Trace every 60 seconds waiting on RS:0;ubuntu:39267 205 active threads Thread 1358 (Timer for 'HBase' metrics system): State: TIMED_WAITING Blocked count: 0 Waited count: 2 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 1355 (process reaper): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 1345 (Attach Listener): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: Thread 1069 (RS_CLOSE_REGION-regionserver/ubuntu:0-2): State: TIMED_WAITING Blocked count: 72 Waited count: 396 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:133) org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:718) org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:605) org.apache.hadoop.hbase.regionserver.wal.WALUtil.doFullAppendTransaction(WALUtil.java:154) org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeMarker(WALUtil.java:132) org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeRegionEventMarker(WALUtil.java:97) org.apache.hadoop.hbase.regionserver.HRegion.writeRegionCloseMarker(HRegion.java:1103) org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1615) org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 1062 (RS_CLOSE_REGION-regionserver/ubuntu:0-1): State: WAITING Blocked count: 54 Waited count: 344 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5fbe8967 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 1061 (RS_CLOSE_REGION-regionserver/ubuntu:0-0): State: TIMED_WAITING Blocked count: 67 Waited count: 403 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:133) org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:718) org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:605) org.apache.hadoop.hbase.regionserver.wal.WALUtil.doFullAppendTransaction(WALUtil.java:154) org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeMarker(WALUtil.java:132) org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeRegionEventMarker(WALUtil.java:97) org.apache.hadoop.hbase.regionserver.HRegion.writeRegionCloseMarker(HRegion.java:1103) org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1615) org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 988 (RS-EventLoopGroup-3-4): State: RUNNABLE Blocked count: 15 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 987 (Default-IPC-NioEventLoopGroup-7-3): State: RUNNABLE Blocked count: 23 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 986 (Default-IPC-NioEventLoopGroup-7-4): State: RUNNABLE Blocked count: 18 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 845 (RS_OPEN_REGION-regionserver/ubuntu:0-2): State: WAITING Blocked count: 50 Waited count: 67 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@72aa087e Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 842 (RS_OPEN_REGION-regionserver/ubuntu:0-1): State: WAITING Blocked count: 42 Waited count: 66 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@72aa087e Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 841 (RS_OPEN_REGION-regionserver/ubuntu:0-0): State: WAITING Blocked count: 42 Waited count: 64 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@72aa087e Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 810 (region-location-4): State: WAITING Blocked count: 5 Waited count: 4 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@282bcdfd Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 809 (region-location-3): State: WAITING Blocked count: 8 Waited count: 4 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@282bcdfd Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 808 (region-location-2): State: WAITING Blocked count: 13 Waited count: 7 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@282bcdfd Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 807 (region-location-1): State: WAITING Blocked count: 11 Waited count: 14 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@282bcdfd Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 571 (RS-EventLoopGroup-1-5): State: RUNNABLE Blocked count: 2 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 570 (Default-IPC-NioEventLoopGroup-7-2): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 568 (RS-EventLoopGroup-4-5): State: RUNNABLE Blocked count: 15 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 567 (Default-IPC-NioEventLoopGroup-7-1): State: RUNNABLE Blocked count: 49 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 560 (RS-EventLoopGroup-3-3): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 553 (RS_OPEN_PRIORITY_REGION-regionserver/ubuntu:0-0): State: WAITING Blocked count: 14 Waited count: 26 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6b2dcd67 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 551 (RS-EventLoopGroup-3-2): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 549 (RS-EventLoopGroup-4-4): State: RUNNABLE Blocked count: 1 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 544 (region-location-0): State: WAITING Blocked count: 12 Waited count: 18 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@282bcdfd Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 521 (RS-EventLoopGroup-4-3): State: RUNNABLE Blocked count: 1 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 508 (RS-EventLoopGroup-4-2): State: RUNNABLE Blocked count: 1 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 504 (AsyncFSWAL-0): State: WAITING Blocked count: 0 Waited count: 169 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1c3dccfe Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 503 (RS-EventLoopGroup-5-8): State: RUNNABLE Blocked count: 7 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 497 (RS-EventLoopGroup-5-7): State: RUNNABLE Blocked count: 1 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 483 (RS-EventLoopGroup-5-6): State: RUNNABLE Blocked count: 3 Waited count: 2 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 480 (LeaseRenewer:zhangduo.hfs.0@localhost:44491): State: TIMED_WAITING Blocked count: 10 Waited count: 340 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:444) org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304) java.lang.Thread.run(Thread.java:748) Thread 482 (RS-EventLoopGroup-5-5): State: RUNNABLE Blocked count: 12 Waited count: 3 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 476 (ubuntu:39267Replication Statistics #0): State: TIMED_WAITING Blocked count: 0 Waited count: 2 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 470 (ReplicationExecutor-0): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@153c8699 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 458 (regionserver/ubuntu:0.procedureResultReporter): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@788f3b3e Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:75) Thread 455 (regionserver/ubuntu:0.logRoller): State: TIMED_WAITING Blocked count: 6 Waited count: 326 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:147) java.lang.Thread.run(Thread.java:748) Thread 450 (regionserver/ubuntu:0.procedureResultReporter): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73ec8f20 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:75) Thread 452 (regionserver/ubuntu:0.procedureResultReporter): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@723b1ca8 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:75) Thread 443 (regionserver/ubuntu:0.Chore.1): State: TIMED_WAITING Blocked count: 1 Waited count: 33 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 431 (JvmPauseMonitor): State: TIMED_WAITING Blocked count: 0 Waited count: 641 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:154) java.lang.Thread.run(Thread.java:748) Thread 423 (ReadOnlyZKClient-localhost:56841@0x796736be): State: TIMED_WAITING Blocked count: 1 Waited count: 8 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:307) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$63/1354899920.run(Unknown Source) java.lang.Thread.run(Thread.java:748) Thread 422 (ReadOnlyZKClient-localhost:56841@0x143faef3): State: TIMED_WAITING Blocked count: 1 Waited count: 8 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:307) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$63/1354899920.run(Unknown Source) java.lang.Thread.run(Thread.java:748) Thread 421 (ReadOnlyZKClient-localhost:56841@0x46e70b17): State: TIMED_WAITING Blocked count: 1 Waited count: 8 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:307) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$63/1354899920.run(Unknown Source) java.lang.Thread.run(Thread.java:748) Thread 419 (IPC Parameter Sending Thread #1): State: TIMED_WAITING Blocked count: 0 Waited count: 1767 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 416 (threadDeathWatcher-6-1): State: TIMED_WAITING Blocked count: 0 Waited count: 321 Stack: java.lang.Thread.sleep(Native Method) org.apache.hbase.thirdparty.io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:152) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 415 (RS-EventLoopGroup-1-4): State: RUNNABLE Blocked count: 29 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 414 (RS-EventLoopGroup-1-3): State: RUNNABLE Blocked count: 18 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 413 (RS-EventLoopGroup-1-2): State: RUNNABLE Blocked count: 16 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 367 (RpcClient-timer-pool1-t1): State: TIMED_WAITING Blocked count: 0 Waited count: 32074 Stack: java.lang.Thread.sleep(Native Method) org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:560) org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:459) java.lang.Thread.run(Thread.java:748) Thread 410 (RS-EventLoopGroup-5-4): State: RUNNABLE Blocked count: 25 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 411 (RS-EventLoopGroup-5-2): State: RUNNABLE Blocked count: 56 Waited count: 3 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 412 (RS-EventLoopGroup-5-3): State: RUNNABLE Blocked count: 31 Waited count: 4 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 407 (snapshot-hfile-cleaner-cache-refresher): State: TIMED_WAITING Blocked count: 1 Waited count: 4 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 400 (M:0;ubuntu:39063-EventThread): State: WAITING Blocked count: 1 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4fcbce78 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501) Thread 399 (M:0;ubuntu:39063-SendThread(localhost:56841)): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141) Thread 289 (RS:0;ubuntu:39267): State: TIMED_WAITING Blocked count: 198 Waited count: 1609 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.regionserver.HRegionServer.sleep(HRegionServer.java:1425) org.apache.hadoop.hbase.regionserver.HRegionServer.waitOnAllRegionsToClose(HRegionServer.java:1411) org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1103) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:187) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$0(MiniHBaseCluster.java:186) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:171) java.security.AccessController.doPrivileged(Native Method) javax.security.auth.Subject.doAs(Subject.java:360) org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726) org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:168) java.lang.Thread.run(Thread.java:748) Thread 368 (Idle-Rpc-Conn-Sweeper-pool2-t1): State: TIMED_WAITING Blocked count: 0 Waited count: 15 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 334 (org.apache.hadoop.hdfs.PeerCache@2e1946df): State: TIMED_WAITING Blocked count: 1 Waited count: 108 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:255) org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:46) org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:124) java.lang.Thread.run(Thread.java:748) Thread 332 (M:0;ubuntu:39063-MemStoreChunkPool Statistics): State: TIMED_WAITING Blocked count: 0 Waited count: 2 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 328 (Monitor thread for TaskMonitor): State: TIMED_WAITING Blocked count: 0 Waited count: 33 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.monitoring.TaskMonitor$MonitorRunnable.run(TaskMonitor.java:302) java.lang.Thread.run(Thread.java:748) Thread 310 (RS-EventLoopGroup-5-1): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 291 (RS-EventLoopGroup-4-1): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 288 (RpcServer.replication.FPBQ.Fifo.handler=2,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@50f24e6c Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 287 (RpcServer.replication.FPBQ.Fifo.handler=1,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@5b366694 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 286 (RpcServer.replication.FPBQ.Fifo.handler=0,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@68239fa7 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 285 (RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=39267): State: WAITING Blocked count: 8 Waited count: 20 Waiting on java.util.concurrent.Semaphore$NonfairSync@64e05a8c Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 284 (RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@26a3f952 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 283 (RpcServer.priority.FPBQ.Fifo.handler=3,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@4ec929d4 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 282 (RpcServer.priority.FPBQ.Fifo.handler=2,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@3ae3490d Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 281 (RpcServer.priority.FPBQ.Fifo.handler=1,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@5cb140e7 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 280 (RpcServer.priority.FPBQ.Fifo.handler=0,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@65614b4a Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 279 (RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39267): State: WAITING Blocked count: 33 Waited count: 56 Waiting on java.util.concurrent.Semaphore$NonfairSync@626159bb Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 278 (RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 2 Waiting on java.util.concurrent.Semaphore$NonfairSync@42eadfdb Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 277 (RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@148a1919 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 276 (RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@7b5e2bd9 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 275 (RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@139a0988 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 272 (RS-EventLoopGroup-3-1): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 248 (RS-EventLoopGroup-1-1): State: RUNNABLE Blocked count: 1 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 247 (HBase-Metrics2-1): State: TIMED_WAITING Blocked count: 0 Waited count: 354 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 235 (ProcessThread(sid:0 cport:56841):): State: WAITING Blocked count: 1 Waited count: 699 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@133caf02 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:122) Thread 234 (SyncThread:0): State: WAITING Blocked count: 0 Waited count: 671 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5c669c88 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:127) Thread 233 (SessionTracker): State: TIMED_WAITING Blocked count: 0 Waited count: 163 Stack: java.lang.Object.wait(Native Method) org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146) Thread 232 (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:56841): State: RUNNABLE Blocked count: 9 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:173) java.lang.Thread.run(Thread.java:748) Thread 229 (java.util.concurrent.ThreadPoolExecutor$Worker@1d726817[State = -1, empty queue]): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 224 (refreshUsed-/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data6/current/BP-35161775-127.0.1.1-1517713219102): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:132) java.lang.Thread.run(Thread.java:748) Thread 222 (refreshUsed-/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data5/current/BP-35161775-127.0.1.1-1517713219102): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:132) java.lang.Thread.run(Thread.java:748) Thread 218 (org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl$LazyWriter@2327f0e2): State: TIMED_WAITING Blocked count: 0 Waited count: 6 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl$LazyWriter.run(FsDatasetImpl.java:3071) java.lang.Thread.run(Thread.java:748) Thread 217 (VolumeScannerThread(/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data6)): State: TIMED_WAITING Blocked count: 1 Waited count: 2 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:622) Thread 216 (VolumeScannerThread(/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data5)): State: TIMED_WAITING Blocked count: 1 Waited count: 2 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:622) Thread 213 (IPC Server handler 9 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 360 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 212 (IPC Server handler 8 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 358 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 211 (IPC Server handler 7 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 346 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 210 (IPC Server handler 6 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 374 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 209 (IPC Server handler 5 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 349 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 208 (IPC Server handler 4 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 375 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 207 (IPC Server handler 3 on 43825): State: TIMED_WAITING Blocked count: 1 Waited count: 329 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 206 (IPC Server handler 2 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 368 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 205 (IPC Server handler 1 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 393 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 204 (IPC Server handler 0 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 327 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 199 (IPC Server listener on 43825): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener.run(Server.java:807) Thread 202 (IPC Server Responder): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.ipc.Server$Responder.doRunLoop(Server.java:982) org.apache.hadoop.ipc.Server$Responder.run(Server.java:965) Thread 161 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@3e65225e): State: RUNNABLE Blocked count: 1 Waited count: 0 Stack: sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:100) org.apache.hadoop.hdfs.net.TcpPeerServer.accept(TcpPeerServer.java:141) org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:135) java.lang.Thread.run(Thread.java:748) Thread 203 (DataNode: [[[DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data5/, [DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data6/]] heartbeating to localhost/127.0.0.1:44491): State: TIMED_WAITING Blocked count: 472 Waited count: 1144 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.IncrementalBlockReportManager.waitTillNextIBR(IncrementalBlockReportManager.java:130) org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:542) org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:659) java.lang.Thread.run(Thread.java:748) Thread 201 (IPC Server idle connection scanner for port 43825): State: TIMED_WAITING Blocked count: 1 Waited count: 34 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 200 (Socket Reader #1 for port 43825): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:745) org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:724) Thread 198 (org.apache.hadoop.util.JvmPauseMonitor$Monitor@3369a0f5): State: TIMED_WAITING Blocked count: 0 Waited count: 649 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:182) java.lang.Thread.run(Thread.java:748) Thread 182 (nioEventLoopGroup-6-1): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) java.lang.Thread.run(Thread.java:748) Thread 181 (Timer-3): State: TIMED_WAITING Blocked count: 0 Waited count: 11 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 180 (java.util.concurrent.ThreadPoolExecutor$Worker@4231ed0c[State = -1, empty queue]): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 175 (refreshUsed-/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data4/current/BP-35161775-127.0.1.1-1517713219102): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:132) java.lang.Thread.run(Thread.java:748) Thread 173 (refreshUsed-/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data3/current/BP-35161775-127.0.1.1-1517713219102): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:132) java.lang.Thread.run(Thread.java:748) Thread 169 (org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl$LazyWriter@221a6ce4): State: TIMED_WAITING Blocked count: 0 Waited count: 6 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl$LazyWriter.run(FsDatasetImpl.java:3071) java.lang.Thread.run(Thread.java:748) Thread 168 (VolumeScannerThread(/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data4)): State: TIMED_WAITING Blocked count: 1 Waited count: 2 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:622) Thread 167 (VolumeScannerThread(/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data3)): State: TIMED_WAITING Blocked count: 1 Waited count: 2 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:622) Thread 164 (103115058@qtp-229466626-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:41633): State: RUNNABLE Blocked count: 2 Waited count: 1 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498) org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192) org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Thread 163 (498148987@qtp-229466626-0): State: TIMED_WAITING Blocked count: 0 Waited count: 6 Stack: java.lang.Object.wait(Native Method) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626) Thread 162 (pool-12-thread-1): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 156 (IPC Server handler 9 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 372 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 155 (IPC Server handler 8 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 327 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 154 (IPC Server handler 7 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 331 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 153 (IPC Server handler 6 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 373 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 152 (IPC Server handler 5 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 353 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 151 (IPC Server handler 4 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 333 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 150 (IPC Server handler 3 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 368 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 149 (IPC Server handler 2 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 372 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 148 (IPC Server handler 1 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 329 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 147 (IPC Server handler 0 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 326 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 142 (IPC Server listener on 33493): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener.run(Server.java:807) Thread 145 (IPC Server Responder): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.ipc.Server$Responder.doRunLoop(Server.java:982) org.apache.hadoop.ipc.Server$Responder.run(Server.java:965) Thread 102 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@26506cf6): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:100) org.apache.hadoop.hdfs.net.TcpPeerServer.accept(TcpPeerServer.java:141) org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:135) java.lang.Thread.run(Thread.java:748) Thread 146 (DataNode: [[[DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data3/, [DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data4/]] heartbeating to localhost/127.0.0.1:44491): State: TIMED_WAITING Blocked count: 453 Waited count: 1139 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.IncrementalBlockReportManager.waitTillNextIBR(IncrementalBlockReportManager.java:130) org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:542) org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:659) java.lang.Thread.run(Thread.java:748) Thread 144 (IPC Server idle connection scanner for port 33493): State: TIMED_WAITING Blocked count: 1 Waited count: 34 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 143 (Socket Reader #1 for port 33493): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:745) org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:724) Thread 141 (org.apache.hadoop.util.JvmPauseMonitor$Monitor@1d004c0c): State: TIMED_WAITING Blocked count: 0 Waited count: 651 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:182) java.lang.Thread.run(Thread.java:748) Thread 125 (nioEventLoopGroup-4-1): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) java.lang.Thread.run(Thread.java:748) Thread 124 (Timer-2): State: TIMED_WAITING Blocked count: 0 Waited count: 11 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 123 (java.util.concurrent.ThreadPoolExecutor$Worker@3adb50a8[State = -1, empty queue]): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 118 (refreshUsed-/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data1/current/BP-35161775-127.0.1.1-1517713219102): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:132) java.lang.Thread.run(Thread.java:748) Thread 117 (refreshUsed-/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data2/current/BP-35161775-127.0.1.1-1517713219102): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:132) java.lang.Thread.run(Thread.java:748) Thread 112 (org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl$LazyWriter@7e498b6b): State: TIMED_WAITING Blocked count: 0 Waited count: 6 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl$LazyWriter.run(FsDatasetImpl.java:3071) java.lang.Thread.run(Thread.java:748) Thread 111 (VolumeScannerThread(/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data2)): State: TIMED_WAITING Blocked count: 11 Waited count: 2 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:622) Thread 110 (VolumeScannerThread(/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data1)): State: TIMED_WAITING Blocked count: 7 Waited count: 2 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:622) Thread 106 (IPC Client (1670261399) connection to localhost/127.0.0.1:44491 from zhangduo): State: TIMED_WAITING Blocked count: 1063 Waited count: 1064 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:934) org.apache.hadoop.ipc.Client$Connection.run(Client.java:979) Thread 105 (1512126834@qtp-261780187-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:34039): State: RUNNABLE Blocked count: 1 Waited count: 1 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498) org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192) org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Thread 104 (1495196264@qtp-261780187-0): State: TIMED_WAITING Blocked count: 0 Waited count: 6 Stack: java.lang.Object.wait(Native Method) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626) Thread 103 (pool-6-thread-1): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 97 (IPC Server handler 9 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 383 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 96 (IPC Server handler 8 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 373 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 95 (IPC Server handler 7 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 370 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 94 (IPC Server handler 6 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 360 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 93 (IPC Server handler 5 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 373 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 92 (IPC Server handler 4 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 360 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 91 (IPC Server handler 3 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 397 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 90 (IPC Server handler 2 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 378 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 89 (IPC Server handler 1 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 387 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 88 (IPC Server handler 0 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 375 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 83 (IPC Server listener on 45595): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener.run(Server.java:807) Thread 86 (IPC Server Responder): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.ipc.Server$Responder.doRunLoop(Server.java:982) org.apache.hadoop.ipc.Server$Responder.run(Server.java:965) Thread 60 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@5ec0030b): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:100) org.apache.hadoop.hdfs.net.TcpPeerServer.accept(TcpPeerServer.java:141) org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:135) java.lang.Thread.run(Thread.java:748) Thread 87 (DataNode: [[[DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data1/, [DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data2/]] heartbeating to localhost/127.0.0.1:44491): State: TIMED_WAITING Blocked count: 465 Waited count: 1140 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.IncrementalBlockReportManager.waitTillNextIBR(IncrementalBlockReportManager.java:130) org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:542) org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:659) java.lang.Thread.run(Thread.java:748) Thread 85 (IPC Server idle connection scanner for port 45595): State: TIMED_WAITING Blocked count: 1 Waited count: 34 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 84 (Socket Reader #1 for port 45595): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:745) org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:724) Thread 82 (org.apache.hadoop.util.JvmPauseMonitor$Monitor@74095914): State: TIMED_WAITING Blocked count: 0 Waited count: 653 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:182) java.lang.Thread.run(Thread.java:748) Thread 65 (nioEventLoopGroup-2-1): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) java.lang.Thread.run(Thread.java:748) Thread 64 (Timer-1): State: TIMED_WAITING Blocked count: 0 Waited count: 11 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 63 (1020827856@qtp-415074966-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:40851): State: RUNNABLE Blocked count: 1 Waited count: 1 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498) org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192) org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Thread 62 (1498522151@qtp-415074966-0): State: TIMED_WAITING Blocked count: 0 Waited count: 6 Stack: java.lang.Object.wait(Native Method) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626) Thread 61 (pool-4-thread-1): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 55 (CacheReplicationMonitor(1454695942)): State: TIMED_WAITING Blocked count: 0 Waited count: 12 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163) org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor.run(CacheReplicationMonitor.java:181) Thread 54 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$LazyPersistFileScrubber@5654e22b): State: TIMED_WAITING Blocked count: 0 Waited count: 3 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.namenode.FSNamesystem$LazyPersistFileScrubber.run(FSNamesystem.java:4725) java.lang.Thread.run(Thread.java:748) Thread 53 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeEditLogRoller@68702a45): State: TIMED_WAITING Blocked count: 0 Waited count: 2 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeEditLogRoller.run(FSNamesystem.java:4642) java.lang.Thread.run(Thread.java:748) Thread 52 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor@3d921e48): State: TIMED_WAITING Blocked count: 0 Waited count: 66 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor.run(FSNamesystem.java:4598) java.lang.Thread.run(Thread.java:748) Thread 51 (org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@fe27903): State: TIMED_WAITING Blocked count: 0 Waited count: 165 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:431) java.lang.Thread.run(Thread.java:748) Thread 50 (IPC Server handler 9 on 44491): State: TIMED_WAITING Blocked count: 3 Waited count: 532 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 49 (IPC Server handler 8 on 44491): State: TIMED_WAITING Blocked count: 2 Waited count: 519 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 48 (IPC Server handler 7 on 44491): State: TIMED_WAITING Blocked count: 3 Waited count: 528 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 47 (IPC Server handler 6 on 44491): State: TIMED_WAITING Blocked count: 3 Waited count: 535 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 46 (IPC Server handler 5 on 44491): State: TIMED_WAITING Blocked count: 18 Waited count: 530 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 45 (IPC Server handler 4 on 44491): State: TIMED_WAITING Blocked count: 5 Waited count: 528 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 44 (IPC Server handler 3 on 44491): State: TIMED_WAITING Blocked count: 6 Waited count: 520 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 43 (IPC Server handler 2 on 44491): State: TIMED_WAITING Blocked count: 8 Waited count: 525 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 42 (IPC Server handler 1 on 44491): State: TIMED_WAITING Blocked count: 4 Waited count: 530 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 41 (IPC Server handler 0 on 44491): State: TIMED_WAITING Blocked count: 7 Waited count: 530 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 31 (IPC Server listener on 44491): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener.run(Server.java:807) Thread 34 (IPC Server Responder): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.ipc.Server$Responder.doRunLoop(Server.java:982) org.apache.hadoop.ipc.Server$Responder.run(Server.java:965) Thread 29 (Block report processor): State: WAITING Blocked count: 2 Waited count: 140 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@c5f3502 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403) org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.processQueue(BlockManager.java:3854) org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.run(BlockManager.java:3843) Thread 28 (org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@2748be45): State: TIMED_WAITING Blocked count: 0 Waited count: 111 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3638) java.lang.Thread.run(Thread.java:748) Thread 30 (org.apache.hadoop.hdfs.server.blockmanagement.HeartbeatManager$Monitor@1706a704): State: TIMED_WAITING Blocked count: 0 Waited count: 66 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.blockmanagement.HeartbeatManager$Monitor.run(HeartbeatManager.java:381) java.lang.Thread.run(Thread.java:748) Thread 40 (DecommissionMonitor-0): State: TIMED_WAITING Blocked count: 0 Waited count: 110 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 39 (org.apache.hadoop.hdfs.server.blockmanagement.PendingReplicationBlocks$PendingReplicationMonitor@28cb973d): State: TIMED_WAITING Blocked count: 0 Waited count: 2 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.blockmanagement.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:221) java.lang.Thread.run(Thread.java:748) Thread 35 (org.apache.hadoop.util.JvmPauseMonitor$Monitor@7501869b): State: TIMED_WAITING Blocked count: 0 Waited count: 656 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:182) java.lang.Thread.run(Thread.java:748) Thread 33 (IPC Server idle connection scanner for port 44491): State: TIMED_WAITING Blocked count: 1 Waited count: 34 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 32 (Socket Reader #1 for port 44491): State: RUNNABLE Blocked count: 1 Waited count: 1 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:745) org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:724) Thread 27 (Timer-0): State: TIMED_WAITING Blocked count: 1 Waited count: 12 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 25 (1941945463@qtp-2007783481-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:43361): State: RUNNABLE Blocked count: 1 Waited count: 1 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498) org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192) org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Thread 24 (1848959351@qtp-2007783481-0): State: TIMED_WAITING Blocked count: 0 Waited count: 6 Stack: java.lang.Object.wait(Native Method) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626) Thread 23 (pool-2-thread-1): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 14 (org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner): State: WAITING Blocked count: 4 Waited count: 5 Waiting on java.lang.ref.ReferenceQueue$Lock@688d8764 Stack: java.lang.Object.wait(Native Method) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164) org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:3063) java.lang.Thread.run(Thread.java:748) Thread 13 (Time-limited test): State: RUNNABLE Blocked count: 91 Waited count: 231 Stack: sun.management.ThreadImpl.getThreadInfo1(Native Method) sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:178) sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:139) org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:168) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) org.apache.hadoop.hbase.util.Threads$PrintThreadInfoLazyHolder$1.printThreadInfo(Threads.java:294) org.apache.hadoop.hbase.util.Threads.printThreadInfo(Threads.java:341) org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:135) org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:367) org.apache.hadoop.hbase.MiniHBaseCluster.waitUntilShutDown(MiniHBaseCluster.java:865) org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:1130) org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:1105) org.apache.hadoop.hbase.TestFullLogReconstruction.tearDownAfterClass(TestFullLogReconstruction.java:68) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) Thread 9 (ReaderThread): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: java.net.SocketInputStream.socketRead0(Native Method) java.net.SocketInputStream.socketRead(SocketInputStream.java:116) java.net.SocketInputStream.read(SocketInputStream.java:171) java.net.SocketInputStream.read(SocketInputStream.java:141) sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284) sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326) sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178) java.io.InputStreamReader.read(InputStreamReader.java:184) java.io.BufferedReader.fill(BufferedReader.java:161) java.io.BufferedReader.readLine(BufferedReader.java:324) java.io.BufferedReader.readLine(BufferedReader.java:389) org.eclipse.jdt.internal.junit.runner.RemoteTestRunner$ReaderThread.run(RemoteTestRunner.java:148) Thread 4 (Signal Dispatcher): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: Thread 3 (Finalizer): State: WAITING Blocked count: 266 Waited count: 15 Waiting on java.lang.ref.ReferenceQueue$Lock@60cb01b5 Stack: java.lang.Object.wait(Native Method) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164) java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209) Thread 2 (Reference Handler): State: WAITING Blocked count: 21 Waited count: 14 Waiting on java.lang.ref.Reference$Lock@6a276aab Stack: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) java.lang.ref.Reference.tryHandlePending(Reference.java:191) java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153) Thread 1 (main): State: TIMED_WAITING Blocked count: 0 Waited count: 2 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.FutureTask.awaitDone(FutureTask.java:426) java.util.concurrent.FutureTask.get(FutureTask.java:204) org.junit.internal.runners.statements.FailOnTimeout.getResult(FailOnTimeout.java:141) org.junit.internal.runners.statements.FailOnTimeout.evaluate(FailOnTimeout.java:127) org.junit.rules.RunRules.evaluate(RunRules.java:20) org.junit.runners.ParentRunner.run(ParentRunner.java:363) org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86) org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38) org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:538) org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:760) org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:460) org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:206) 2018-02-04 11:06:28,706 DEBUG [ReadOnlyZKClient-localhost:56841@0x143faef3] zookeeper.ReadOnlyZKClient(316): 0x143faef3 to localhost:56841 inactive for 60000ms; closing (Will reconnect when new requests) 2018-02-04 11:06:28,713 DEBUG [ReadOnlyZKClient-localhost:56841@0x46e70b17] zookeeper.ReadOnlyZKClient(316): 0x46e70b17 to localhost:56841 inactive for 60000ms; closing (Will reconnect when new requests) 2018-02-04 11:06:28,715 DEBUG [ReadOnlyZKClient-localhost:56841@0x796736be] zookeeper.ReadOnlyZKClient(316): 0x796736be to localhost:56841 inactive for 60000ms; closing (Will reconnect when new requests) Process Thread Dump: Automatic Stack Trace every 60 seconds waiting on RS:0;ubuntu:39267 204 active threads Thread 1358 (Timer for 'HBase' metrics system): State: TIMED_WAITING Blocked count: 0 Waited count: 8 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 1345 (Attach Listener): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: Thread 1069 (RS_CLOSE_REGION-regionserver/ubuntu:0-2): State: TIMED_WAITING Blocked count: 72 Waited count: 457 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:133) org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:718) org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:605) org.apache.hadoop.hbase.regionserver.wal.WALUtil.doFullAppendTransaction(WALUtil.java:154) org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeMarker(WALUtil.java:132) org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeRegionEventMarker(WALUtil.java:97) org.apache.hadoop.hbase.regionserver.HRegion.writeRegionCloseMarker(HRegion.java:1103) org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1615) org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 1062 (RS_CLOSE_REGION-regionserver/ubuntu:0-1): State: WAITING Blocked count: 54 Waited count: 344 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5fbe8967 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 1061 (RS_CLOSE_REGION-regionserver/ubuntu:0-0): State: TIMED_WAITING Blocked count: 67 Waited count: 463 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:133) org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:718) org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:605) org.apache.hadoop.hbase.regionserver.wal.WALUtil.doFullAppendTransaction(WALUtil.java:154) org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeMarker(WALUtil.java:132) org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeRegionEventMarker(WALUtil.java:97) org.apache.hadoop.hbase.regionserver.HRegion.writeRegionCloseMarker(HRegion.java:1103) org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1615) org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 988 (RS-EventLoopGroup-3-4): State: RUNNABLE Blocked count: 15 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 987 (Default-IPC-NioEventLoopGroup-7-3): State: RUNNABLE Blocked count: 23 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 986 (Default-IPC-NioEventLoopGroup-7-4): State: RUNNABLE Blocked count: 18 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 845 (RS_OPEN_REGION-regionserver/ubuntu:0-2): State: WAITING Blocked count: 50 Waited count: 67 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@72aa087e Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 842 (RS_OPEN_REGION-regionserver/ubuntu:0-1): State: WAITING Blocked count: 42 Waited count: 66 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@72aa087e Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 841 (RS_OPEN_REGION-regionserver/ubuntu:0-0): State: WAITING Blocked count: 42 Waited count: 64 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@72aa087e Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 810 (region-location-4): State: WAITING Blocked count: 5 Waited count: 4 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@282bcdfd Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 809 (region-location-3): State: WAITING Blocked count: 8 Waited count: 4 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@282bcdfd Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 808 (region-location-2): State: WAITING Blocked count: 13 Waited count: 7 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@282bcdfd Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 807 (region-location-1): State: WAITING Blocked count: 11 Waited count: 14 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@282bcdfd Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 571 (RS-EventLoopGroup-1-5): State: RUNNABLE Blocked count: 2 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 570 (Default-IPC-NioEventLoopGroup-7-2): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 568 (RS-EventLoopGroup-4-5): State: RUNNABLE Blocked count: 15 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 567 (Default-IPC-NioEventLoopGroup-7-1): State: RUNNABLE Blocked count: 49 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 560 (RS-EventLoopGroup-3-3): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 553 (RS_OPEN_PRIORITY_REGION-regionserver/ubuntu:0-0): State: WAITING Blocked count: 14 Waited count: 26 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6b2dcd67 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 551 (RS-EventLoopGroup-3-2): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 549 (RS-EventLoopGroup-4-4): State: RUNNABLE Blocked count: 1 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 544 (region-location-0): State: WAITING Blocked count: 12 Waited count: 18 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@282bcdfd Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 521 (RS-EventLoopGroup-4-3): State: RUNNABLE Blocked count: 1 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 508 (RS-EventLoopGroup-4-2): State: RUNNABLE Blocked count: 1 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 504 (AsyncFSWAL-0): State: WAITING Blocked count: 0 Waited count: 169 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1c3dccfe Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 503 (RS-EventLoopGroup-5-8): State: RUNNABLE Blocked count: 7 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 497 (RS-EventLoopGroup-5-7): State: RUNNABLE Blocked count: 1 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 483 (RS-EventLoopGroup-5-6): State: RUNNABLE Blocked count: 3 Waited count: 2 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 480 (LeaseRenewer:zhangduo.hfs.0@localhost:44491): State: TIMED_WAITING Blocked count: 12 Waited count: 404 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:444) org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304) java.lang.Thread.run(Thread.java:748) Thread 482 (RS-EventLoopGroup-5-5): State: RUNNABLE Blocked count: 12 Waited count: 3 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 476 (ubuntu:39267Replication Statistics #0): State: TIMED_WAITING Blocked count: 0 Waited count: 2 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 470 (ReplicationExecutor-0): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@153c8699 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 458 (regionserver/ubuntu:0.procedureResultReporter): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@788f3b3e Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:75) Thread 455 (regionserver/ubuntu:0.logRoller): State: TIMED_WAITING Blocked count: 6 Waited count: 386 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:147) java.lang.Thread.run(Thread.java:748) Thread 450 (regionserver/ubuntu:0.procedureResultReporter): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73ec8f20 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:75) Thread 452 (regionserver/ubuntu:0.procedureResultReporter): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@723b1ca8 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:75) Thread 443 (regionserver/ubuntu:0.Chore.1): State: TIMED_WAITING Blocked count: 1 Waited count: 35 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 431 (JvmPauseMonitor): State: TIMED_WAITING Blocked count: 0 Waited count: 762 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:154) java.lang.Thread.run(Thread.java:748) Thread 423 (ReadOnlyZKClient-localhost:56841@0x796736be): State: TIMED_WAITING Blocked count: 1 Waited count: 9 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:307) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$63/1354899920.run(Unknown Source) java.lang.Thread.run(Thread.java:748) Thread 422 (ReadOnlyZKClient-localhost:56841@0x143faef3): State: TIMED_WAITING Blocked count: 1 Waited count: 9 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:307) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$63/1354899920.run(Unknown Source) java.lang.Thread.run(Thread.java:748) Thread 421 (ReadOnlyZKClient-localhost:56841@0x46e70b17): State: TIMED_WAITING Blocked count: 1 Waited count: 9 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:307) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$63/1354899920.run(Unknown Source) java.lang.Thread.run(Thread.java:748) Thread 419 (IPC Parameter Sending Thread #1): State: TIMED_WAITING Blocked count: 0 Waited count: 1949 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 416 (threadDeathWatcher-6-1): State: TIMED_WAITING Blocked count: 0 Waited count: 382 Stack: java.lang.Thread.sleep(Native Method) org.apache.hbase.thirdparty.io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:152) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 415 (RS-EventLoopGroup-1-4): State: RUNNABLE Blocked count: 29 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 414 (RS-EventLoopGroup-1-3): State: RUNNABLE Blocked count: 18 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 413 (RS-EventLoopGroup-1-2): State: RUNNABLE Blocked count: 16 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 367 (RpcClient-timer-pool1-t1): State: TIMED_WAITING Blocked count: 0 Waited count: 38092 Stack: java.lang.Thread.sleep(Native Method) org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:560) org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:459) java.lang.Thread.run(Thread.java:748) Thread 410 (RS-EventLoopGroup-5-4): State: RUNNABLE Blocked count: 25 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 411 (RS-EventLoopGroup-5-2): State: RUNNABLE Blocked count: 56 Waited count: 3 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 412 (RS-EventLoopGroup-5-3): State: RUNNABLE Blocked count: 31 Waited count: 4 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 407 (snapshot-hfile-cleaner-cache-refresher): State: TIMED_WAITING Blocked count: 1 Waited count: 4 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 400 (M:0;ubuntu:39063-EventThread): State: WAITING Blocked count: 1 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4fcbce78 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501) Thread 399 (M:0;ubuntu:39063-SendThread(localhost:56841)): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141) Thread 289 (RS:0;ubuntu:39267): State: TIMED_WAITING Blocked count: 198 Waited count: 1910 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.regionserver.HRegionServer.sleep(HRegionServer.java:1425) org.apache.hadoop.hbase.regionserver.HRegionServer.waitOnAllRegionsToClose(HRegionServer.java:1411) org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1103) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:187) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$0(MiniHBaseCluster.java:186) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:171) java.security.AccessController.doPrivileged(Native Method) javax.security.auth.Subject.doAs(Subject.java:360) org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726) org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:168) java.lang.Thread.run(Thread.java:748) Thread 368 (Idle-Rpc-Conn-Sweeper-pool2-t1): State: TIMED_WAITING Blocked count: 0 Waited count: 19 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 334 (org.apache.hadoop.hdfs.PeerCache@2e1946df): State: TIMED_WAITING Blocked count: 1 Waited count: 128 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:255) org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:46) org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:124) java.lang.Thread.run(Thread.java:748) Thread 332 (M:0;ubuntu:39063-MemStoreChunkPool Statistics): State: TIMED_WAITING Blocked count: 0 Waited count: 2 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 328 (Monitor thread for TaskMonitor): State: TIMED_WAITING Blocked count: 0 Waited count: 39 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.monitoring.TaskMonitor$MonitorRunnable.run(TaskMonitor.java:302) java.lang.Thread.run(Thread.java:748) Thread 310 (RS-EventLoopGroup-5-1): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 291 (RS-EventLoopGroup-4-1): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 288 (RpcServer.replication.FPBQ.Fifo.handler=2,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@50f24e6c Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 287 (RpcServer.replication.FPBQ.Fifo.handler=1,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@5b366694 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 286 (RpcServer.replication.FPBQ.Fifo.handler=0,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@68239fa7 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 285 (RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=39267): State: WAITING Blocked count: 8 Waited count: 20 Waiting on java.util.concurrent.Semaphore$NonfairSync@64e05a8c Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 284 (RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@26a3f952 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 283 (RpcServer.priority.FPBQ.Fifo.handler=3,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@4ec929d4 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 282 (RpcServer.priority.FPBQ.Fifo.handler=2,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@3ae3490d Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 281 (RpcServer.priority.FPBQ.Fifo.handler=1,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@5cb140e7 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 280 (RpcServer.priority.FPBQ.Fifo.handler=0,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@65614b4a Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 279 (RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39267): State: WAITING Blocked count: 33 Waited count: 56 Waiting on java.util.concurrent.Semaphore$NonfairSync@626159bb Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 278 (RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 2 Waiting on java.util.concurrent.Semaphore$NonfairSync@42eadfdb Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 277 (RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@148a1919 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 276 (RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@7b5e2bd9 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 275 (RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@139a0988 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 272 (RS-EventLoopGroup-3-1): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 248 (RS-EventLoopGroup-1-1): State: RUNNABLE Blocked count: 1 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 247 (HBase-Metrics2-1): State: TIMED_WAITING Blocked count: 0 Waited count: 419 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 235 (ProcessThread(sid:0 cport:56841):): State: WAITING Blocked count: 1 Waited count: 744 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@133caf02 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:122) Thread 234 (SyncThread:0): State: WAITING Blocked count: 0 Waited count: 716 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5c669c88 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:127) Thread 233 (SessionTracker): State: TIMED_WAITING Blocked count: 0 Waited count: 193 Stack: java.lang.Object.wait(Native Method) org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146) Thread 232 (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:56841): State: RUNNABLE Blocked count: 9 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:173) java.lang.Thread.run(Thread.java:748) Thread 229 (java.util.concurrent.ThreadPoolExecutor$Worker@1d726817[State = -1, empty queue]): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 224 (refreshUsed-/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data6/current/BP-35161775-127.0.1.1-1517713219102): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:132) java.lang.Thread.run(Thread.java:748) Thread 222 (refreshUsed-/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data5/current/BP-35161775-127.0.1.1-1517713219102): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:132) java.lang.Thread.run(Thread.java:748) Thread 218 (org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl$LazyWriter@2327f0e2): State: TIMED_WAITING Blocked count: 0 Waited count: 7 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl$LazyWriter.run(FsDatasetImpl.java:3071) java.lang.Thread.run(Thread.java:748) Thread 217 (VolumeScannerThread(/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data6)): State: TIMED_WAITING Blocked count: 1 Waited count: 2 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:622) Thread 216 (VolumeScannerThread(/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data5)): State: TIMED_WAITING Blocked count: 1 Waited count: 2 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:622) Thread 213 (IPC Server handler 9 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 421 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 212 (IPC Server handler 8 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 421 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 211 (IPC Server handler 7 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 406 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 210 (IPC Server handler 6 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 435 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 209 (IPC Server handler 5 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 412 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 208 (IPC Server handler 4 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 437 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 207 (IPC Server handler 3 on 43825): State: TIMED_WAITING Blocked count: 1 Waited count: 389 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 206 (IPC Server handler 2 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 433 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 205 (IPC Server handler 1 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 469 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 204 (IPC Server handler 0 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 387 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 199 (IPC Server listener on 43825): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener.run(Server.java:807) Thread 202 (IPC Server Responder): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.ipc.Server$Responder.doRunLoop(Server.java:982) org.apache.hadoop.ipc.Server$Responder.run(Server.java:965) Thread 161 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@3e65225e): State: RUNNABLE Blocked count: 1 Waited count: 0 Stack: sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:100) org.apache.hadoop.hdfs.net.TcpPeerServer.accept(TcpPeerServer.java:141) org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:135) java.lang.Thread.run(Thread.java:748) Thread 203 (DataNode: [[[DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data5/, [DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data6/]] heartbeating to localhost/127.0.0.1:44491): State: TIMED_WAITING Blocked count: 532 Waited count: 1324 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.IncrementalBlockReportManager.waitTillNextIBR(IncrementalBlockReportManager.java:130) org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:542) org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:659) java.lang.Thread.run(Thread.java:748) Thread 201 (IPC Server idle connection scanner for port 43825): State: TIMED_WAITING Blocked count: 1 Waited count: 40 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 200 (Socket Reader #1 for port 43825): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:745) org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:724) Thread 198 (org.apache.hadoop.util.JvmPauseMonitor$Monitor@3369a0f5): State: TIMED_WAITING Blocked count: 0 Waited count: 770 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:182) java.lang.Thread.run(Thread.java:748) Thread 182 (nioEventLoopGroup-6-1): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) java.lang.Thread.run(Thread.java:748) Thread 181 (Timer-3): State: TIMED_WAITING Blocked count: 0 Waited count: 13 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 180 (java.util.concurrent.ThreadPoolExecutor$Worker@4231ed0c[State = -1, empty queue]): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 175 (refreshUsed-/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data4/current/BP-35161775-127.0.1.1-1517713219102): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:132) java.lang.Thread.run(Thread.java:748) Thread 173 (refreshUsed-/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data3/current/BP-35161775-127.0.1.1-1517713219102): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:132) java.lang.Thread.run(Thread.java:748) Thread 169 (org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl$LazyWriter@221a6ce4): State: TIMED_WAITING Blocked count: 0 Waited count: 7 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl$LazyWriter.run(FsDatasetImpl.java:3071) java.lang.Thread.run(Thread.java:748) Thread 168 (VolumeScannerThread(/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data4)): State: TIMED_WAITING Blocked count: 1 Waited count: 2 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:622) Thread 167 (VolumeScannerThread(/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data3)): State: TIMED_WAITING Blocked count: 1 Waited count: 2 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:622) Thread 164 (103115058@qtp-229466626-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:41633): State: RUNNABLE Blocked count: 2 Waited count: 1 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498) org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192) org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Thread 163 (498148987@qtp-229466626-0): State: TIMED_WAITING Blocked count: 0 Waited count: 7 Stack: java.lang.Object.wait(Native Method) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626) Thread 162 (pool-12-thread-1): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 156 (IPC Server handler 9 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 432 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 155 (IPC Server handler 8 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 387 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 154 (IPC Server handler 7 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 391 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 153 (IPC Server handler 6 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 433 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 152 (IPC Server handler 5 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 417 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 151 (IPC Server handler 4 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 393 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 150 (IPC Server handler 3 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 434 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 149 (IPC Server handler 2 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 432 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 148 (IPC Server handler 1 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 389 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 147 (IPC Server handler 0 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 387 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 142 (IPC Server listener on 33493): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener.run(Server.java:807) Thread 145 (IPC Server Responder): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.ipc.Server$Responder.doRunLoop(Server.java:982) org.apache.hadoop.ipc.Server$Responder.run(Server.java:965) Thread 102 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@26506cf6): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:100) org.apache.hadoop.hdfs.net.TcpPeerServer.accept(TcpPeerServer.java:141) org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:135) java.lang.Thread.run(Thread.java:748) Thread 146 (DataNode: [[[DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data3/, [DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data4/]] heartbeating to localhost/127.0.0.1:44491): State: TIMED_WAITING Blocked count: 513 Waited count: 1319 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.IncrementalBlockReportManager.waitTillNextIBR(IncrementalBlockReportManager.java:130) org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:542) org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:659) java.lang.Thread.run(Thread.java:748) Thread 144 (IPC Server idle connection scanner for port 33493): State: TIMED_WAITING Blocked count: 1 Waited count: 40 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 143 (Socket Reader #1 for port 33493): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:745) org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:724) Thread 141 (org.apache.hadoop.util.JvmPauseMonitor$Monitor@1d004c0c): State: TIMED_WAITING Blocked count: 0 Waited count: 772 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:182) java.lang.Thread.run(Thread.java:748) Thread 125 (nioEventLoopGroup-4-1): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) java.lang.Thread.run(Thread.java:748) Thread 124 (Timer-2): State: TIMED_WAITING Blocked count: 0 Waited count: 13 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 123 (java.util.concurrent.ThreadPoolExecutor$Worker@3adb50a8[State = -1, empty queue]): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 118 (refreshUsed-/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data1/current/BP-35161775-127.0.1.1-1517713219102): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:132) java.lang.Thread.run(Thread.java:748) Thread 117 (refreshUsed-/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data2/current/BP-35161775-127.0.1.1-1517713219102): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:132) java.lang.Thread.run(Thread.java:748) Thread 112 (org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl$LazyWriter@7e498b6b): State: TIMED_WAITING Blocked count: 0 Waited count: 7 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl$LazyWriter.run(FsDatasetImpl.java:3071) java.lang.Thread.run(Thread.java:748) Thread 111 (VolumeScannerThread(/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data2)): State: TIMED_WAITING Blocked count: 11 Waited count: 2 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:622) Thread 110 (VolumeScannerThread(/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data1)): State: TIMED_WAITING Blocked count: 7 Waited count: 2 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:622) Thread 106 (IPC Client (1670261399) connection to localhost/127.0.0.1:44491 from zhangduo): State: TIMED_WAITING Blocked count: 1244 Waited count: 1245 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:934) org.apache.hadoop.ipc.Client$Connection.run(Client.java:979) Thread 105 (1512126834@qtp-261780187-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:34039): State: RUNNABLE Blocked count: 1 Waited count: 1 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498) org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192) org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Thread 104 (1495196264@qtp-261780187-0): State: TIMED_WAITING Blocked count: 0 Waited count: 7 Stack: java.lang.Object.wait(Native Method) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626) Thread 103 (pool-6-thread-1): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 97 (IPC Server handler 9 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 455 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 96 (IPC Server handler 8 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 440 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 95 (IPC Server handler 7 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 437 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 94 (IPC Server handler 6 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 430 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 93 (IPC Server handler 5 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 434 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 92 (IPC Server handler 4 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 420 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 91 (IPC Server handler 3 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 457 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 90 (IPC Server handler 2 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 442 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 89 (IPC Server handler 1 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 459 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 88 (IPC Server handler 0 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 442 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 83 (IPC Server listener on 45595): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener.run(Server.java:807) Thread 86 (IPC Server Responder): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.ipc.Server$Responder.doRunLoop(Server.java:982) org.apache.hadoop.ipc.Server$Responder.run(Server.java:965) Thread 60 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@5ec0030b): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:100) org.apache.hadoop.hdfs.net.TcpPeerServer.accept(TcpPeerServer.java:141) org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:135) java.lang.Thread.run(Thread.java:748) Thread 87 (DataNode: [[[DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data1/, [DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data2/]] heartbeating to localhost/127.0.0.1:44491): State: TIMED_WAITING Blocked count: 526 Waited count: 1323 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.IncrementalBlockReportManager.waitTillNextIBR(IncrementalBlockReportManager.java:130) org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:542) org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:659) java.lang.Thread.run(Thread.java:748) Thread 85 (IPC Server idle connection scanner for port 45595): State: TIMED_WAITING Blocked count: 1 Waited count: 40 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 84 (Socket Reader #1 for port 45595): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:745) org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:724) Thread 82 (org.apache.hadoop.util.JvmPauseMonitor$Monitor@74095914): State: TIMED_WAITING Blocked count: 0 Waited count: 774 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:182) java.lang.Thread.run(Thread.java:748) Thread 65 (nioEventLoopGroup-2-1): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) java.lang.Thread.run(Thread.java:748) Thread 64 (Timer-1): State: TIMED_WAITING Blocked count: 0 Waited count: 13 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 63 (1020827856@qtp-415074966-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:40851): State: RUNNABLE Blocked count: 1 Waited count: 1 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498) org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192) org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Thread 62 (1498522151@qtp-415074966-0): State: TIMED_WAITING Blocked count: 0 Waited count: 7 Stack: java.lang.Object.wait(Native Method) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626) Thread 61 (pool-4-thread-1): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 55 (CacheReplicationMonitor(1454695942)): State: TIMED_WAITING Blocked count: 0 Waited count: 14 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163) org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor.run(CacheReplicationMonitor.java:181) Thread 54 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$LazyPersistFileScrubber@5654e22b): State: TIMED_WAITING Blocked count: 0 Waited count: 3 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.namenode.FSNamesystem$LazyPersistFileScrubber.run(FSNamesystem.java:4725) java.lang.Thread.run(Thread.java:748) Thread 53 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeEditLogRoller@68702a45): State: TIMED_WAITING Blocked count: 0 Waited count: 2 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeEditLogRoller.run(FSNamesystem.java:4642) java.lang.Thread.run(Thread.java:748) Thread 52 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor@3d921e48): State: TIMED_WAITING Blocked count: 0 Waited count: 78 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor.run(FSNamesystem.java:4598) java.lang.Thread.run(Thread.java:748) Thread 51 (org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@fe27903): State: TIMED_WAITING Blocked count: 0 Waited count: 196 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:431) java.lang.Thread.run(Thread.java:748) Thread 50 (IPC Server handler 9 on 44491): State: TIMED_WAITING Blocked count: 3 Waited count: 594 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 49 (IPC Server handler 8 on 44491): State: TIMED_WAITING Blocked count: 2 Waited count: 581 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 48 (IPC Server handler 7 on 44491): State: TIMED_WAITING Blocked count: 3 Waited count: 592 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 47 (IPC Server handler 6 on 44491): State: TIMED_WAITING Blocked count: 3 Waited count: 598 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 46 (IPC Server handler 5 on 44491): State: TIMED_WAITING Blocked count: 18 Waited count: 592 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 45 (IPC Server handler 4 on 44491): State: TIMED_WAITING Blocked count: 5 Waited count: 597 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 44 (IPC Server handler 3 on 44491): State: TIMED_WAITING Blocked count: 6 Waited count: 586 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 43 (IPC Server handler 2 on 44491): State: TIMED_WAITING Blocked count: 8 Waited count: 588 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 42 (IPC Server handler 1 on 44491): State: TIMED_WAITING Blocked count: 4 Waited count: 596 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 41 (IPC Server handler 0 on 44491): State: TIMED_WAITING Blocked count: 7 Waited count: 593 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 31 (IPC Server listener on 44491): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener.run(Server.java:807) Thread 34 (IPC Server Responder): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.ipc.Server$Responder.doRunLoop(Server.java:982) org.apache.hadoop.ipc.Server$Responder.run(Server.java:965) Thread 29 (Block report processor): State: WAITING Blocked count: 2 Waited count: 140 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@c5f3502 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403) org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.processQueue(BlockManager.java:3854) org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.run(BlockManager.java:3843) Thread 28 (org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@2748be45): State: TIMED_WAITING Blocked count: 0 Waited count: 131 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3638) java.lang.Thread.run(Thread.java:748) Thread 30 (org.apache.hadoop.hdfs.server.blockmanagement.HeartbeatManager$Monitor@1706a704): State: TIMED_WAITING Blocked count: 0 Waited count: 78 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.blockmanagement.HeartbeatManager$Monitor.run(HeartbeatManager.java:381) java.lang.Thread.run(Thread.java:748) Thread 40 (DecommissionMonitor-0): State: TIMED_WAITING Blocked count: 0 Waited count: 130 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 39 (org.apache.hadoop.hdfs.server.blockmanagement.PendingReplicationBlocks$PendingReplicationMonitor@28cb973d): State: TIMED_WAITING Blocked count: 0 Waited count: 2 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.blockmanagement.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:221) java.lang.Thread.run(Thread.java:748) Thread 35 (org.apache.hadoop.util.JvmPauseMonitor$Monitor@7501869b): State: TIMED_WAITING Blocked count: 0 Waited count: 776 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:182) java.lang.Thread.run(Thread.java:748) Thread 33 (IPC Server idle connection scanner for port 44491): State: TIMED_WAITING Blocked count: 1 Waited count: 40 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 32 (Socket Reader #1 for port 44491): State: RUNNABLE Blocked count: 1 Waited count: 1 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:745) org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:724) Thread 27 (Timer-0): State: TIMED_WAITING Blocked count: 1 Waited count: 14 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 25 (1941945463@qtp-2007783481-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:43361): State: RUNNABLE Blocked count: 1 Waited count: 1 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498) org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192) org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Thread 24 (1848959351@qtp-2007783481-0): State: TIMED_WAITING Blocked count: 0 Waited count: 7 Stack: java.lang.Object.wait(Native Method) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626) Thread 23 (pool-2-thread-1): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 14 (org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner): State: WAITING Blocked count: 4 Waited count: 5 Waiting on java.lang.ref.ReferenceQueue$Lock@688d8764 Stack: java.lang.Object.wait(Native Method) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164) org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:3063) java.lang.Thread.run(Thread.java:748) Thread 13 (Time-limited test): State: RUNNABLE Blocked count: 91 Waited count: 232 Stack: sun.management.ThreadImpl.getThreadInfo1(Native Method) sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:178) sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:139) org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:168) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) org.apache.hadoop.hbase.util.Threads$PrintThreadInfoLazyHolder$1.printThreadInfo(Threads.java:294) org.apache.hadoop.hbase.util.Threads.printThreadInfo(Threads.java:341) org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:135) org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:367) org.apache.hadoop.hbase.MiniHBaseCluster.waitUntilShutDown(MiniHBaseCluster.java:865) org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:1130) org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:1105) org.apache.hadoop.hbase.TestFullLogReconstruction.tearDownAfterClass(TestFullLogReconstruction.java:68) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) Thread 9 (ReaderThread): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: java.net.SocketInputStream.socketRead0(Native Method) java.net.SocketInputStream.socketRead(SocketInputStream.java:116) java.net.SocketInputStream.read(SocketInputStream.java:171) java.net.SocketInputStream.read(SocketInputStream.java:141) sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284) sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326) sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178) java.io.InputStreamReader.read(InputStreamReader.java:184) java.io.BufferedReader.fill(BufferedReader.java:161) java.io.BufferedReader.readLine(BufferedReader.java:324) java.io.BufferedReader.readLine(BufferedReader.java:389) org.eclipse.jdt.internal.junit.runner.RemoteTestRunner$ReaderThread.run(RemoteTestRunner.java:148) Thread 4 (Signal Dispatcher): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: Thread 3 (Finalizer): State: WAITING Blocked count: 266 Waited count: 15 Waiting on java.lang.ref.ReferenceQueue$Lock@60cb01b5 Stack: java.lang.Object.wait(Native Method) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164) java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209) Thread 2 (Reference Handler): State: WAITING Blocked count: 21 Waited count: 14 Waiting on java.lang.ref.Reference$Lock@6a276aab Stack: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) java.lang.ref.Reference.tryHandlePending(Reference.java:191) java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153) Thread 1 (main): State: TIMED_WAITING Blocked count: 0 Waited count: 2 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.FutureTask.awaitDone(FutureTask.java:426) java.util.concurrent.FutureTask.get(FutureTask.java:204) org.junit.internal.runners.statements.FailOnTimeout.getResult(FailOnTimeout.java:141) org.junit.internal.runners.statements.FailOnTimeout.evaluate(FailOnTimeout.java:127) org.junit.rules.RunRules.evaluate(RunRules.java:20) org.junit.runners.ParentRunner.run(ParentRunner.java:363) org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86) org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38) org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:538) org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:760) org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:460) org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:206) 2018-02-04 11:07:28,707 DEBUG [ReadOnlyZKClient-localhost:56841@0x143faef3] zookeeper.ReadOnlyZKClient(316): 0x143faef3 to localhost:56841 inactive for 60000ms; closing (Will reconnect when new requests) 2018-02-04 11:07:28,713 DEBUG [ReadOnlyZKClient-localhost:56841@0x46e70b17] zookeeper.ReadOnlyZKClient(316): 0x46e70b17 to localhost:56841 inactive for 60000ms; closing (Will reconnect when new requests) 2018-02-04 11:07:28,715 DEBUG [ReadOnlyZKClient-localhost:56841@0x796736be] zookeeper.ReadOnlyZKClient(316): 0x796736be to localhost:56841 inactive for 60000ms; closing (Will reconnect when new requests) Process Thread Dump: Automatic Stack Trace every 60 seconds waiting on RS:0;ubuntu:39267 204 active threads Thread 1358 (Timer for 'HBase' metrics system): State: TIMED_WAITING Blocked count: 0 Waited count: 14 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 1345 (Attach Listener): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: Thread 1069 (RS_CLOSE_REGION-regionserver/ubuntu:0-2): State: TIMED_WAITING Blocked count: 72 Waited count: 517 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:133) org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:718) org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:605) org.apache.hadoop.hbase.regionserver.wal.WALUtil.doFullAppendTransaction(WALUtil.java:154) org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeMarker(WALUtil.java:132) org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeRegionEventMarker(WALUtil.java:97) org.apache.hadoop.hbase.regionserver.HRegion.writeRegionCloseMarker(HRegion.java:1103) org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1615) org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 1062 (RS_CLOSE_REGION-regionserver/ubuntu:0-1): State: WAITING Blocked count: 54 Waited count: 344 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5fbe8967 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 1061 (RS_CLOSE_REGION-regionserver/ubuntu:0-0): State: TIMED_WAITING Blocked count: 67 Waited count: 523 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:133) org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:718) org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:605) org.apache.hadoop.hbase.regionserver.wal.WALUtil.doFullAppendTransaction(WALUtil.java:154) org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeMarker(WALUtil.java:132) org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeRegionEventMarker(WALUtil.java:97) org.apache.hadoop.hbase.regionserver.HRegion.writeRegionCloseMarker(HRegion.java:1103) org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1615) org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 988 (RS-EventLoopGroup-3-4): State: RUNNABLE Blocked count: 15 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 987 (Default-IPC-NioEventLoopGroup-7-3): State: RUNNABLE Blocked count: 23 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 986 (Default-IPC-NioEventLoopGroup-7-4): State: RUNNABLE Blocked count: 18 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 845 (RS_OPEN_REGION-regionserver/ubuntu:0-2): State: WAITING Blocked count: 50 Waited count: 67 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@72aa087e Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 842 (RS_OPEN_REGION-regionserver/ubuntu:0-1): State: WAITING Blocked count: 42 Waited count: 66 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@72aa087e Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 841 (RS_OPEN_REGION-regionserver/ubuntu:0-0): State: WAITING Blocked count: 42 Waited count: 64 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@72aa087e Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 810 (region-location-4): State: WAITING Blocked count: 5 Waited count: 4 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@282bcdfd Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 809 (region-location-3): State: WAITING Blocked count: 8 Waited count: 4 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@282bcdfd Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 808 (region-location-2): State: WAITING Blocked count: 13 Waited count: 7 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@282bcdfd Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 807 (region-location-1): State: WAITING Blocked count: 11 Waited count: 14 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@282bcdfd Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 571 (RS-EventLoopGroup-1-5): State: RUNNABLE Blocked count: 2 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 570 (Default-IPC-NioEventLoopGroup-7-2): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 568 (RS-EventLoopGroup-4-5): State: RUNNABLE Blocked count: 15 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 567 (Default-IPC-NioEventLoopGroup-7-1): State: RUNNABLE Blocked count: 49 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 560 (RS-EventLoopGroup-3-3): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 553 (RS_OPEN_PRIORITY_REGION-regionserver/ubuntu:0-0): State: WAITING Blocked count: 14 Waited count: 26 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6b2dcd67 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 551 (RS-EventLoopGroup-3-2): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 549 (RS-EventLoopGroup-4-4): State: RUNNABLE Blocked count: 1 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 544 (region-location-0): State: WAITING Blocked count: 12 Waited count: 18 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@282bcdfd Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 521 (RS-EventLoopGroup-4-3): State: RUNNABLE Blocked count: 1 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 508 (RS-EventLoopGroup-4-2): State: RUNNABLE Blocked count: 1 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 504 (AsyncFSWAL-0): State: WAITING Blocked count: 0 Waited count: 169 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1c3dccfe Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 503 (RS-EventLoopGroup-5-8): State: RUNNABLE Blocked count: 7 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 497 (RS-EventLoopGroup-5-7): State: RUNNABLE Blocked count: 1 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 483 (RS-EventLoopGroup-5-6): State: RUNNABLE Blocked count: 3 Waited count: 2 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 480 (LeaseRenewer:zhangduo.hfs.0@localhost:44491): State: TIMED_WAITING Blocked count: 14 Waited count: 468 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:444) org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304) java.lang.Thread.run(Thread.java:748) Thread 482 (RS-EventLoopGroup-5-5): State: RUNNABLE Blocked count: 12 Waited count: 3 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 476 (ubuntu:39267Replication Statistics #0): State: TIMED_WAITING Blocked count: 0 Waited count: 2 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 470 (ReplicationExecutor-0): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@153c8699 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 458 (regionserver/ubuntu:0.procedureResultReporter): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@788f3b3e Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:75) Thread 455 (regionserver/ubuntu:0.logRoller): State: TIMED_WAITING Blocked count: 6 Waited count: 446 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:147) java.lang.Thread.run(Thread.java:748) Thread 450 (regionserver/ubuntu:0.procedureResultReporter): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73ec8f20 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:75) Thread 452 (regionserver/ubuntu:0.procedureResultReporter): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@723b1ca8 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:75) Thread 443 (regionserver/ubuntu:0.Chore.1): State: TIMED_WAITING Blocked count: 1 Waited count: 36 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 431 (JvmPauseMonitor): State: TIMED_WAITING Blocked count: 0 Waited count: 882 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:154) java.lang.Thread.run(Thread.java:748) Thread 423 (ReadOnlyZKClient-localhost:56841@0x796736be): State: TIMED_WAITING Blocked count: 1 Waited count: 10 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:307) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$63/1354899920.run(Unknown Source) java.lang.Thread.run(Thread.java:748) Thread 422 (ReadOnlyZKClient-localhost:56841@0x143faef3): State: TIMED_WAITING Blocked count: 1 Waited count: 10 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:307) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$63/1354899920.run(Unknown Source) java.lang.Thread.run(Thread.java:748) Thread 421 (ReadOnlyZKClient-localhost:56841@0x46e70b17): State: TIMED_WAITING Blocked count: 1 Waited count: 10 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:307) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$63/1354899920.run(Unknown Source) java.lang.Thread.run(Thread.java:748) Thread 419 (IPC Parameter Sending Thread #1): State: TIMED_WAITING Blocked count: 0 Waited count: 2132 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 416 (threadDeathWatcher-6-1): State: TIMED_WAITING Blocked count: 0 Waited count: 442 Stack: java.lang.Thread.sleep(Native Method) org.apache.hbase.thirdparty.io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:152) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 415 (RS-EventLoopGroup-1-4): State: RUNNABLE Blocked count: 29 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 414 (RS-EventLoopGroup-1-3): State: RUNNABLE Blocked count: 18 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 413 (RS-EventLoopGroup-1-2): State: RUNNABLE Blocked count: 16 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 367 (RpcClient-timer-pool1-t1): State: TIMED_WAITING Blocked count: 0 Waited count: 44109 Stack: java.lang.Thread.sleep(Native Method) org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:560) org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:459) java.lang.Thread.run(Thread.java:748) Thread 410 (RS-EventLoopGroup-5-4): State: RUNNABLE Blocked count: 25 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 411 (RS-EventLoopGroup-5-2): State: RUNNABLE Blocked count: 56 Waited count: 3 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 412 (RS-EventLoopGroup-5-3): State: RUNNABLE Blocked count: 31 Waited count: 4 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 407 (snapshot-hfile-cleaner-cache-refresher): State: TIMED_WAITING Blocked count: 1 Waited count: 4 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 400 (M:0;ubuntu:39063-EventThread): State: WAITING Blocked count: 1 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4fcbce78 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501) Thread 399 (M:0;ubuntu:39063-SendThread(localhost:56841)): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141) Thread 289 (RS:0;ubuntu:39267): State: TIMED_WAITING Blocked count: 198 Waited count: 2210 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.regionserver.HRegionServer.sleep(HRegionServer.java:1425) org.apache.hadoop.hbase.regionserver.HRegionServer.waitOnAllRegionsToClose(HRegionServer.java:1411) org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1103) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:187) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$0(MiniHBaseCluster.java:186) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:171) java.security.AccessController.doPrivileged(Native Method) javax.security.auth.Subject.doAs(Subject.java:360) org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726) org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:168) java.lang.Thread.run(Thread.java:748) Thread 368 (Idle-Rpc-Conn-Sweeper-pool2-t1): State: TIMED_WAITING Blocked count: 0 Waited count: 19 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 334 (org.apache.hadoop.hdfs.PeerCache@2e1946df): State: TIMED_WAITING Blocked count: 1 Waited count: 148 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:255) org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:46) org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:124) java.lang.Thread.run(Thread.java:748) Thread 332 (M:0;ubuntu:39063-MemStoreChunkPool Statistics): State: TIMED_WAITING Blocked count: 0 Waited count: 2 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 328 (Monitor thread for TaskMonitor): State: TIMED_WAITING Blocked count: 0 Waited count: 45 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.monitoring.TaskMonitor$MonitorRunnable.run(TaskMonitor.java:302) java.lang.Thread.run(Thread.java:748) Thread 310 (RS-EventLoopGroup-5-1): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 291 (RS-EventLoopGroup-4-1): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 288 (RpcServer.replication.FPBQ.Fifo.handler=2,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@50f24e6c Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 287 (RpcServer.replication.FPBQ.Fifo.handler=1,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@5b366694 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 286 (RpcServer.replication.FPBQ.Fifo.handler=0,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@68239fa7 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 285 (RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=39267): State: WAITING Blocked count: 8 Waited count: 20 Waiting on java.util.concurrent.Semaphore$NonfairSync@64e05a8c Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 284 (RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@26a3f952 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 283 (RpcServer.priority.FPBQ.Fifo.handler=3,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@4ec929d4 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 282 (RpcServer.priority.FPBQ.Fifo.handler=2,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@3ae3490d Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 281 (RpcServer.priority.FPBQ.Fifo.handler=1,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@5cb140e7 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 280 (RpcServer.priority.FPBQ.Fifo.handler=0,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@65614b4a Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 279 (RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39267): State: WAITING Blocked count: 33 Waited count: 56 Waiting on java.util.concurrent.Semaphore$NonfairSync@626159bb Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 278 (RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 2 Waiting on java.util.concurrent.Semaphore$NonfairSync@42eadfdb Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 277 (RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@148a1919 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 276 (RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@7b5e2bd9 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 275 (RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@139a0988 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 272 (RS-EventLoopGroup-3-1): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 248 (RS-EventLoopGroup-1-1): State: RUNNABLE Blocked count: 1 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 247 (HBase-Metrics2-1): State: TIMED_WAITING Blocked count: 0 Waited count: 479 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 235 (ProcessThread(sid:0 cport:56841):): State: WAITING Blocked count: 1 Waited count: 789 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@133caf02 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:122) Thread 234 (SyncThread:0): State: WAITING Blocked count: 0 Waited count: 761 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5c669c88 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:127) Thread 233 (SessionTracker): State: TIMED_WAITING Blocked count: 0 Waited count: 224 Stack: java.lang.Object.wait(Native Method) org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146) Thread 232 (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:56841): State: RUNNABLE Blocked count: 9 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:173) java.lang.Thread.run(Thread.java:748) Thread 229 (java.util.concurrent.ThreadPoolExecutor$Worker@1d726817[State = -1, empty queue]): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 224 (refreshUsed-/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data6/current/BP-35161775-127.0.1.1-1517713219102): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:132) java.lang.Thread.run(Thread.java:748) Thread 222 (refreshUsed-/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data5/current/BP-35161775-127.0.1.1-1517713219102): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:132) java.lang.Thread.run(Thread.java:748) Thread 218 (org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl$LazyWriter@2327f0e2): State: TIMED_WAITING Blocked count: 0 Waited count: 8 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl$LazyWriter.run(FsDatasetImpl.java:3071) java.lang.Thread.run(Thread.java:748) Thread 217 (VolumeScannerThread(/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data6)): State: TIMED_WAITING Blocked count: 1 Waited count: 2 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:622) Thread 216 (VolumeScannerThread(/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data5)): State: TIMED_WAITING Blocked count: 1 Waited count: 2 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:622) Thread 213 (IPC Server handler 9 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 485 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 212 (IPC Server handler 8 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 489 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 211 (IPC Server handler 7 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 466 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 210 (IPC Server handler 6 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 503 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 209 (IPC Server handler 5 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 475 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 208 (IPC Server handler 4 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 498 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 207 (IPC Server handler 3 on 43825): State: TIMED_WAITING Blocked count: 1 Waited count: 450 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 206 (IPC Server handler 2 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 496 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 205 (IPC Server handler 1 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 537 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 204 (IPC Server handler 0 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 448 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 199 (IPC Server listener on 43825): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener.run(Server.java:807) Thread 202 (IPC Server Responder): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.ipc.Server$Responder.doRunLoop(Server.java:982) org.apache.hadoop.ipc.Server$Responder.run(Server.java:965) Thread 161 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@3e65225e): State: RUNNABLE Blocked count: 1 Waited count: 0 Stack: sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:100) org.apache.hadoop.hdfs.net.TcpPeerServer.accept(TcpPeerServer.java:141) org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:135) java.lang.Thread.run(Thread.java:748) Thread 203 (DataNode: [[[DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data5/, [DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data6/]] heartbeating to localhost/127.0.0.1:44491): State: TIMED_WAITING Blocked count: 592 Waited count: 1504 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.IncrementalBlockReportManager.waitTillNextIBR(IncrementalBlockReportManager.java:130) org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:542) org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:659) java.lang.Thread.run(Thread.java:748) Thread 201 (IPC Server idle connection scanner for port 43825): State: TIMED_WAITING Blocked count: 1 Waited count: 46 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 200 (Socket Reader #1 for port 43825): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:745) org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:724) Thread 198 (org.apache.hadoop.util.JvmPauseMonitor$Monitor@3369a0f5): State: TIMED_WAITING Blocked count: 0 Waited count: 890 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:182) java.lang.Thread.run(Thread.java:748) Thread 182 (nioEventLoopGroup-6-1): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) java.lang.Thread.run(Thread.java:748) Thread 181 (Timer-3): State: TIMED_WAITING Blocked count: 0 Waited count: 15 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 180 (java.util.concurrent.ThreadPoolExecutor$Worker@4231ed0c[State = -1, empty queue]): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 175 (refreshUsed-/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data4/current/BP-35161775-127.0.1.1-1517713219102): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:132) java.lang.Thread.run(Thread.java:748) Thread 173 (refreshUsed-/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data3/current/BP-35161775-127.0.1.1-1517713219102): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:132) java.lang.Thread.run(Thread.java:748) Thread 169 (org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl$LazyWriter@221a6ce4): State: TIMED_WAITING Blocked count: 0 Waited count: 8 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl$LazyWriter.run(FsDatasetImpl.java:3071) java.lang.Thread.run(Thread.java:748) Thread 168 (VolumeScannerThread(/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data4)): State: TIMED_WAITING Blocked count: 1 Waited count: 2 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:622) Thread 167 (VolumeScannerThread(/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data3)): State: TIMED_WAITING Blocked count: 1 Waited count: 2 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:622) Thread 164 (103115058@qtp-229466626-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:41633): State: RUNNABLE Blocked count: 2 Waited count: 1 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498) org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192) org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Thread 163 (498148987@qtp-229466626-0): State: TIMED_WAITING Blocked count: 0 Waited count: 8 Stack: java.lang.Object.wait(Native Method) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626) Thread 162 (pool-12-thread-1): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 156 (IPC Server handler 9 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 492 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 155 (IPC Server handler 8 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 447 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 154 (IPC Server handler 7 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 451 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 153 (IPC Server handler 6 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 496 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 152 (IPC Server handler 5 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 479 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 151 (IPC Server handler 4 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 457 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 150 (IPC Server handler 3 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 498 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 149 (IPC Server handler 2 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 493 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 148 (IPC Server handler 1 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 450 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 147 (IPC Server handler 0 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 447 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 142 (IPC Server listener on 33493): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener.run(Server.java:807) Thread 145 (IPC Server Responder): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.ipc.Server$Responder.doRunLoop(Server.java:982) org.apache.hadoop.ipc.Server$Responder.run(Server.java:965) Thread 102 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@26506cf6): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:100) org.apache.hadoop.hdfs.net.TcpPeerServer.accept(TcpPeerServer.java:141) org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:135) java.lang.Thread.run(Thread.java:748) Thread 146 (DataNode: [[[DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data3/, [DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data4/]] heartbeating to localhost/127.0.0.1:44491): State: TIMED_WAITING Blocked count: 573 Waited count: 1499 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.IncrementalBlockReportManager.waitTillNextIBR(IncrementalBlockReportManager.java:130) org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:542) org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:659) java.lang.Thread.run(Thread.java:748) Thread 144 (IPC Server idle connection scanner for port 33493): State: TIMED_WAITING Blocked count: 1 Waited count: 46 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 143 (Socket Reader #1 for port 33493): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:745) org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:724) Thread 141 (org.apache.hadoop.util.JvmPauseMonitor$Monitor@1d004c0c): State: TIMED_WAITING Blocked count: 0 Waited count: 892 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:182) java.lang.Thread.run(Thread.java:748) Thread 125 (nioEventLoopGroup-4-1): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) java.lang.Thread.run(Thread.java:748) Thread 124 (Timer-2): State: TIMED_WAITING Blocked count: 0 Waited count: 15 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 123 (java.util.concurrent.ThreadPoolExecutor$Worker@3adb50a8[State = -1, empty queue]): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 118 (refreshUsed-/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data1/current/BP-35161775-127.0.1.1-1517713219102): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:132) java.lang.Thread.run(Thread.java:748) Thread 117 (refreshUsed-/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data2/current/BP-35161775-127.0.1.1-1517713219102): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:132) java.lang.Thread.run(Thread.java:748) Thread 112 (org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl$LazyWriter@7e498b6b): State: TIMED_WAITING Blocked count: 0 Waited count: 8 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl$LazyWriter.run(FsDatasetImpl.java:3071) java.lang.Thread.run(Thread.java:748) Thread 111 (VolumeScannerThread(/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data2)): State: TIMED_WAITING Blocked count: 11 Waited count: 2 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:622) Thread 110 (VolumeScannerThread(/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data1)): State: TIMED_WAITING Blocked count: 7 Waited count: 2 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:622) Thread 106 (IPC Client (1670261399) connection to localhost/127.0.0.1:44491 from zhangduo): State: TIMED_WAITING Blocked count: 1424 Waited count: 1425 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:934) org.apache.hadoop.ipc.Client$Connection.run(Client.java:979) Thread 105 (1512126834@qtp-261780187-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:34039): State: RUNNABLE Blocked count: 1 Waited count: 1 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498) org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192) org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Thread 104 (1495196264@qtp-261780187-0): State: TIMED_WAITING Blocked count: 0 Waited count: 8 Stack: java.lang.Object.wait(Native Method) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626) Thread 103 (pool-6-thread-1): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 97 (IPC Server handler 9 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 541 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 96 (IPC Server handler 8 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 500 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 95 (IPC Server handler 7 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 525 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 94 (IPC Server handler 6 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 510 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 93 (IPC Server handler 5 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 511 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 92 (IPC Server handler 4 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 485 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 91 (IPC Server handler 3 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 524 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 90 (IPC Server handler 2 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 522 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 89 (IPC Server handler 1 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 535 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 88 (IPC Server handler 0 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 530 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 83 (IPC Server listener on 45595): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener.run(Server.java:807) Thread 86 (IPC Server Responder): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.ipc.Server$Responder.doRunLoop(Server.java:982) org.apache.hadoop.ipc.Server$Responder.run(Server.java:965) Thread 60 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@5ec0030b): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:100) org.apache.hadoop.hdfs.net.TcpPeerServer.accept(TcpPeerServer.java:141) org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:135) java.lang.Thread.run(Thread.java:748) Thread 87 (DataNode: [[[DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data1/, [DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data2/]] heartbeating to localhost/127.0.0.1:44491): State: TIMED_WAITING Blocked count: 586 Waited count: 1503 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.IncrementalBlockReportManager.waitTillNextIBR(IncrementalBlockReportManager.java:130) org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:542) org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:659) java.lang.Thread.run(Thread.java:748) Thread 85 (IPC Server idle connection scanner for port 45595): State: TIMED_WAITING Blocked count: 1 Waited count: 46 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 84 (Socket Reader #1 for port 45595): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:745) org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:724) Thread 82 (org.apache.hadoop.util.JvmPauseMonitor$Monitor@74095914): State: TIMED_WAITING Blocked count: 0 Waited count: 894 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:182) java.lang.Thread.run(Thread.java:748) Thread 65 (nioEventLoopGroup-2-1): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) java.lang.Thread.run(Thread.java:748) Thread 64 (Timer-1): State: TIMED_WAITING Blocked count: 0 Waited count: 15 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 63 (1020827856@qtp-415074966-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:40851): State: RUNNABLE Blocked count: 1 Waited count: 1 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498) org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192) org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Thread 62 (1498522151@qtp-415074966-0): State: TIMED_WAITING Blocked count: 0 Waited count: 8 Stack: java.lang.Object.wait(Native Method) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626) Thread 61 (pool-4-thread-1): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 55 (CacheReplicationMonitor(1454695942)): State: TIMED_WAITING Blocked count: 0 Waited count: 16 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163) org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor.run(CacheReplicationMonitor.java:181) Thread 54 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$LazyPersistFileScrubber@5654e22b): State: TIMED_WAITING Blocked count: 0 Waited count: 3 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.namenode.FSNamesystem$LazyPersistFileScrubber.run(FSNamesystem.java:4725) java.lang.Thread.run(Thread.java:748) Thread 53 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeEditLogRoller@68702a45): State: TIMED_WAITING Blocked count: 0 Waited count: 2 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeEditLogRoller.run(FSNamesystem.java:4642) java.lang.Thread.run(Thread.java:748) Thread 52 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor@3d921e48): State: TIMED_WAITING Blocked count: 0 Waited count: 90 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor.run(FSNamesystem.java:4598) java.lang.Thread.run(Thread.java:748) Thread 51 (org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@fe27903): State: TIMED_WAITING Blocked count: 0 Waited count: 226 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:431) java.lang.Thread.run(Thread.java:748) Thread 50 (IPC Server handler 9 on 44491): State: TIMED_WAITING Blocked count: 3 Waited count: 661 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 49 (IPC Server handler 8 on 44491): State: TIMED_WAITING Blocked count: 2 Waited count: 651 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 48 (IPC Server handler 7 on 44491): State: TIMED_WAITING Blocked count: 3 Waited count: 658 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 47 (IPC Server handler 6 on 44491): State: TIMED_WAITING Blocked count: 3 Waited count: 665 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 46 (IPC Server handler 5 on 44491): State: TIMED_WAITING Blocked count: 18 Waited count: 662 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 45 (IPC Server handler 4 on 44491): State: TIMED_WAITING Blocked count: 5 Waited count: 671 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 44 (IPC Server handler 3 on 44491): State: TIMED_WAITING Blocked count: 6 Waited count: 655 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 43 (IPC Server handler 2 on 44491): State: TIMED_WAITING Blocked count: 8 Waited count: 656 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 42 (IPC Server handler 1 on 44491): State: TIMED_WAITING Blocked count: 4 Waited count: 668 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 41 (IPC Server handler 0 on 44491): State: TIMED_WAITING Blocked count: 7 Waited count: 664 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 31 (IPC Server listener on 44491): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener.run(Server.java:807) Thread 34 (IPC Server Responder): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.ipc.Server$Responder.doRunLoop(Server.java:982) org.apache.hadoop.ipc.Server$Responder.run(Server.java:965) Thread 29 (Block report processor): State: WAITING Blocked count: 2 Waited count: 140 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@c5f3502 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403) org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.processQueue(BlockManager.java:3854) org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.run(BlockManager.java:3843) Thread 28 (org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@2748be45): State: TIMED_WAITING Blocked count: 0 Waited count: 151 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3638) java.lang.Thread.run(Thread.java:748) Thread 30 (org.apache.hadoop.hdfs.server.blockmanagement.HeartbeatManager$Monitor@1706a704): State: TIMED_WAITING Blocked count: 0 Waited count: 90 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.blockmanagement.HeartbeatManager$Monitor.run(HeartbeatManager.java:381) java.lang.Thread.run(Thread.java:748) Thread 40 (DecommissionMonitor-0): State: TIMED_WAITING Blocked count: 0 Waited count: 150 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 39 (org.apache.hadoop.hdfs.server.blockmanagement.PendingReplicationBlocks$PendingReplicationMonitor@28cb973d): State: TIMED_WAITING Blocked count: 0 Waited count: 2 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.blockmanagement.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:221) java.lang.Thread.run(Thread.java:748) Thread 35 (org.apache.hadoop.util.JvmPauseMonitor$Monitor@7501869b): State: TIMED_WAITING Blocked count: 0 Waited count: 896 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:182) java.lang.Thread.run(Thread.java:748) Thread 33 (IPC Server idle connection scanner for port 44491): State: TIMED_WAITING Blocked count: 1 Waited count: 46 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 32 (Socket Reader #1 for port 44491): State: RUNNABLE Blocked count: 1 Waited count: 1 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:745) org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:724) Thread 27 (Timer-0): State: TIMED_WAITING Blocked count: 1 Waited count: 16 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 25 (1941945463@qtp-2007783481-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:43361): State: RUNNABLE Blocked count: 1 Waited count: 1 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498) org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192) org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Thread 24 (1848959351@qtp-2007783481-0): State: TIMED_WAITING Blocked count: 0 Waited count: 8 Stack: java.lang.Object.wait(Native Method) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626) Thread 23 (pool-2-thread-1): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 14 (org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner): State: WAITING Blocked count: 4 Waited count: 5 Waiting on java.lang.ref.ReferenceQueue$Lock@688d8764 Stack: java.lang.Object.wait(Native Method) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164) org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:3063) java.lang.Thread.run(Thread.java:748) Thread 13 (Time-limited test): State: RUNNABLE Blocked count: 91 Waited count: 233 Stack: sun.management.ThreadImpl.getThreadInfo1(Native Method) sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:178) sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:139) org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:168) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) org.apache.hadoop.hbase.util.Threads$PrintThreadInfoLazyHolder$1.printThreadInfo(Threads.java:294) org.apache.hadoop.hbase.util.Threads.printThreadInfo(Threads.java:341) org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:135) org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:367) org.apache.hadoop.hbase.MiniHBaseCluster.waitUntilShutDown(MiniHBaseCluster.java:865) org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:1130) org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:1105) org.apache.hadoop.hbase.TestFullLogReconstruction.tearDownAfterClass(TestFullLogReconstruction.java:68) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) Thread 9 (ReaderThread): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: java.net.SocketInputStream.socketRead0(Native Method) java.net.SocketInputStream.socketRead(SocketInputStream.java:116) java.net.SocketInputStream.read(SocketInputStream.java:171) java.net.SocketInputStream.read(SocketInputStream.java:141) sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284) sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326) sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178) java.io.InputStreamReader.read(InputStreamReader.java:184) java.io.BufferedReader.fill(BufferedReader.java:161) java.io.BufferedReader.readLine(BufferedReader.java:324) java.io.BufferedReader.readLine(BufferedReader.java:389) org.eclipse.jdt.internal.junit.runner.RemoteTestRunner$ReaderThread.run(RemoteTestRunner.java:148) Thread 4 (Signal Dispatcher): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: Thread 3 (Finalizer): State: WAITING Blocked count: 266 Waited count: 15 Waiting on java.lang.ref.ReferenceQueue$Lock@60cb01b5 Stack: java.lang.Object.wait(Native Method) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164) java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209) Thread 2 (Reference Handler): State: WAITING Blocked count: 21 Waited count: 14 Waiting on java.lang.ref.Reference$Lock@6a276aab Stack: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) java.lang.ref.Reference.tryHandlePending(Reference.java:191) java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153) Thread 1 (main): State: TIMED_WAITING Blocked count: 0 Waited count: 2 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.FutureTask.awaitDone(FutureTask.java:426) java.util.concurrent.FutureTask.get(FutureTask.java:204) org.junit.internal.runners.statements.FailOnTimeout.getResult(FailOnTimeout.java:141) org.junit.internal.runners.statements.FailOnTimeout.evaluate(FailOnTimeout.java:127) org.junit.rules.RunRules.evaluate(RunRules.java:20) org.junit.runners.ParentRunner.run(ParentRunner.java:363) org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86) org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38) org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:538) org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:760) org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:460) org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:206) 2018-02-04 11:08:28,708 DEBUG [ReadOnlyZKClient-localhost:56841@0x143faef3] zookeeper.ReadOnlyZKClient(316): 0x143faef3 to localhost:56841 inactive for 60000ms; closing (Will reconnect when new requests) 2018-02-04 11:08:28,714 DEBUG [ReadOnlyZKClient-localhost:56841@0x46e70b17] zookeeper.ReadOnlyZKClient(316): 0x46e70b17 to localhost:56841 inactive for 60000ms; closing (Will reconnect when new requests) 2018-02-04 11:08:28,716 DEBUG [ReadOnlyZKClient-localhost:56841@0x796736be] zookeeper.ReadOnlyZKClient(316): 0x796736be to localhost:56841 inactive for 60000ms; closing (Will reconnect when new requests) Process Thread Dump: Automatic Stack Trace every 60 seconds waiting on RS:0;ubuntu:39267 204 active threads Thread 1358 (Timer for 'HBase' metrics system): State: TIMED_WAITING Blocked count: 0 Waited count: 20 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 1345 (Attach Listener): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: Thread 1069 (RS_CLOSE_REGION-regionserver/ubuntu:0-2): State: TIMED_WAITING Blocked count: 72 Waited count: 577 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:133) org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:718) org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:605) org.apache.hadoop.hbase.regionserver.wal.WALUtil.doFullAppendTransaction(WALUtil.java:154) org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeMarker(WALUtil.java:132) org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeRegionEventMarker(WALUtil.java:97) org.apache.hadoop.hbase.regionserver.HRegion.writeRegionCloseMarker(HRegion.java:1103) org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1615) org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 1062 (RS_CLOSE_REGION-regionserver/ubuntu:0-1): State: WAITING Blocked count: 54 Waited count: 344 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5fbe8967 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 1061 (RS_CLOSE_REGION-regionserver/ubuntu:0-0): State: TIMED_WAITING Blocked count: 67 Waited count: 583 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:133) org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:718) org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:605) org.apache.hadoop.hbase.regionserver.wal.WALUtil.doFullAppendTransaction(WALUtil.java:154) org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeMarker(WALUtil.java:132) org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeRegionEventMarker(WALUtil.java:97) org.apache.hadoop.hbase.regionserver.HRegion.writeRegionCloseMarker(HRegion.java:1103) org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1615) org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 988 (RS-EventLoopGroup-3-4): State: RUNNABLE Blocked count: 15 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 987 (Default-IPC-NioEventLoopGroup-7-3): State: RUNNABLE Blocked count: 23 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 986 (Default-IPC-NioEventLoopGroup-7-4): State: RUNNABLE Blocked count: 18 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 845 (RS_OPEN_REGION-regionserver/ubuntu:0-2): State: WAITING Blocked count: 50 Waited count: 67 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@72aa087e Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 842 (RS_OPEN_REGION-regionserver/ubuntu:0-1): State: WAITING Blocked count: 42 Waited count: 66 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@72aa087e Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 841 (RS_OPEN_REGION-regionserver/ubuntu:0-0): State: WAITING Blocked count: 42 Waited count: 64 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@72aa087e Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 810 (region-location-4): State: WAITING Blocked count: 5 Waited count: 4 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@282bcdfd Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 809 (region-location-3): State: WAITING Blocked count: 8 Waited count: 4 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@282bcdfd Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 808 (region-location-2): State: WAITING Blocked count: 13 Waited count: 7 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@282bcdfd Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 807 (region-location-1): State: WAITING Blocked count: 11 Waited count: 14 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@282bcdfd Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 571 (RS-EventLoopGroup-1-5): State: RUNNABLE Blocked count: 2 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 570 (Default-IPC-NioEventLoopGroup-7-2): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 568 (RS-EventLoopGroup-4-5): State: RUNNABLE Blocked count: 15 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 567 (Default-IPC-NioEventLoopGroup-7-1): State: RUNNABLE Blocked count: 49 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 560 (RS-EventLoopGroup-3-3): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 553 (RS_OPEN_PRIORITY_REGION-regionserver/ubuntu:0-0): State: WAITING Blocked count: 14 Waited count: 26 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6b2dcd67 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 551 (RS-EventLoopGroup-3-2): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 549 (RS-EventLoopGroup-4-4): State: RUNNABLE Blocked count: 1 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 544 (region-location-0): State: WAITING Blocked count: 12 Waited count: 18 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@282bcdfd Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 521 (RS-EventLoopGroup-4-3): State: RUNNABLE Blocked count: 1 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 508 (RS-EventLoopGroup-4-2): State: RUNNABLE Blocked count: 1 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 504 (AsyncFSWAL-0): State: WAITING Blocked count: 0 Waited count: 169 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1c3dccfe Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 503 (RS-EventLoopGroup-5-8): State: RUNNABLE Blocked count: 7 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 497 (RS-EventLoopGroup-5-7): State: RUNNABLE Blocked count: 1 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 483 (RS-EventLoopGroup-5-6): State: RUNNABLE Blocked count: 3 Waited count: 2 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 480 (LeaseRenewer:zhangduo.hfs.0@localhost:44491): State: TIMED_WAITING Blocked count: 16 Waited count: 532 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:444) org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304) java.lang.Thread.run(Thread.java:748) Thread 482 (RS-EventLoopGroup-5-5): State: RUNNABLE Blocked count: 12 Waited count: 3 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 476 (ubuntu:39267Replication Statistics #0): State: TIMED_WAITING Blocked count: 0 Waited count: 2 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 470 (ReplicationExecutor-0): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@153c8699 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 458 (regionserver/ubuntu:0.procedureResultReporter): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@788f3b3e Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:75) Thread 455 (regionserver/ubuntu:0.logRoller): State: TIMED_WAITING Blocked count: 6 Waited count: 506 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:147) java.lang.Thread.run(Thread.java:748) Thread 450 (regionserver/ubuntu:0.procedureResultReporter): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73ec8f20 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:75) Thread 452 (regionserver/ubuntu:0.procedureResultReporter): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@723b1ca8 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:75) Thread 443 (regionserver/ubuntu:0.Chore.1): State: TIMED_WAITING Blocked count: 1 Waited count: 37 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 431 (JvmPauseMonitor): State: TIMED_WAITING Blocked count: 0 Waited count: 1002 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:154) java.lang.Thread.run(Thread.java:748) Thread 423 (ReadOnlyZKClient-localhost:56841@0x796736be): State: TIMED_WAITING Blocked count: 1 Waited count: 11 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:307) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$63/1354899920.run(Unknown Source) java.lang.Thread.run(Thread.java:748) Thread 422 (ReadOnlyZKClient-localhost:56841@0x143faef3): State: TIMED_WAITING Blocked count: 1 Waited count: 11 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:307) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$63/1354899920.run(Unknown Source) java.lang.Thread.run(Thread.java:748) Thread 421 (ReadOnlyZKClient-localhost:56841@0x46e70b17): State: TIMED_WAITING Blocked count: 1 Waited count: 11 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:307) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$63/1354899920.run(Unknown Source) java.lang.Thread.run(Thread.java:748) Thread 419 (IPC Parameter Sending Thread #1): State: TIMED_WAITING Blocked count: 0 Waited count: 2314 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 416 (threadDeathWatcher-6-1): State: TIMED_WAITING Blocked count: 0 Waited count: 502 Stack: java.lang.Thread.sleep(Native Method) org.apache.hbase.thirdparty.io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:152) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 415 (RS-EventLoopGroup-1-4): State: RUNNABLE Blocked count: 29 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 414 (RS-EventLoopGroup-1-3): State: RUNNABLE Blocked count: 18 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 413 (RS-EventLoopGroup-1-2): State: RUNNABLE Blocked count: 16 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 367 (RpcClient-timer-pool1-t1): State: TIMED_WAITING Blocked count: 0 Waited count: 50124 Stack: java.lang.Thread.sleep(Native Method) org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:560) org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:459) java.lang.Thread.run(Thread.java:748) Thread 410 (RS-EventLoopGroup-5-4): State: RUNNABLE Blocked count: 25 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 411 (RS-EventLoopGroup-5-2): State: RUNNABLE Blocked count: 56 Waited count: 3 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 412 (RS-EventLoopGroup-5-3): State: RUNNABLE Blocked count: 31 Waited count: 4 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 407 (snapshot-hfile-cleaner-cache-refresher): State: TIMED_WAITING Blocked count: 1 Waited count: 4 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 400 (M:0;ubuntu:39063-EventThread): State: WAITING Blocked count: 1 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4fcbce78 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501) Thread 399 (M:0;ubuntu:39063-SendThread(localhost:56841)): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141) Thread 289 (RS:0;ubuntu:39267): State: TIMED_WAITING Blocked count: 198 Waited count: 2510 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.regionserver.HRegionServer.sleep(HRegionServer.java:1425) org.apache.hadoop.hbase.regionserver.HRegionServer.waitOnAllRegionsToClose(HRegionServer.java:1411) org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1103) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:187) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$0(MiniHBaseCluster.java:186) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:171) java.security.AccessController.doPrivileged(Native Method) javax.security.auth.Subject.doAs(Subject.java:360) org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726) org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:168) java.lang.Thread.run(Thread.java:748) Thread 368 (Idle-Rpc-Conn-Sweeper-pool2-t1): State: TIMED_WAITING Blocked count: 0 Waited count: 23 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 334 (org.apache.hadoop.hdfs.PeerCache@2e1946df): State: TIMED_WAITING Blocked count: 1 Waited count: 168 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:255) org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:46) org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:124) java.lang.Thread.run(Thread.java:748) Thread 332 (M:0;ubuntu:39063-MemStoreChunkPool Statistics): State: TIMED_WAITING Blocked count: 0 Waited count: 2 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 328 (Monitor thread for TaskMonitor): State: TIMED_WAITING Blocked count: 0 Waited count: 51 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.monitoring.TaskMonitor$MonitorRunnable.run(TaskMonitor.java:302) java.lang.Thread.run(Thread.java:748) Thread 310 (RS-EventLoopGroup-5-1): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 291 (RS-EventLoopGroup-4-1): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 288 (RpcServer.replication.FPBQ.Fifo.handler=2,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@50f24e6c Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 287 (RpcServer.replication.FPBQ.Fifo.handler=1,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@5b366694 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 286 (RpcServer.replication.FPBQ.Fifo.handler=0,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@68239fa7 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 285 (RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=39267): State: WAITING Blocked count: 8 Waited count: 20 Waiting on java.util.concurrent.Semaphore$NonfairSync@64e05a8c Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 284 (RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@26a3f952 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 283 (RpcServer.priority.FPBQ.Fifo.handler=3,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@4ec929d4 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 282 (RpcServer.priority.FPBQ.Fifo.handler=2,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@3ae3490d Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 281 (RpcServer.priority.FPBQ.Fifo.handler=1,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@5cb140e7 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 280 (RpcServer.priority.FPBQ.Fifo.handler=0,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@65614b4a Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 279 (RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39267): State: WAITING Blocked count: 33 Waited count: 56 Waiting on java.util.concurrent.Semaphore$NonfairSync@626159bb Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 278 (RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 2 Waiting on java.util.concurrent.Semaphore$NonfairSync@42eadfdb Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 277 (RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@148a1919 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 276 (RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@7b5e2bd9 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 275 (RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@139a0988 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 272 (RS-EventLoopGroup-3-1): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 248 (RS-EventLoopGroup-1-1): State: RUNNABLE Blocked count: 1 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 247 (HBase-Metrics2-1): State: TIMED_WAITING Blocked count: 0 Waited count: 547 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 235 (ProcessThread(sid:0 cport:56841):): State: WAITING Blocked count: 1 Waited count: 834 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@133caf02 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:122) Thread 234 (SyncThread:0): State: WAITING Blocked count: 0 Waited count: 806 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5c669c88 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:127) Thread 233 (SessionTracker): State: TIMED_WAITING Blocked count: 0 Waited count: 254 Stack: java.lang.Object.wait(Native Method) org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146) Thread 232 (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:56841): State: RUNNABLE Blocked count: 9 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:173) java.lang.Thread.run(Thread.java:748) Thread 229 (java.util.concurrent.ThreadPoolExecutor$Worker@1d726817[State = -1, empty queue]): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 224 (refreshUsed-/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data6/current/BP-35161775-127.0.1.1-1517713219102): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:132) java.lang.Thread.run(Thread.java:748) Thread 222 (refreshUsed-/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data5/current/BP-35161775-127.0.1.1-1517713219102): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:132) java.lang.Thread.run(Thread.java:748) Thread 218 (org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl$LazyWriter@2327f0e2): State: TIMED_WAITING Blocked count: 0 Waited count: 9 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl$LazyWriter.run(FsDatasetImpl.java:3071) java.lang.Thread.run(Thread.java:748) Thread 217 (VolumeScannerThread(/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data6)): State: TIMED_WAITING Blocked count: 1 Waited count: 2 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:622) Thread 216 (VolumeScannerThread(/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data5)): State: TIMED_WAITING Blocked count: 1 Waited count: 2 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:622) Thread 213 (IPC Server handler 9 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 548 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 212 (IPC Server handler 8 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 556 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 211 (IPC Server handler 7 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 526 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 210 (IPC Server handler 6 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 569 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 209 (IPC Server handler 5 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 536 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 208 (IPC Server handler 4 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 562 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 207 (IPC Server handler 3 on 43825): State: TIMED_WAITING Blocked count: 1 Waited count: 510 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 206 (IPC Server handler 2 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 561 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 205 (IPC Server handler 1 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 605 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 204 (IPC Server handler 0 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 508 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 199 (IPC Server listener on 43825): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener.run(Server.java:807) Thread 202 (IPC Server Responder): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.ipc.Server$Responder.doRunLoop(Server.java:982) org.apache.hadoop.ipc.Server$Responder.run(Server.java:965) Thread 161 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@3e65225e): State: RUNNABLE Blocked count: 1 Waited count: 0 Stack: sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:100) org.apache.hadoop.hdfs.net.TcpPeerServer.accept(TcpPeerServer.java:141) org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:135) java.lang.Thread.run(Thread.java:748) Thread 203 (DataNode: [[[DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data5/, [DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data6/]] heartbeating to localhost/127.0.0.1:44491): State: TIMED_WAITING Blocked count: 652 Waited count: 1684 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.IncrementalBlockReportManager.waitTillNextIBR(IncrementalBlockReportManager.java:130) org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:542) org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:659) java.lang.Thread.run(Thread.java:748) Thread 201 (IPC Server idle connection scanner for port 43825): State: TIMED_WAITING Blocked count: 1 Waited count: 52 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 200 (Socket Reader #1 for port 43825): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:745) org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:724) Thread 198 (org.apache.hadoop.util.JvmPauseMonitor$Monitor@3369a0f5): State: TIMED_WAITING Blocked count: 0 Waited count: 1010 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:182) java.lang.Thread.run(Thread.java:748) Thread 182 (nioEventLoopGroup-6-1): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) java.lang.Thread.run(Thread.java:748) Thread 181 (Timer-3): State: TIMED_WAITING Blocked count: 0 Waited count: 17 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 180 (java.util.concurrent.ThreadPoolExecutor$Worker@4231ed0c[State = -1, empty queue]): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 175 (refreshUsed-/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data4/current/BP-35161775-127.0.1.1-1517713219102): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:132) java.lang.Thread.run(Thread.java:748) Thread 173 (refreshUsed-/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data3/current/BP-35161775-127.0.1.1-1517713219102): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:132) java.lang.Thread.run(Thread.java:748) Thread 169 (org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl$LazyWriter@221a6ce4): State: TIMED_WAITING Blocked count: 0 Waited count: 9 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl$LazyWriter.run(FsDatasetImpl.java:3071) java.lang.Thread.run(Thread.java:748) Thread 168 (VolumeScannerThread(/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data4)): State: TIMED_WAITING Blocked count: 1 Waited count: 2 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:622) Thread 167 (VolumeScannerThread(/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data3)): State: TIMED_WAITING Blocked count: 1 Waited count: 2 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:622) Thread 164 (103115058@qtp-229466626-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:41633): State: RUNNABLE Blocked count: 2 Waited count: 1 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498) org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192) org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Thread 163 (498148987@qtp-229466626-0): State: TIMED_WAITING Blocked count: 0 Waited count: 9 Stack: java.lang.Object.wait(Native Method) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626) Thread 162 (pool-12-thread-1): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 156 (IPC Server handler 9 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 554 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 155 (IPC Server handler 8 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 511 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 154 (IPC Server handler 7 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 515 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 153 (IPC Server handler 6 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 565 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 152 (IPC Server handler 5 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 540 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 151 (IPC Server handler 4 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 524 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 150 (IPC Server handler 3 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 561 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 149 (IPC Server handler 2 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 554 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 148 (IPC Server handler 1 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 510 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 147 (IPC Server handler 0 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 507 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 142 (IPC Server listener on 33493): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener.run(Server.java:807) Thread 145 (IPC Server Responder): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.ipc.Server$Responder.doRunLoop(Server.java:982) org.apache.hadoop.ipc.Server$Responder.run(Server.java:965) Thread 102 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@26506cf6): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:100) org.apache.hadoop.hdfs.net.TcpPeerServer.accept(TcpPeerServer.java:141) org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:135) java.lang.Thread.run(Thread.java:748) Thread 146 (DataNode: [[[DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data3/, [DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data4/]] heartbeating to localhost/127.0.0.1:44491): State: TIMED_WAITING Blocked count: 633 Waited count: 1679 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.IncrementalBlockReportManager.waitTillNextIBR(IncrementalBlockReportManager.java:130) org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:542) org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:659) java.lang.Thread.run(Thread.java:748) Thread 144 (IPC Server idle connection scanner for port 33493): State: TIMED_WAITING Blocked count: 1 Waited count: 52 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 143 (Socket Reader #1 for port 33493): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:745) org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:724) Thread 141 (org.apache.hadoop.util.JvmPauseMonitor$Monitor@1d004c0c): State: TIMED_WAITING Blocked count: 0 Waited count: 1012 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:182) java.lang.Thread.run(Thread.java:748) Thread 125 (nioEventLoopGroup-4-1): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) java.lang.Thread.run(Thread.java:748) Thread 124 (Timer-2): State: TIMED_WAITING Blocked count: 0 Waited count: 17 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 123 (java.util.concurrent.ThreadPoolExecutor$Worker@3adb50a8[State = -1, empty queue]): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 118 (refreshUsed-/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data1/current/BP-35161775-127.0.1.1-1517713219102): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:132) java.lang.Thread.run(Thread.java:748) Thread 117 (refreshUsed-/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data2/current/BP-35161775-127.0.1.1-1517713219102): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:132) java.lang.Thread.run(Thread.java:748) Thread 112 (org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl$LazyWriter@7e498b6b): State: TIMED_WAITING Blocked count: 0 Waited count: 9 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl$LazyWriter.run(FsDatasetImpl.java:3071) java.lang.Thread.run(Thread.java:748) Thread 111 (VolumeScannerThread(/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data2)): State: TIMED_WAITING Blocked count: 11 Waited count: 2 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:622) Thread 110 (VolumeScannerThread(/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data1)): State: TIMED_WAITING Blocked count: 7 Waited count: 2 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:622) Thread 106 (IPC Client (1670261399) connection to localhost/127.0.0.1:44491 from zhangduo): State: TIMED_WAITING Blocked count: 1604 Waited count: 1605 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:934) org.apache.hadoop.ipc.Client$Connection.run(Client.java:979) Thread 105 (1512126834@qtp-261780187-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:34039): State: RUNNABLE Blocked count: 1 Waited count: 1 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498) org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192) org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Thread 104 (1495196264@qtp-261780187-0): State: TIMED_WAITING Blocked count: 0 Waited count: 9 Stack: java.lang.Object.wait(Native Method) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626) Thread 103 (pool-6-thread-1): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 97 (IPC Server handler 9 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 623 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 96 (IPC Server handler 8 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 564 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 95 (IPC Server handler 7 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 595 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 94 (IPC Server handler 6 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 582 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 93 (IPC Server handler 5 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 581 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 92 (IPC Server handler 4 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 548 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 91 (IPC Server handler 3 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 586 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 90 (IPC Server handler 2 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 596 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 89 (IPC Server handler 1 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 611 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 88 (IPC Server handler 0 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 601 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 83 (IPC Server listener on 45595): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener.run(Server.java:807) Thread 86 (IPC Server Responder): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.ipc.Server$Responder.doRunLoop(Server.java:982) org.apache.hadoop.ipc.Server$Responder.run(Server.java:965) Thread 60 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@5ec0030b): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:100) org.apache.hadoop.hdfs.net.TcpPeerServer.accept(TcpPeerServer.java:141) org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:135) java.lang.Thread.run(Thread.java:748) Thread 87 (DataNode: [[[DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data1/, [DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data2/]] heartbeating to localhost/127.0.0.1:44491): State: TIMED_WAITING Blocked count: 646 Waited count: 1683 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.IncrementalBlockReportManager.waitTillNextIBR(IncrementalBlockReportManager.java:130) org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:542) org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:659) java.lang.Thread.run(Thread.java:748) Thread 85 (IPC Server idle connection scanner for port 45595): State: TIMED_WAITING Blocked count: 1 Waited count: 52 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 84 (Socket Reader #1 for port 45595): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:745) org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:724) Thread 82 (org.apache.hadoop.util.JvmPauseMonitor$Monitor@74095914): State: TIMED_WAITING Blocked count: 0 Waited count: 1014 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:182) java.lang.Thread.run(Thread.java:748) Thread 65 (nioEventLoopGroup-2-1): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) java.lang.Thread.run(Thread.java:748) Thread 64 (Timer-1): State: TIMED_WAITING Blocked count: 0 Waited count: 17 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 63 (1020827856@qtp-415074966-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:40851): State: RUNNABLE Blocked count: 1 Waited count: 1 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498) org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192) org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Thread 62 (1498522151@qtp-415074966-0): State: TIMED_WAITING Blocked count: 0 Waited count: 9 Stack: java.lang.Object.wait(Native Method) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626) Thread 61 (pool-4-thread-1): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 55 (CacheReplicationMonitor(1454695942)): State: TIMED_WAITING Blocked count: 0 Waited count: 18 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163) org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor.run(CacheReplicationMonitor.java:181) Thread 54 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$LazyPersistFileScrubber@5654e22b): State: TIMED_WAITING Blocked count: 0 Waited count: 3 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.namenode.FSNamesystem$LazyPersistFileScrubber.run(FSNamesystem.java:4725) java.lang.Thread.run(Thread.java:748) Thread 53 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeEditLogRoller@68702a45): State: TIMED_WAITING Blocked count: 0 Waited count: 2 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeEditLogRoller.run(FSNamesystem.java:4642) java.lang.Thread.run(Thread.java:748) Thread 52 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor@3d921e48): State: TIMED_WAITING Blocked count: 0 Waited count: 102 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor.run(FSNamesystem.java:4598) java.lang.Thread.run(Thread.java:748) Thread 51 (org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@fe27903): State: TIMED_WAITING Blocked count: 0 Waited count: 256 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:431) java.lang.Thread.run(Thread.java:748) Thread 50 (IPC Server handler 9 on 44491): State: TIMED_WAITING Blocked count: 3 Waited count: 731 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 49 (IPC Server handler 8 on 44491): State: TIMED_WAITING Blocked count: 2 Waited count: 719 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 48 (IPC Server handler 7 on 44491): State: TIMED_WAITING Blocked count: 3 Waited count: 729 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 47 (IPC Server handler 6 on 44491): State: TIMED_WAITING Blocked count: 3 Waited count: 732 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 46 (IPC Server handler 5 on 44491): State: TIMED_WAITING Blocked count: 18 Waited count: 733 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 45 (IPC Server handler 4 on 44491): State: TIMED_WAITING Blocked count: 5 Waited count: 740 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 44 (IPC Server handler 3 on 44491): State: TIMED_WAITING Blocked count: 6 Waited count: 723 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 43 (IPC Server handler 2 on 44491): State: TIMED_WAITING Blocked count: 8 Waited count: 728 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 42 (IPC Server handler 1 on 44491): State: TIMED_WAITING Blocked count: 4 Waited count: 740 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 41 (IPC Server handler 0 on 44491): State: TIMED_WAITING Blocked count: 7 Waited count: 736 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 31 (IPC Server listener on 44491): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener.run(Server.java:807) Thread 34 (IPC Server Responder): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.ipc.Server$Responder.doRunLoop(Server.java:982) org.apache.hadoop.ipc.Server$Responder.run(Server.java:965) Thread 29 (Block report processor): State: WAITING Blocked count: 2 Waited count: 140 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@c5f3502 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403) org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.processQueue(BlockManager.java:3854) org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.run(BlockManager.java:3843) Thread 28 (org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@2748be45): State: TIMED_WAITING Blocked count: 0 Waited count: 171 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3638) java.lang.Thread.run(Thread.java:748) Thread 30 (org.apache.hadoop.hdfs.server.blockmanagement.HeartbeatManager$Monitor@1706a704): State: TIMED_WAITING Blocked count: 0 Waited count: 102 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.blockmanagement.HeartbeatManager$Monitor.run(HeartbeatManager.java:381) java.lang.Thread.run(Thread.java:748) Thread 40 (DecommissionMonitor-0): State: TIMED_WAITING Blocked count: 0 Waited count: 170 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 39 (org.apache.hadoop.hdfs.server.blockmanagement.PendingReplicationBlocks$PendingReplicationMonitor@28cb973d): State: TIMED_WAITING Blocked count: 0 Waited count: 2 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.blockmanagement.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:221) java.lang.Thread.run(Thread.java:748) Thread 35 (org.apache.hadoop.util.JvmPauseMonitor$Monitor@7501869b): State: TIMED_WAITING Blocked count: 0 Waited count: 1017 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:182) java.lang.Thread.run(Thread.java:748) Thread 33 (IPC Server idle connection scanner for port 44491): State: TIMED_WAITING Blocked count: 1 Waited count: 52 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 32 (Socket Reader #1 for port 44491): State: RUNNABLE Blocked count: 1 Waited count: 1 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:745) org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:724) Thread 27 (Timer-0): State: TIMED_WAITING Blocked count: 1 Waited count: 18 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 25 (1941945463@qtp-2007783481-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:43361): State: RUNNABLE Blocked count: 1 Waited count: 1 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498) org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192) org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Thread 24 (1848959351@qtp-2007783481-0): State: TIMED_WAITING Blocked count: 0 Waited count: 9 Stack: java.lang.Object.wait(Native Method) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626) Thread 23 (pool-2-thread-1): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 14 (org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner): State: WAITING Blocked count: 4 Waited count: 5 Waiting on java.lang.ref.ReferenceQueue$Lock@688d8764 Stack: java.lang.Object.wait(Native Method) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164) org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:3063) java.lang.Thread.run(Thread.java:748) Thread 13 (Time-limited test): State: RUNNABLE Blocked count: 91 Waited count: 234 Stack: sun.management.ThreadImpl.getThreadInfo1(Native Method) sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:178) sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:139) org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:168) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) org.apache.hadoop.hbase.util.Threads$PrintThreadInfoLazyHolder$1.printThreadInfo(Threads.java:294) org.apache.hadoop.hbase.util.Threads.printThreadInfo(Threads.java:341) org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:135) org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:367) org.apache.hadoop.hbase.MiniHBaseCluster.waitUntilShutDown(MiniHBaseCluster.java:865) org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:1130) org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:1105) org.apache.hadoop.hbase.TestFullLogReconstruction.tearDownAfterClass(TestFullLogReconstruction.java:68) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) Thread 9 (ReaderThread): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: java.net.SocketInputStream.socketRead0(Native Method) java.net.SocketInputStream.socketRead(SocketInputStream.java:116) java.net.SocketInputStream.read(SocketInputStream.java:171) java.net.SocketInputStream.read(SocketInputStream.java:141) sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284) sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326) sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178) java.io.InputStreamReader.read(InputStreamReader.java:184) java.io.BufferedReader.fill(BufferedReader.java:161) java.io.BufferedReader.readLine(BufferedReader.java:324) java.io.BufferedReader.readLine(BufferedReader.java:389) org.eclipse.jdt.internal.junit.runner.RemoteTestRunner$ReaderThread.run(RemoteTestRunner.java:148) Thread 4 (Signal Dispatcher): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: Thread 3 (Finalizer): State: WAITING Blocked count: 266 Waited count: 15 Waiting on java.lang.ref.ReferenceQueue$Lock@60cb01b5 Stack: java.lang.Object.wait(Native Method) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164) java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209) Thread 2 (Reference Handler): State: WAITING Blocked count: 21 Waited count: 14 Waiting on java.lang.ref.Reference$Lock@6a276aab Stack: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) java.lang.ref.Reference.tryHandlePending(Reference.java:191) java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153) Thread 1 (main): State: TIMED_WAITING Blocked count: 0 Waited count: 2 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.FutureTask.awaitDone(FutureTask.java:426) java.util.concurrent.FutureTask.get(FutureTask.java:204) org.junit.internal.runners.statements.FailOnTimeout.getResult(FailOnTimeout.java:141) org.junit.internal.runners.statements.FailOnTimeout.evaluate(FailOnTimeout.java:127) org.junit.rules.RunRules.evaluate(RunRules.java:20) org.junit.runners.ParentRunner.run(ParentRunner.java:363) org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86) org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38) org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:538) org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:760) org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:460) org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:206) 2018-02-04 11:09:28,709 DEBUG [ReadOnlyZKClient-localhost:56841@0x143faef3] zookeeper.ReadOnlyZKClient(316): 0x143faef3 to localhost:56841 inactive for 60000ms; closing (Will reconnect when new requests) 2018-02-04 11:09:28,715 DEBUG [ReadOnlyZKClient-localhost:56841@0x46e70b17] zookeeper.ReadOnlyZKClient(316): 0x46e70b17 to localhost:56841 inactive for 60000ms; closing (Will reconnect when new requests) 2018-02-04 11:09:28,717 DEBUG [ReadOnlyZKClient-localhost:56841@0x796736be] zookeeper.ReadOnlyZKClient(316): 0x796736be to localhost:56841 inactive for 60000ms; closing (Will reconnect when new requests) Process Thread Dump: Automatic Stack Trace every 60 seconds waiting on RS:0;ubuntu:39267 205 active threads Thread 1371 (process reaper): State: TIMED_WAITING Blocked count: 0 Waited count: 3 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 1358 (Timer for 'HBase' metrics system): State: TIMED_WAITING Blocked count: 0 Waited count: 26 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 1345 (Attach Listener): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: Thread 1069 (RS_CLOSE_REGION-regionserver/ubuntu:0-2): State: TIMED_WAITING Blocked count: 72 Waited count: 637 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:133) org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:718) org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:605) org.apache.hadoop.hbase.regionserver.wal.WALUtil.doFullAppendTransaction(WALUtil.java:154) org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeMarker(WALUtil.java:132) org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeRegionEventMarker(WALUtil.java:97) org.apache.hadoop.hbase.regionserver.HRegion.writeRegionCloseMarker(HRegion.java:1103) org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1615) org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 1062 (RS_CLOSE_REGION-regionserver/ubuntu:0-1): State: WAITING Blocked count: 54 Waited count: 344 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5fbe8967 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 1061 (RS_CLOSE_REGION-regionserver/ubuntu:0-0): State: TIMED_WAITING Blocked count: 67 Waited count: 643 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:133) org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:718) org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:605) org.apache.hadoop.hbase.regionserver.wal.WALUtil.doFullAppendTransaction(WALUtil.java:154) org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeMarker(WALUtil.java:132) org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeRegionEventMarker(WALUtil.java:97) org.apache.hadoop.hbase.regionserver.HRegion.writeRegionCloseMarker(HRegion.java:1103) org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1615) org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 988 (RS-EventLoopGroup-3-4): State: RUNNABLE Blocked count: 15 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 987 (Default-IPC-NioEventLoopGroup-7-3): State: RUNNABLE Blocked count: 23 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 986 (Default-IPC-NioEventLoopGroup-7-4): State: RUNNABLE Blocked count: 18 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 845 (RS_OPEN_REGION-regionserver/ubuntu:0-2): State: WAITING Blocked count: 50 Waited count: 67 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@72aa087e Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 842 (RS_OPEN_REGION-regionserver/ubuntu:0-1): State: WAITING Blocked count: 42 Waited count: 66 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@72aa087e Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 841 (RS_OPEN_REGION-regionserver/ubuntu:0-0): State: WAITING Blocked count: 42 Waited count: 64 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@72aa087e Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 810 (region-location-4): State: WAITING Blocked count: 5 Waited count: 4 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@282bcdfd Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 809 (region-location-3): State: WAITING Blocked count: 8 Waited count: 4 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@282bcdfd Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 808 (region-location-2): State: WAITING Blocked count: 13 Waited count: 7 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@282bcdfd Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 807 (region-location-1): State: WAITING Blocked count: 11 Waited count: 14 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@282bcdfd Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 571 (RS-EventLoopGroup-1-5): State: RUNNABLE Blocked count: 2 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 570 (Default-IPC-NioEventLoopGroup-7-2): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 568 (RS-EventLoopGroup-4-5): State: RUNNABLE Blocked count: 15 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 567 (Default-IPC-NioEventLoopGroup-7-1): State: RUNNABLE Blocked count: 49 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 560 (RS-EventLoopGroup-3-3): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 553 (RS_OPEN_PRIORITY_REGION-regionserver/ubuntu:0-0): State: WAITING Blocked count: 14 Waited count: 26 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6b2dcd67 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 551 (RS-EventLoopGroup-3-2): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 549 (RS-EventLoopGroup-4-4): State: RUNNABLE Blocked count: 1 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 544 (region-location-0): State: WAITING Blocked count: 12 Waited count: 18 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@282bcdfd Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 521 (RS-EventLoopGroup-4-3): State: RUNNABLE Blocked count: 1 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 508 (RS-EventLoopGroup-4-2): State: RUNNABLE Blocked count: 1 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 504 (AsyncFSWAL-0): State: WAITING Blocked count: 0 Waited count: 169 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1c3dccfe Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 503 (RS-EventLoopGroup-5-8): State: RUNNABLE Blocked count: 7 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 497 (RS-EventLoopGroup-5-7): State: RUNNABLE Blocked count: 1 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 483 (RS-EventLoopGroup-5-6): State: RUNNABLE Blocked count: 3 Waited count: 2 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 480 (LeaseRenewer:zhangduo.hfs.0@localhost:44491): State: TIMED_WAITING Blocked count: 18 Waited count: 596 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:444) org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304) java.lang.Thread.run(Thread.java:748) Thread 482 (RS-EventLoopGroup-5-5): State: RUNNABLE Blocked count: 12 Waited count: 3 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 476 (ubuntu:39267Replication Statistics #0): State: TIMED_WAITING Blocked count: 0 Waited count: 2 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 470 (ReplicationExecutor-0): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@153c8699 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 458 (regionserver/ubuntu:0.procedureResultReporter): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@788f3b3e Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:75) Thread 455 (regionserver/ubuntu:0.logRoller): State: TIMED_WAITING Blocked count: 6 Waited count: 567 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:147) java.lang.Thread.run(Thread.java:748) Thread 450 (regionserver/ubuntu:0.procedureResultReporter): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73ec8f20 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:75) Thread 452 (regionserver/ubuntu:0.procedureResultReporter): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@723b1ca8 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:75) Thread 443 (regionserver/ubuntu:0.Chore.1): State: TIMED_WAITING Blocked count: 1 Waited count: 39 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 431 (JvmPauseMonitor): State: TIMED_WAITING Blocked count: 0 Waited count: 1123 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:154) java.lang.Thread.run(Thread.java:748) Thread 423 (ReadOnlyZKClient-localhost:56841@0x796736be): State: TIMED_WAITING Blocked count: 1 Waited count: 12 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:307) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$63/1354899920.run(Unknown Source) java.lang.Thread.run(Thread.java:748) Thread 422 (ReadOnlyZKClient-localhost:56841@0x143faef3): State: TIMED_WAITING Blocked count: 1 Waited count: 12 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:307) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$63/1354899920.run(Unknown Source) java.lang.Thread.run(Thread.java:748) Thread 421 (ReadOnlyZKClient-localhost:56841@0x46e70b17): State: TIMED_WAITING Blocked count: 1 Waited count: 12 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:307) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$63/1354899920.run(Unknown Source) java.lang.Thread.run(Thread.java:748) Thread 419 (IPC Parameter Sending Thread #1): State: TIMED_WAITING Blocked count: 0 Waited count: 2496 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 416 (threadDeathWatcher-6-1): State: TIMED_WAITING Blocked count: 0 Waited count: 562 Stack: java.lang.Thread.sleep(Native Method) org.apache.hbase.thirdparty.io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:152) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 415 (RS-EventLoopGroup-1-4): State: RUNNABLE Blocked count: 29 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 414 (RS-EventLoopGroup-1-3): State: RUNNABLE Blocked count: 18 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 413 (RS-EventLoopGroup-1-2): State: RUNNABLE Blocked count: 16 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 367 (RpcClient-timer-pool1-t1): State: TIMED_WAITING Blocked count: 0 Waited count: 56144 Stack: java.lang.Thread.sleep(Native Method) org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:560) org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:459) java.lang.Thread.run(Thread.java:748) Thread 410 (RS-EventLoopGroup-5-4): State: RUNNABLE Blocked count: 25 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 411 (RS-EventLoopGroup-5-2): State: RUNNABLE Blocked count: 56 Waited count: 3 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 412 (RS-EventLoopGroup-5-3): State: RUNNABLE Blocked count: 31 Waited count: 4 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 407 (snapshot-hfile-cleaner-cache-refresher): State: TIMED_WAITING Blocked count: 1 Waited count: 4 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 400 (M:0;ubuntu:39063-EventThread): State: WAITING Blocked count: 1 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4fcbce78 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501) Thread 399 (M:0;ubuntu:39063-SendThread(localhost:56841)): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141) Thread 289 (RS:0;ubuntu:39267): State: TIMED_WAITING Blocked count: 198 Waited count: 2810 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.regionserver.HRegionServer.sleep(HRegionServer.java:1425) org.apache.hadoop.hbase.regionserver.HRegionServer.waitOnAllRegionsToClose(HRegionServer.java:1411) org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1103) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:187) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$0(MiniHBaseCluster.java:186) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:171) java.security.AccessController.doPrivileged(Native Method) javax.security.auth.Subject.doAs(Subject.java:360) org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726) org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:168) java.lang.Thread.run(Thread.java:748) Thread 368 (Idle-Rpc-Conn-Sweeper-pool2-t1): State: TIMED_WAITING Blocked count: 0 Waited count: 23 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 334 (org.apache.hadoop.hdfs.PeerCache@2e1946df): State: TIMED_WAITING Blocked count: 1 Waited count: 188 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:255) org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:46) org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:124) java.lang.Thread.run(Thread.java:748) Thread 332 (M:0;ubuntu:39063-MemStoreChunkPool Statistics): State: TIMED_WAITING Blocked count: 0 Waited count: 2 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 328 (Monitor thread for TaskMonitor): State: TIMED_WAITING Blocked count: 0 Waited count: 57 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.monitoring.TaskMonitor$MonitorRunnable.run(TaskMonitor.java:302) java.lang.Thread.run(Thread.java:748) Thread 310 (RS-EventLoopGroup-5-1): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 291 (RS-EventLoopGroup-4-1): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 288 (RpcServer.replication.FPBQ.Fifo.handler=2,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@50f24e6c Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 287 (RpcServer.replication.FPBQ.Fifo.handler=1,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@5b366694 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 286 (RpcServer.replication.FPBQ.Fifo.handler=0,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@68239fa7 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 285 (RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=39267): State: WAITING Blocked count: 8 Waited count: 20 Waiting on java.util.concurrent.Semaphore$NonfairSync@64e05a8c Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 284 (RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@26a3f952 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 283 (RpcServer.priority.FPBQ.Fifo.handler=3,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@4ec929d4 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 282 (RpcServer.priority.FPBQ.Fifo.handler=2,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@3ae3490d Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 281 (RpcServer.priority.FPBQ.Fifo.handler=1,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@5cb140e7 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 280 (RpcServer.priority.FPBQ.Fifo.handler=0,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@65614b4a Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 279 (RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39267): State: WAITING Blocked count: 33 Waited count: 56 Waiting on java.util.concurrent.Semaphore$NonfairSync@626159bb Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 278 (RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 2 Waiting on java.util.concurrent.Semaphore$NonfairSync@42eadfdb Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 277 (RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@148a1919 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 276 (RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@7b5e2bd9 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 275 (RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39267): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@139a0988 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:103) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Thread 272 (RS-EventLoopGroup-3-1): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 248 (RS-EventLoopGroup-1-1): State: RUNNABLE Blocked count: 1 Waited count: 0 Stack: org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Thread 247 (HBase-Metrics2-1): State: TIMED_WAITING Blocked count: 0 Waited count: 607 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 235 (ProcessThread(sid:0 cport:56841):): State: WAITING Blocked count: 1 Waited count: 880 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@133caf02 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:122) Thread 234 (SyncThread:0): State: WAITING Blocked count: 0 Waited count: 852 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5c669c88 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:127) Thread 233 (SessionTracker): State: TIMED_WAITING Blocked count: 0 Waited count: 284 Stack: java.lang.Object.wait(Native Method) org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146) Thread 232 (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:56841): State: RUNNABLE Blocked count: 9 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:173) java.lang.Thread.run(Thread.java:748) Thread 229 (java.util.concurrent.ThreadPoolExecutor$Worker@1d726817[State = -1, empty queue]): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 224 (refreshUsed-/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data6/current/BP-35161775-127.0.1.1-1517713219102): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:132) java.lang.Thread.run(Thread.java:748) Thread 222 (refreshUsed-/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data5/current/BP-35161775-127.0.1.1-1517713219102): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:132) java.lang.Thread.run(Thread.java:748) Thread 218 (org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl$LazyWriter@2327f0e2): State: TIMED_WAITING Blocked count: 0 Waited count: 10 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl$LazyWriter.run(FsDatasetImpl.java:3071) java.lang.Thread.run(Thread.java:748) Thread 217 (VolumeScannerThread(/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data6)): State: TIMED_WAITING Blocked count: 1 Waited count: 2 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:622) Thread 216 (VolumeScannerThread(/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data5)): State: TIMED_WAITING Blocked count: 1 Waited count: 2 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:622) Thread 213 (IPC Server handler 9 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 608 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 212 (IPC Server handler 8 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 622 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 211 (IPC Server handler 7 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 587 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 210 (IPC Server handler 6 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 639 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 209 (IPC Server handler 5 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 596 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 208 (IPC Server handler 4 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 622 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 207 (IPC Server handler 3 on 43825): State: TIMED_WAITING Blocked count: 1 Waited count: 570 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 206 (IPC Server handler 2 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 628 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 205 (IPC Server handler 1 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 672 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 204 (IPC Server handler 0 on 43825): State: TIMED_WAITING Blocked count: 0 Waited count: 568 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 199 (IPC Server listener on 43825): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener.run(Server.java:807) Thread 202 (IPC Server Responder): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.ipc.Server$Responder.doRunLoop(Server.java:982) org.apache.hadoop.ipc.Server$Responder.run(Server.java:965) Thread 161 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@3e65225e): State: RUNNABLE Blocked count: 1 Waited count: 0 Stack: sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:100) org.apache.hadoop.hdfs.net.TcpPeerServer.accept(TcpPeerServer.java:141) org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:135) java.lang.Thread.run(Thread.java:748) Thread 203 (DataNode: [[[DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data5/, [DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data6/]] heartbeating to localhost/127.0.0.1:44491): State: TIMED_WAITING Blocked count: 713 Waited count: 1867 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.IncrementalBlockReportManager.waitTillNextIBR(IncrementalBlockReportManager.java:130) org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:542) org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:659) java.lang.Thread.run(Thread.java:748) Thread 201 (IPC Server idle connection scanner for port 43825): State: TIMED_WAITING Blocked count: 1 Waited count: 58 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 200 (Socket Reader #1 for port 43825): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:745) org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:724) Thread 198 (org.apache.hadoop.util.JvmPauseMonitor$Monitor@3369a0f5): State: TIMED_WAITING Blocked count: 0 Waited count: 1130 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:182) java.lang.Thread.run(Thread.java:748) Thread 182 (nioEventLoopGroup-6-1): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) java.lang.Thread.run(Thread.java:748) Thread 181 (Timer-3): State: TIMED_WAITING Blocked count: 0 Waited count: 19 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 180 (java.util.concurrent.ThreadPoolExecutor$Worker@4231ed0c[State = -1, empty queue]): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 175 (refreshUsed-/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data4/current/BP-35161775-127.0.1.1-1517713219102): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:132) java.lang.Thread.run(Thread.java:748) Thread 173 (refreshUsed-/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data3/current/BP-35161775-127.0.1.1-1517713219102): State: TIMED_WAITING Blocked count: 1 Waited count: 3 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:132) java.lang.Thread.run(Thread.java:748) Thread 169 (org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl$LazyWriter@221a6ce4): State: TIMED_WAITING Blocked count: 0 Waited count: 10 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl$LazyWriter.run(FsDatasetImpl.java:3071) java.lang.Thread.run(Thread.java:748) Thread 168 (VolumeScannerThread(/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data4)): State: TIMED_WAITING Blocked count: 1 Waited count: 2 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:622) Thread 167 (VolumeScannerThread(/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data3)): State: TIMED_WAITING Blocked count: 1 Waited count: 2 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:622) Thread 164 (103115058@qtp-229466626-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:41633): State: RUNNABLE Blocked count: 2 Waited count: 1 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498) org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192) org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Thread 163 (498148987@qtp-229466626-0): State: TIMED_WAITING Blocked count: 0 Waited count: 10 Stack: java.lang.Object.wait(Native Method) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626) Thread 162 (pool-12-thread-1): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 156 (IPC Server handler 9 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 631 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 155 (IPC Server handler 8 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 576 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 154 (IPC Server handler 7 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 577 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 153 (IPC Server handler 6 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 625 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 152 (IPC Server handler 5 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 600 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 151 (IPC Server handler 4 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 584 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 150 (IPC Server handler 3 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 633 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 149 (IPC Server handler 2 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 614 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 148 (IPC Server handler 1 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 570 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 147 (IPC Server handler 0 on 33493): State: TIMED_WAITING Blocked count: 0 Waited count: 567 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 142 (IPC Server listener on 33493): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener.run(Server.java:807) Thread 145 (IPC Server Responder): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.ipc.Server$Responder.doRunLoop(Server.java:982) org.apache.hadoop.ipc.Server$Responder.run(Server.java:965) Thread 102 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@26506cf6): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:100) org.apache.hadoop.hdfs.net.TcpPeerServer.accept(TcpPeerServer.java:141) org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:135) java.lang.Thread.run(Thread.java:748) Thread 146 (DataNode: [[[DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data3/, [DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data4/]] heartbeating to localhost/127.0.0.1:44491): State: TIMED_WAITING Blocked count: 693 Waited count: 1859 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.IncrementalBlockReportManager.waitTillNextIBR(IncrementalBlockReportManager.java:130) org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:542) org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:659) java.lang.Thread.run(Thread.java:748) Thread 144 (IPC Server idle connection scanner for port 33493): State: TIMED_WAITING Blocked count: 1 Waited count: 58 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 143 (Socket Reader #1 for port 33493): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:745) org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:724) Thread 141 (org.apache.hadoop.util.JvmPauseMonitor$Monitor@1d004c0c): State: TIMED_WAITING Blocked count: 0 Waited count: 1132 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:182) java.lang.Thread.run(Thread.java:748) Thread 125 (nioEventLoopGroup-4-1): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) java.lang.Thread.run(Thread.java:748) Thread 124 (Timer-2): State: TIMED_WAITING Blocked count: 0 Waited count: 19 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 123 (java.util.concurrent.ThreadPoolExecutor$Worker@3adb50a8[State = -1, empty queue]): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 118 (refreshUsed-/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data1/current/BP-35161775-127.0.1.1-1517713219102): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:132) java.lang.Thread.run(Thread.java:748) Thread 117 (refreshUsed-/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data2/current/BP-35161775-127.0.1.1-1517713219102): State: TIMED_WAITING Blocked count: 1 Waited count: 3 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:132) java.lang.Thread.run(Thread.java:748) Thread 112 (org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl$LazyWriter@7e498b6b): State: TIMED_WAITING Blocked count: 0 Waited count: 10 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl$LazyWriter.run(FsDatasetImpl.java:3071) java.lang.Thread.run(Thread.java:748) Thread 111 (VolumeScannerThread(/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data2)): State: TIMED_WAITING Blocked count: 11 Waited count: 2 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:622) Thread 110 (VolumeScannerThread(/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data1)): State: TIMED_WAITING Blocked count: 7 Waited count: 2 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:622) Thread 106 (IPC Client (1670261399) connection to localhost/127.0.0.1:44491 from zhangduo): State: TIMED_WAITING Blocked count: 1785 Waited count: 1786 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:934) org.apache.hadoop.ipc.Client$Connection.run(Client.java:979) Thread 105 (1512126834@qtp-261780187-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:34039): State: RUNNABLE Blocked count: 1 Waited count: 1 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498) org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192) org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Thread 104 (1495196264@qtp-261780187-0): State: TIMED_WAITING Blocked count: 0 Waited count: 10 Stack: java.lang.Object.wait(Native Method) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626) Thread 103 (pool-6-thread-1): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 97 (IPC Server handler 9 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 689 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 96 (IPC Server handler 8 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 648 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 95 (IPC Server handler 7 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 662 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 94 (IPC Server handler 6 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 657 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 93 (IPC Server handler 5 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 647 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 92 (IPC Server handler 4 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 615 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 91 (IPC Server handler 3 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 658 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 90 (IPC Server handler 2 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 664 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 89 (IPC Server handler 1 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 687 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 88 (IPC Server handler 0 on 45595): State: TIMED_WAITING Blocked count: 0 Waited count: 669 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 83 (IPC Server listener on 45595): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener.run(Server.java:807) Thread 86 (IPC Server Responder): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.ipc.Server$Responder.doRunLoop(Server.java:982) org.apache.hadoop.ipc.Server$Responder.run(Server.java:965) Thread 60 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@5ec0030b): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:100) org.apache.hadoop.hdfs.net.TcpPeerServer.accept(TcpPeerServer.java:141) org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:135) java.lang.Thread.run(Thread.java:748) Thread 87 (DataNode: [[[DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data1/, [DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data2/]] heartbeating to localhost/127.0.0.1:44491): State: TIMED_WAITING Blocked count: 706 Waited count: 1863 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.IncrementalBlockReportManager.waitTillNextIBR(IncrementalBlockReportManager.java:130) org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:542) org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:659) java.lang.Thread.run(Thread.java:748) Thread 85 (IPC Server idle connection scanner for port 45595): State: TIMED_WAITING Blocked count: 1 Waited count: 58 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 84 (Socket Reader #1 for port 45595): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:745) org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:724) Thread 82 (org.apache.hadoop.util.JvmPauseMonitor$Monitor@74095914): State: TIMED_WAITING Blocked count: 0 Waited count: 1134 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:182) java.lang.Thread.run(Thread.java:748) Thread 65 (nioEventLoopGroup-2-1): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) java.lang.Thread.run(Thread.java:748) Thread 64 (Timer-1): State: TIMED_WAITING Blocked count: 0 Waited count: 19 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 63 (1020827856@qtp-415074966-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:40851): State: RUNNABLE Blocked count: 1 Waited count: 1 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498) org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192) org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Thread 62 (1498522151@qtp-415074966-0): State: TIMED_WAITING Blocked count: 0 Waited count: 10 Stack: java.lang.Object.wait(Native Method) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626) Thread 61 (pool-4-thread-1): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 55 (CacheReplicationMonitor(1454695942)): State: TIMED_WAITING Blocked count: 0 Waited count: 20 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163) org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor.run(CacheReplicationMonitor.java:181) Thread 54 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$LazyPersistFileScrubber@5654e22b): State: TIMED_WAITING Blocked count: 0 Waited count: 3 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.namenode.FSNamesystem$LazyPersistFileScrubber.run(FSNamesystem.java:4725) java.lang.Thread.run(Thread.java:748) Thread 53 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeEditLogRoller@68702a45): State: TIMED_WAITING Blocked count: 0 Waited count: 2 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeEditLogRoller.run(FSNamesystem.java:4642) java.lang.Thread.run(Thread.java:748) Thread 52 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor@3d921e48): State: TIMED_WAITING Blocked count: 0 Waited count: 114 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor.run(FSNamesystem.java:4598) java.lang.Thread.run(Thread.java:748) Thread 51 (org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@fe27903): State: TIMED_WAITING Blocked count: 0 Waited count: 286 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:431) java.lang.Thread.run(Thread.java:748) Thread 50 (IPC Server handler 9 on 44491): State: TIMED_WAITING Blocked count: 3 Waited count: 797 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 49 (IPC Server handler 8 on 44491): State: TIMED_WAITING Blocked count: 2 Waited count: 782 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 48 (IPC Server handler 7 on 44491): State: TIMED_WAITING Blocked count: 3 Waited count: 790 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 47 (IPC Server handler 6 on 44491): State: TIMED_WAITING Blocked count: 3 Waited count: 797 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 46 (IPC Server handler 5 on 44491): State: TIMED_WAITING Blocked count: 18 Waited count: 800 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 45 (IPC Server handler 4 on 44491): State: TIMED_WAITING Blocked count: 5 Waited count: 803 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 44 (IPC Server handler 3 on 44491): State: TIMED_WAITING Blocked count: 6 Waited count: 790 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 43 (IPC Server handler 2 on 44491): State: TIMED_WAITING Blocked count: 8 Waited count: 795 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 42 (IPC Server handler 1 on 44491): State: TIMED_WAITING Blocked count: 4 Waited count: 807 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 41 (IPC Server handler 0 on 44491): State: TIMED_WAITING Blocked count: 7 Waited count: 802 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) Thread 31 (IPC Server listener on 44491): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener.run(Server.java:807) Thread 34 (IPC Server Responder): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.ipc.Server$Responder.doRunLoop(Server.java:982) org.apache.hadoop.ipc.Server$Responder.run(Server.java:965) Thread 29 (Block report processor): State: WAITING Blocked count: 2 Waited count: 140 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@c5f3502 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403) org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.processQueue(BlockManager.java:3854) org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.run(BlockManager.java:3843) Thread 28 (org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@2748be45): State: TIMED_WAITING Blocked count: 0 Waited count: 191 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3638) java.lang.Thread.run(Thread.java:748) Thread 30 (org.apache.hadoop.hdfs.server.blockmanagement.HeartbeatManager$Monitor@1706a704): State: TIMED_WAITING Blocked count: 0 Waited count: 114 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.blockmanagement.HeartbeatManager$Monitor.run(HeartbeatManager.java:381) java.lang.Thread.run(Thread.java:748) Thread 40 (DecommissionMonitor-0): State: TIMED_WAITING Blocked count: 0 Waited count: 190 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 39 (org.apache.hadoop.hdfs.server.blockmanagement.PendingReplicationBlocks$PendingReplicationMonitor@28cb973d): State: TIMED_WAITING Blocked count: 0 Waited count: 2 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.blockmanagement.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:221) java.lang.Thread.run(Thread.java:748) Thread 35 (org.apache.hadoop.util.JvmPauseMonitor$Monitor@7501869b): State: TIMED_WAITING Blocked count: 0 Waited count: 1137 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:182) java.lang.Thread.run(Thread.java:748) Thread 33 (IPC Server idle connection scanner for port 44491): State: TIMED_WAITING Blocked count: 1 Waited count: 58 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 32 (Socket Reader #1 for port 44491): State: RUNNABLE Blocked count: 1 Waited count: 1 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:745) org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:724) Thread 27 (Timer-0): State: TIMED_WAITING Blocked count: 1 Waited count: 20 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 25 (1941945463@qtp-2007783481-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:43361): State: RUNNABLE Blocked count: 1 Waited count: 1 Stack: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498) org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192) org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Thread 24 (1848959351@qtp-2007783481-0): State: TIMED_WAITING Blocked count: 0 Waited count: 10 Stack: java.lang.Object.wait(Native Method) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626) Thread 23 (pool-2-thread-1): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 14 (org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner): State: WAITING Blocked count: 4 Waited count: 5 Waiting on java.lang.ref.ReferenceQueue$Lock@688d8764 Stack: java.lang.Object.wait(Native Method) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164) org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:3063) java.lang.Thread.run(Thread.java:748) Thread 13 (Time-limited test): State: RUNNABLE Blocked count: 91 Waited count: 235 Stack: sun.management.ThreadImpl.getThreadInfo1(Native Method) sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:178) sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:139) org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:168) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) org.apache.hadoop.hbase.util.Threads$PrintThreadInfoLazyHolder$1.printThreadInfo(Threads.java:294) org.apache.hadoop.hbase.util.Threads.printThreadInfo(Threads.java:341) org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:135) org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:367) org.apache.hadoop.hbase.MiniHBaseCluster.waitUntilShutDown(MiniHBaseCluster.java:865) org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:1130) org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:1105) org.apache.hadoop.hbase.TestFullLogReconstruction.tearDownAfterClass(TestFullLogReconstruction.java:68) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) Thread 9 (ReaderThread): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: java.net.SocketInputStream.socketRead0(Native Method) java.net.SocketInputStream.socketRead(SocketInputStream.java:116) java.net.SocketInputStream.read(SocketInputStream.java:171) java.net.SocketInputStream.read(SocketInputStream.java:141) sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284) sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326) sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178) java.io.InputStreamReader.read(InputStreamReader.java:184) java.io.BufferedReader.fill(BufferedReader.java:161) java.io.BufferedReader.readLine(BufferedReader.java:324) java.io.BufferedReader.readLine(BufferedReader.java:389) org.eclipse.jdt.internal.junit.runner.RemoteTestRunner$ReaderThread.run(RemoteTestRunner.java:148) Thread 4 (Signal Dispatcher): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: Thread 3 (Finalizer): State: WAITING Blocked count: 266 Waited count: 15 Waiting on java.lang.ref.ReferenceQueue$Lock@60cb01b5 Stack: java.lang.Object.wait(Native Method) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164) java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209) Thread 2 (Reference Handler): State: WAITING Blocked count: 21 Waited count: 14 Waiting on java.lang.ref.Reference$Lock@6a276aab Stack: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) java.lang.ref.Reference.tryHandlePending(Reference.java:191) java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153) Thread 1 (main): State: TIMED_WAITING Blocked count: 0 Waited count: 2 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.FutureTask.awaitDone(FutureTask.java:426) java.util.concurrent.FutureTask.get(FutureTask.java:204) org.junit.internal.runners.statements.FailOnTimeout.getResult(FailOnTimeout.java:141) org.junit.internal.runners.statements.FailOnTimeout.evaluate(FailOnTimeout.java:127) org.junit.rules.RunRules.evaluate(RunRules.java:20) org.junit.runners.ParentRunner.run(ParentRunner.java:363) org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86) org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38) org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:538) org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:760) org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:460) org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:206) 2018-02-04 11:10:17,246 DEBUG [Time-limited test] hbase.LocalHBaseCluster(369): Interrupted java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1260) at org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:133) at org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:367) at org.apache.hadoop.hbase.MiniHBaseCluster.waitUntilShutDown(MiniHBaseCluster.java:865) at org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:1130) at org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:1105) at org.apache.hadoop.hbase.TestFullLogReconstruction.tearDownAfterClass(TestFullLogReconstruction.java:68) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) 2018-02-04 11:10:17,249 WARN [Time-limited test] datanode.DirectoryScanner(529): DirectoryScanner: shutdown has been called 2018-02-04 11:10:17,272 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(112): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@3cabec11 2018-02-04 11:10:17,273 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(135): Shutdown hook finished. 2018-02-04 11:10:17,273 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(112): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@3cabec11 2018-02-04 11:10:17,273 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(135): Shutdown hook finished. 2018-02-04 11:10:17,292 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(112): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@3cabec11 2018-02-04 11:10:17,297 INFO [Time-limited test] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2018-02-04 11:10:17,403 WARN [DataNode: [[[DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data5/, [DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data6/]] heartbeating to localhost/127.0.0.1:44491] datanode.IncrementalBlockReportManager(132): IncrementalBlockReportManager interrupted 2018-02-04 11:10:17,403 WARN [DataNode: [[[DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data5/, [DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data6/]] heartbeating to localhost/127.0.0.1:44491] datanode.BPServiceActor(670): Ending block pool service for: Block pool BP-35161775-127.0.1.1-1517713219102 (Datanode Uuid 321c6636-03c4-4e6f-90e5-eb2f341bbda2) service to localhost/127.0.0.1:44491 2018-02-04 11:10:17,416 WARN [Time-limited test] datanode.DirectoryScanner(529): DirectoryScanner: shutdown has been called 2018-02-04 11:10:17,434 INFO [Time-limited test] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2018-02-04 11:10:17,542 WARN [DataNode: [[[DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data3/, [DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data4/]] heartbeating to localhost/127.0.0.1:44491] datanode.IncrementalBlockReportManager(132): IncrementalBlockReportManager interrupted 2018-02-04 11:10:17,545 WARN [DataNode: [[[DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data3/, [DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data4/]] heartbeating to localhost/127.0.0.1:44491] datanode.BPServiceActor(670): Ending block pool service for: Block pool BP-35161775-127.0.1.1-1517713219102 (Datanode Uuid ec0e09c5-3c3f-4fbb-b904-e820adf8f4b9) service to localhost/127.0.0.1:44491 2018-02-04 11:10:17,555 WARN [Time-limited test] datanode.DirectoryScanner(529): DirectoryScanner: shutdown has been called 2018-02-04 11:10:17,579 INFO [Time-limited test] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2018-02-04 11:10:17,682 WARN [DataNode: [[[DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data1/, [DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data2/]] heartbeating to localhost/127.0.0.1:44491] datanode.IncrementalBlockReportManager(132): IncrementalBlockReportManager interrupted 2018-02-04 11:10:17,683 WARN [DataNode: [[[DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data1/, [DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/0c37b2c0-3e73-47e6-bc07-f4f010cd6195/cluster_c1ee159c-4d97-4568-a29c-e8d635e1ffb8/dfs/data/data2/]] heartbeating to localhost/127.0.0.1:44491] datanode.BPServiceActor(670): Ending block pool service for: Block pool BP-35161775-127.0.1.1-1517713219102 (Datanode Uuid 1f27f1f7-7f7e-4fcc-97ef-d20669d8600a) service to localhost/127.0.0.1:44491 2018-02-04 11:10:17,705 INFO [Time-limited test] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2018-02-04 11:10:17,815 ERROR [Time-limited test] server.ZooKeeperServer(472): ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes 2018-02-04 11:10:17,820 INFO [Time-limited test] zookeeper.MiniZooKeeperCluster(324): Shutdown MiniZK cluster with all ZK servers 2018-02-04 11:10:17,863 INFO [Time-limited test] hbase.HBaseTestingUtility(1111): Minicluster is down 2018-02-04 11:10:17,916 DEBUG [M:0;ubuntu:39063-EventThread] zookeeper.ZKWatcher(443): replicationLogCleaner-0x1615ec1f6c10008, quorum=localhost:56841, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2018-02-04 11:10:17,917 DEBUG [M:0;ubuntu:39063-EventThread] zookeeper.ZKWatcher(511): replicationLogCleaner-0x1615ec1f6c10008, quorum=localhost:56841, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring 2018-02-04 11:10:27,561 DEBUG [M:0;ubuntu:39063-MemStoreChunkPool Statistics] regionserver.ChunkCreator$MemStoreChunkPool$StatisticsThread(339): Stats: current pool size=24,created chunk count=29,reused chunk count=0,reuseRatio=0.00% 2018-02-04 11:10:28,417 WARN [snapshot-hfile-cleaner-cache-refresher] snapshot.SnapshotFileCache$RefreshCacheTask(315): Failed to refresh snapshot hfile cache! java.net.ConnectException: Call From ubuntu/127.0.1.1 to localhost:44491 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732) at org.apache.hadoop.ipc.Client.call(Client.java:1480) at org.apache.hadoop.ipc.Client.call(Client.java:1413) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy28.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:776) at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy29.getFileInfo(Unknown Source) at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:372) at com.sun.proxy.$Proxy30.getFileInfo(Unknown Source) at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:372) at com.sun.proxy.$Proxy30.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2108) at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305) at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317) at org.apache.hadoop.hbase.master.snapshot.SnapshotFileCache.refreshCache(SnapshotFileCache.java:211) at org.apache.hadoop.hbase.master.snapshot.SnapshotFileCache.access$0(SnapshotFileCache.java:207) at org.apache.hadoop.hbase.master.snapshot.SnapshotFileCache$RefreshCacheTask.run(SnapshotFileCache.java:313) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:615) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:713) at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:376) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1529) at org.apache.hadoop.ipc.Client.call(Client.java:1452) ... 30 more 2018-02-04 11:10:28,710 DEBUG [ReadOnlyZKClient-localhost:56841@0x143faef3] zookeeper.ReadOnlyZKClient(316): 0x143faef3 to localhost:56841 inactive for 60000ms; closing (Will reconnect when new requests) 2018-02-04 11:10:28,717 DEBUG [ReadOnlyZKClient-localhost:56841@0x46e70b17] zookeeper.ReadOnlyZKClient(316): 0x46e70b17 to localhost:56841 inactive for 60000ms; closing (Will reconnect when new requests) 2018-02-04 11:10:28,719 DEBUG [ReadOnlyZKClient-localhost:56841@0x796736be] zookeeper.ReadOnlyZKClient(316): 0x796736be to localhost:56841 inactive for 60000ms; closing (Will reconnect when new requests) 2018-02-04 11:10:30,676 WARN [LeaseRenewer:zhangduo.hfs.0@localhost:44491] hdfs.LeaseRenewer(468): Failed to renew lease for [DFSClient_NONMAPREDUCE_426488853_13] for 30 seconds. Will retry shortly ... java.net.ConnectException: Call From ubuntu/127.0.1.1 to localhost:44491 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732) at org.apache.hadoop.ipc.Client.call(Client.java:1480) at org.apache.hadoop.ipc.Client.call(Client.java:1413) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy28.renewLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:595) at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy29.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:372) at com.sun.proxy.$Proxy30.renewLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:892) at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:423) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:448) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304) at java.lang.Thread.run(Thread.java:748) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:615) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:713) at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:376) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1529) at org.apache.hadoop.ipc.Client.call(Client.java:1452) ... 21 more 2018-02-04 11:10:31,679 WARN [LeaseRenewer:zhangduo.hfs.0@localhost:44491] hdfs.LeaseRenewer(468): Failed to renew lease for [DFSClient_NONMAPREDUCE_426488853_13] for 31 seconds. Will retry shortly ... java.net.ConnectException: Call From ubuntu/127.0.1.1 to localhost:44491 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732) at org.apache.hadoop.ipc.Client.call(Client.java:1480) at org.apache.hadoop.ipc.Client.call(Client.java:1413) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy28.renewLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:595) at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy29.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:372) at com.sun.proxy.$Proxy30.renewLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:892) at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:423) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:448) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304) at java.lang.Thread.run(Thread.java:748) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:615) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:713) at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:376) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1529) at org.apache.hadoop.ipc.Client.call(Client.java:1452) ... 21 more 2018-02-04 11:10:32,683 WARN [LeaseRenewer:zhangduo.hfs.0@localhost:44491] hdfs.LeaseRenewer(468): Failed to renew lease for [DFSClient_NONMAPREDUCE_426488853_13] for 32 seconds. Will retry shortly ... java.net.ConnectException: Call From ubuntu/127.0.1.1 to localhost:44491 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732) at org.apache.hadoop.ipc.Client.call(Client.java:1480) at org.apache.hadoop.ipc.Client.call(Client.java:1413) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy28.renewLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:595) at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy29.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:372) at com.sun.proxy.$Proxy30.renewLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:892) at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:423) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:448) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304) at java.lang.Thread.run(Thread.java:748) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:615) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:713) at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:376) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1529) at org.apache.hadoop.ipc.Client.call(Client.java:1452) ... 21 more 2018-02-04 11:10:33,685 WARN [LeaseRenewer:zhangduo.hfs.0@localhost:44491] hdfs.LeaseRenewer(468): Failed to renew lease for [DFSClient_NONMAPREDUCE_426488853_13] for 33 seconds. Will retry shortly ... java.net.ConnectException: Call From ubuntu/127.0.1.1 to localhost:44491 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732) at org.apache.hadoop.ipc.Client.call(Client.java:1480) at org.apache.hadoop.ipc.Client.call(Client.java:1413) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy28.renewLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:595) at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy29.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:372) at com.sun.proxy.$Proxy30.renewLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:892) at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:423) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:448) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304) at java.lang.Thread.run(Thread.java:748) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:615) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:713) at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:376) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1529) at org.apache.hadoop.ipc.Client.call(Client.java:1452) ... 21 more 2018-02-04 11:10:34,689 WARN [LeaseRenewer:zhangduo.hfs.0@localhost:44491] hdfs.LeaseRenewer(468): Failed to renew lease for [DFSClient_NONMAPREDUCE_426488853_13] for 34 seconds. Will retry shortly ... java.net.ConnectException: Call From ubuntu/127.0.1.1 to localhost:44491 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732) at org.apache.hadoop.ipc.Client.call(Client.java:1480) at org.apache.hadoop.ipc.Client.call(Client.java:1413) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy28.renewLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:595) at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy29.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:372) at com.sun.proxy.$Proxy30.renewLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:892) at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:423) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:448) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304) at java.lang.Thread.run(Thread.java:748) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:615) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:713) at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:376) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1529) at org.apache.hadoop.ipc.Client.call(Client.java:1452) ... 21 more 2018-02-04 11:10:35,692 WARN [LeaseRenewer:zhangduo.hfs.0@localhost:44491] hdfs.LeaseRenewer(468): Failed to renew lease for [DFSClient_NONMAPREDUCE_426488853_13] for 35 seconds. Will retry shortly ... java.net.ConnectException: Call From ubuntu/127.0.1.1 to localhost:44491 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732) at org.apache.hadoop.ipc.Client.call(Client.java:1480) at org.apache.hadoop.ipc.Client.call(Client.java:1413) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy28.renewLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:595) at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy29.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:372) at com.sun.proxy.$Proxy30.renewLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:892) at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:423) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:448) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304) at java.lang.Thread.run(Thread.java:748) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:615) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:713) at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:376) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1529) at org.apache.hadoop.ipc.Client.call(Client.java:1452) ... 21 more 2018-02-04 11:10:36,157 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2018-02-04 11:10:36,695 WARN [LeaseRenewer:zhangduo.hfs.0@localhost:44491] hdfs.LeaseRenewer(468): Failed to renew lease for [DFSClient_NONMAPREDUCE_426488853_13] for 36 seconds. Will retry shortly ... java.net.ConnectException: Call From ubuntu/127.0.1.1 to localhost:44491 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732) at org.apache.hadoop.ipc.Client.call(Client.java:1480) at org.apache.hadoop.ipc.Client.call(Client.java:1413) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy28.renewLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:595) at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy29.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:372) at com.sun.proxy.$Proxy30.renewLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:892) at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:423) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:448) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304) at java.lang.Thread.run(Thread.java:748) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:615) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:713) at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:376) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1529) at org.apache.hadoop.ipc.Client.call(Client.java:1452) ... 21 more 2018-02-04 11:10:37,698 WARN [LeaseRenewer:zhangduo.hfs.0@localhost:44491] hdfs.LeaseRenewer(468): Failed to renew lease for [DFSClient_NONMAPREDUCE_426488853_13] for 37 seconds. Will retry shortly ... java.net.ConnectException: Call From ubuntu/127.0.1.1 to localhost:44491 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732) at org.apache.hadoop.ipc.Client.call(Client.java:1480) at org.apache.hadoop.ipc.Client.call(Client.java:1413) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy28.renewLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:595) at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy29.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:372) at com.sun.proxy.$Proxy30.renewLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:892) at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:423) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:448) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304) at java.lang.Thread.run(Thread.java:748) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:615) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:713) at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:376) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1529) at org.apache.hadoop.ipc.Client.call(Client.java:1452) ... 21 more 2018-02-04 11:10:38,702 WARN [LeaseRenewer:zhangduo.hfs.0@localhost:44491] hdfs.LeaseRenewer(468): Failed to renew lease for [DFSClient_NONMAPREDUCE_426488853_13] for 38 seconds. Will retry shortly ... java.net.ConnectException: Call From ubuntu/127.0.1.1 to localhost:44491 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732) at org.apache.hadoop.ipc.Client.call(Client.java:1480) at org.apache.hadoop.ipc.Client.call(Client.java:1413) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy28.renewLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:595) at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy29.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:372) at com.sun.proxy.$Proxy30.renewLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:892) at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:423) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:448) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304) at java.lang.Thread.run(Thread.java:748) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:615) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:713) at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:376) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1529) at org.apache.hadoop.ipc.Client.call(Client.java:1452) ... 21 more 2018-02-04 11:10:39,707 WARN [LeaseRenewer:zhangduo.hfs.0@localhost:44491] hdfs.LeaseRenewer(468): Failed to renew lease for [DFSClient_NONMAPREDUCE_426488853_13] for 39 seconds. Will retry shortly ... java.net.ConnectException: Call From ubuntu/127.0.1.1 to localhost:44491 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732) at org.apache.hadoop.ipc.Client.call(Client.java:1480) at org.apache.hadoop.ipc.Client.call(Client.java:1413) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy28.renewLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:595) at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy29.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:372) at com.sun.proxy.$Proxy30.renewLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:892) at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:423) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:448) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304) at java.lang.Thread.run(Thread.java:748) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:615) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:713) at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:376) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1529) at org.apache.hadoop.ipc.Client.call(Client.java:1452) ... 21 more 2018-02-04 11:10:40,470 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-0] helpers.MarkerIgnoringBase(159): ***** ABORTING region server ubuntu,39267,1517713227316: Unrecoverable exception while closing region tabletest,ttt,1517713232695.31b3e09f2db65dd6057a91f45cb8b163., still finishing close ***** org.apache.hadoop.hbase.exceptions.TimeoutIOException: Failed to get sync result after 300000 ms for txid=143, WAL system stuck? at org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:136) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:718) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:605) at org.apache.hadoop.hbase.regionserver.wal.WALUtil.doFullAppendTransaction(WALUtil.java:154) at org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeMarker(WALUtil.java:132) at org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeRegionEventMarker(WALUtil.java:97) at org.apache.hadoop.hbase.regionserver.HRegion.writeRegionCloseMarker(HRegion.java:1103) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1615) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-02-04 11:10:40,471 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-0] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-02-04 11:10:40,477 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegionServer(2350): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 881852416, "init" : 197132288, "max" : 2794455040, "used" : 373627256 }, "NonHeapMemoryUsage" : { "committed" : 124674048, "init" : 2555904, "max" : -1, "used" : 122778312 }, "Verbose" : false, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : -1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 6227078, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 207, "TotalCallTime_min" : 0, "TotalCallTime_max" : 0, "TotalCallTime_mean" : 0, "TotalCallTime_25th_percentile" : 0, "TotalCallTime_median" : 0, "TotalCallTime_75th_percentile" : 0, "TotalCallTime_90th_percentile" : 0, "TotalCallTime_95th_percentile" : 0, "TotalCallTime_98th_percentile" : 0, "TotalCallTime_99th_percentile" : 0, "TotalCallTime_99.9th_percentile" : 0, "TotalCallTime_TimeRangeCount_600000-inf" : 207, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 207, "ResponseSize_min" : 0, "ResponseSize_max" : 0, "ResponseSize_mean" : 0, "ResponseSize_25th_percentile" : 0, "ResponseSize_median" : 0, "ResponseSize_75th_percentile" : 0, "ResponseSize_90th_percentile" : 0, "ResponseSize_95th_percentile" : 0, "ResponseSize_98th_percentile" : 0, "ResponseSize_99th_percentile" : 0, "ResponseSize_99.9th_percentile" : 0, "ResponseSize_SizeRangeCount_100000000-inf" : 207, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 207, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 0, "ProcessCallTime_mean" : 0, "ProcessCallTime_25th_percentile" : 0, "ProcessCallTime_median" : 0, "ProcessCallTime_75th_percentile" : 0, "ProcessCallTime_90th_percentile" : 0, "ProcessCallTime_95th_percentile" : 0, "ProcessCallTime_98th_percentile" : 0, "ProcessCallTime_99th_percentile" : 0, "ProcessCallTime_99.9th_percentile" : 0, "ProcessCallTime_TimeRangeCount_600000-inf" : 207, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 9, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 207, "RequestSize_min" : 0, "RequestSize_max" : 0, "RequestSize_mean" : 0, "RequestSize_25th_percentile" : 0, "RequestSize_median" : 0, "RequestSize_75th_percentile" : 0, "RequestSize_90th_percentile" : 0, "RequestSize_95th_percentile" : 0, "RequestSize_98th_percentile" : 0, "RequestSize_99th_percentile" : 0, "RequestSize_99.9th_percentile" : 0, "RequestSize_SizeRangeCount_100000000-inf" : 207, "sentBytes" : 2283235, "QueueCallTime_num_ops" : 207, "QueueCallTime_min" : 0, "QueueCallTime_max" : 0, "QueueCallTime_mean" : 0, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 0, "QueueCallTime_90th_percentile" : 0, "QueueCallTime_95th_percentile" : 0, "QueueCallTime_98th_percentile" : 0, "QueueCallTime_99th_percentile" : 0, "QueueCallTime_99.9th_percentile" : 0, "QueueCallTime_TimeRangeCount_600000-inf" : 207, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:56841", "tag.serverName" : "ubuntu,39267,1517713227316", "tag.clusterId" : "990c040f-15f4-4bef-a30c-08f39bc273d8", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "regionCount" : 8, "storeCount" : 8, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 0, "memStoreSize" : 534358, "storeFileSize" : 0, "maxStoreFileAge" : 0, "minStoreFileAge" : 298656, "avgStoreFileAge" : 298880, "numReferenceFiles" : 0, "regionServerStartTime" : 1517713227316, "averageRegionSize" : 66794, "totalRequestCount" : 32, "totalRowActionRequestCount" : 34451, "readRequestCount" : 9844, "filteredReadRequestCount" : 0, "writeRequestCount" : 24607, "rpcGetRequestCount" : 4, "rpcScanRequestCount" : 19, "rpcMultiRequestCount" : 5, "rpcMutateRequestCount" : 2, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 0, "staticIndexSize" : 0, "staticBloomSize" : 0, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1116962296, "blockCacheCount" : 0, "blockCacheSize" : 819720, "blockCacheHitCount" : 0, "blockCacheHitCountPrimary" : 0, "blockCacheMissCount" : 0, "blockCacheMissCountPrimary" : 0, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 0.0, "blockCacheExpressHitPercent" : 0.0, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 0, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 0, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 0, "l1CacheMissCount" : 0, "l1CacheHitRatio" : NaN, "l1CacheMissRatio" : NaN, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 10545, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 400710, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 19, "ScanTime_min" : 0, "ScanTime_max" : 0, "ScanTime_mean" : 0, "ScanTime_25th_percentile" : 0, "ScanTime_median" : 0, "ScanTime_75th_percentile" : 0, "ScanTime_90th_percentile" : 0, "ScanTime_95th_percentile" : 0, "ScanTime_98th_percentile" : 0, "ScanTime_99th_percentile" : 0, "ScanTime_99.9th_percentile" : 0, "ScanTime_TimeRangeCount_600000-inf" : 19, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 2, "Put_min" : 0, "Put_max" : 0, "Put_mean" : 0, "Put_25th_percentile" : 0, "Put_median" : 0, "Put_75th_percentile" : 0, "Put_90th_percentile" : 0, "Put_95th_percentile" : 0, "Put_98th_percentile" : 0, "Put_99th_percentile" : 0, "Put_99.9th_percentile" : 0, "Put_TimeRangeCount_600000-inf" : 2, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 2, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "FlushMemstoreSize_SizeRangeCount_100000000-inf" : 2, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 45, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "PutBatch_TimeRangeCount_600000-inf" : 45, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 4, "Get_min" : 0, "Get_max" : 0, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 0, "Get_75th_percentile" : 0, "Get_90th_percentile" : 0, "Get_95th_percentile" : 0, "Get_98th_percentile" : 0, "Get_99th_percentile" : 0, "Get_99.9th_percentile" : 0, "Get_TimeRangeCount_600000-inf" : 4, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 66522, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 2, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "FlushTime_TimeRangeCount_600000-inf" : 2, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 19, "ScanSize_min" : 0, "ScanSize_max" : 0, "ScanSize_mean" : 0, "ScanSize_25th_percentile" : 0, "ScanSize_median" : 0, "ScanSize_75th_percentile" : 0, "ScanSize_90th_percentile" : 0, "ScanSize_95th_percentile" : 0, "ScanSize_98th_percentile" : 0, "ScanSize_99th_percentile" : 0, "ScanSize_99.9th_percentile" : 0, "ScanSize_SizeRangeCount_100000000-inf" : 19, "slowGetCount" : 0, "flushedMemstoreBytes" : 267140, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 2, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "FlushOutputSize_SizeRangeCount_100000000-inf" : 2, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-02-04 11:10:40,479 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] ipc.NettyRpcConnection(254): Connecting to ubuntu/127.0.1.1:39063 2018-02-04 11:10:40,483 DEBUG [RS-EventLoopGroup-5-5] ipc.FailedServers(56): Added failed server with address ubuntu/127.0.1.1:39063 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: syscall:getsockopt(..) failed: Connection refused: ubuntu/127.0.1.1:39063 2018-02-04 11:10:40,483 WARN [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegionServer(2371): Unable to report fatal error to master org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: java.net.ConnectException: Call to ubuntu/127.0.1.1:39063 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: syscall:getsockopt(..) failed: Connection refused: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$0(AbstractRpcClient.java:324) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:571) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.reportRSFatalError(RegionServerStatusProtos.java:13568) at org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:2368) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.abortRegionServer(MiniHBaseCluster.java:208) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$1(MiniHBaseCluster.java:207) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$2.run(MiniHBaseCluster.java:201) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.abort(MiniHBaseCluster.java:198) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:115) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.net.ConnectException: Call to ubuntu/127.0.1.1:39063 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: syscall:getsockopt(..) failed: Connection refused: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:165) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$3(AbstractRpcClient.java:375) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:1) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.BufferCallBeforeInitHandler.userEventTriggered(BufferCallBeforeInitHandler.java:92) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.userEventTriggered(DefaultChannelPipeline.java:1377) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireUserEventTriggered(DefaultChannelPipeline.java:929) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.failInit(NettyRpcConnection.java:179) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.access$5(NettyRpcConnection.java:176) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:267) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:1) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:122) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:659) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:678) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:552) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:304) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) ... 1 more Caused by: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: syscall:getsockopt(..) failed: Connection refused: ubuntu/127.0.1.1:39063 at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(..)(Unknown Source) Caused by: org.apache.hbase.thirdparty.io.netty.channel.unix.Errors$NativeConnectException: syscall:getsockopt(..) failed: Connection refused ... 1 more 2018-02-04 11:10:40,486 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-0] executor.EventHandler(186): Caught throwable while processing event M_RS_CLOSE_REGION java.lang.RuntimeException: org.apache.hadoop.hbase.exceptions.TimeoutIOException: Failed to get sync result after 300000 ms for txid=143, WAL system stuck? at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:117) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.hbase.exceptions.TimeoutIOException: Failed to get sync result after 300000 ms for txid=143, WAL system stuck? at org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:136) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:718) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:605) at org.apache.hadoop.hbase.regionserver.wal.WALUtil.doFullAppendTransaction(WALUtil.java:154) at org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeMarker(WALUtil.java:132) at org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeRegionEventMarker(WALUtil.java:97) at org.apache.hadoop.hbase.regionserver.HRegion.writeRegionCloseMarker(HRegion.java:1103) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1615) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) ... 4 more 2018-02-04 11:10:40,487 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-0] helpers.MarkerIgnoringBase(159): ***** ABORTING region server ubuntu,39267,1517713227316: Caught throwable while processing event M_RS_CLOSE_REGION ***** java.lang.RuntimeException: org.apache.hadoop.hbase.exceptions.TimeoutIOException: Failed to get sync result after 300000 ms for txid=143, WAL system stuck? at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:117) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.hbase.exceptions.TimeoutIOException: Failed to get sync result after 300000 ms for txid=143, WAL system stuck? at org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:136) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:718) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:605) at org.apache.hadoop.hbase.regionserver.wal.WALUtil.doFullAppendTransaction(WALUtil.java:154) at org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeMarker(WALUtil.java:132) at org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeRegionEventMarker(WALUtil.java:97) at org.apache.hadoop.hbase.regionserver.HRegion.writeRegionCloseMarker(HRegion.java:1103) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1615) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) ... 4 more 2018-02-04 11:10:40,487 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-0] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-02-04 11:10:40,489 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegionServer(2350): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 881852416, "init" : 197132288, "max" : 2794455040, "used" : 374904344 }, "NonHeapMemoryUsage" : { "committed" : 124674048, "init" : 2555904, "max" : -1, "used" : 122778312 }, "Verbose" : false, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : -1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 6227078, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 207, "TotalCallTime_min" : 0, "TotalCallTime_max" : 0, "TotalCallTime_mean" : 0, "TotalCallTime_25th_percentile" : 0, "TotalCallTime_median" : 0, "TotalCallTime_75th_percentile" : 0, "TotalCallTime_90th_percentile" : 0, "TotalCallTime_95th_percentile" : 0, "TotalCallTime_98th_percentile" : 0, "TotalCallTime_99th_percentile" : 0, "TotalCallTime_99.9th_percentile" : 0, "TotalCallTime_TimeRangeCount_600000-inf" : 207, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 207, "ResponseSize_min" : 0, "ResponseSize_max" : 0, "ResponseSize_mean" : 0, "ResponseSize_25th_percentile" : 0, "ResponseSize_median" : 0, "ResponseSize_75th_percentile" : 0, "ResponseSize_90th_percentile" : 0, "ResponseSize_95th_percentile" : 0, "ResponseSize_98th_percentile" : 0, "ResponseSize_99th_percentile" : 0, "ResponseSize_99.9th_percentile" : 0, "ResponseSize_SizeRangeCount_100000000-inf" : 207, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 207, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 0, "ProcessCallTime_mean" : 0, "ProcessCallTime_25th_percentile" : 0, "ProcessCallTime_median" : 0, "ProcessCallTime_75th_percentile" : 0, "ProcessCallTime_90th_percentile" : 0, "ProcessCallTime_95th_percentile" : 0, "ProcessCallTime_98th_percentile" : 0, "ProcessCallTime_99th_percentile" : 0, "ProcessCallTime_99.9th_percentile" : 0, "ProcessCallTime_TimeRangeCount_600000-inf" : 207, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 9, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 207, "RequestSize_min" : 0, "RequestSize_max" : 0, "RequestSize_mean" : 0, "RequestSize_25th_percentile" : 0, "RequestSize_median" : 0, "RequestSize_75th_percentile" : 0, "RequestSize_90th_percentile" : 0, "RequestSize_95th_percentile" : 0, "RequestSize_98th_percentile" : 0, "RequestSize_99th_percentile" : 0, "RequestSize_99.9th_percentile" : 0, "RequestSize_SizeRangeCount_100000000-inf" : 207, "sentBytes" : 2283235, "QueueCallTime_num_ops" : 207, "QueueCallTime_min" : 0, "QueueCallTime_max" : 0, "QueueCallTime_mean" : 0, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 0, "QueueCallTime_90th_percentile" : 0, "QueueCallTime_95th_percentile" : 0, "QueueCallTime_98th_percentile" : 0, "QueueCallTime_99th_percentile" : 0, "QueueCallTime_99.9th_percentile" : 0, "QueueCallTime_TimeRangeCount_600000-inf" : 207, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:56841", "tag.serverName" : "ubuntu,39267,1517713227316", "tag.clusterId" : "990c040f-15f4-4bef-a30c-08f39bc273d8", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "regionCount" : 8, "storeCount" : 8, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 0, "memStoreSize" : 534358, "storeFileSize" : 0, "maxStoreFileAge" : 0, "minStoreFileAge" : 298656, "avgStoreFileAge" : 298880, "numReferenceFiles" : 0, "regionServerStartTime" : 1517713227316, "averageRegionSize" : 66794, "totalRequestCount" : 32, "totalRowActionRequestCount" : 34451, "readRequestCount" : 9844, "filteredReadRequestCount" : 0, "writeRequestCount" : 24607, "rpcGetRequestCount" : 4, "rpcScanRequestCount" : 19, "rpcMultiRequestCount" : 5, "rpcMutateRequestCount" : 2, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 0, "staticIndexSize" : 0, "staticBloomSize" : 0, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1116962296, "blockCacheCount" : 0, "blockCacheSize" : 819720, "blockCacheHitCount" : 0, "blockCacheHitCountPrimary" : 0, "blockCacheMissCount" : 0, "blockCacheMissCountPrimary" : 0, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 0.0, "blockCacheExpressHitPercent" : 0.0, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 0, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 0, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 0, "l1CacheMissCount" : 0, "l1CacheHitRatio" : NaN, "l1CacheMissRatio" : NaN, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 10545, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 400710, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 19, "ScanTime_min" : 0, "ScanTime_max" : 0, "ScanTime_mean" : 0, "ScanTime_25th_percentile" : 0, "ScanTime_median" : 0, "ScanTime_75th_percentile" : 0, "ScanTime_90th_percentile" : 0, "ScanTime_95th_percentile" : 0, "ScanTime_98th_percentile" : 0, "ScanTime_99th_percentile" : 0, "ScanTime_99.9th_percentile" : 0, "ScanTime_TimeRangeCount_600000-inf" : 19, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 2, "Put_min" : 0, "Put_max" : 0, "Put_mean" : 0, "Put_25th_percentile" : 0, "Put_median" : 0, "Put_75th_percentile" : 0, "Put_90th_percentile" : 0, "Put_95th_percentile" : 0, "Put_98th_percentile" : 0, "Put_99th_percentile" : 0, "Put_99.9th_percentile" : 0, "Put_TimeRangeCount_600000-inf" : 2, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 2, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "FlushMemstoreSize_SizeRangeCount_100000000-inf" : 2, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 45, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "PutBatch_TimeRangeCount_600000-inf" : 45, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 4, "Get_min" : 0, "Get_max" : 0, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 0, "Get_75th_percentile" : 0, "Get_90th_percentile" : 0, "Get_95th_percentile" : 0, "Get_98th_percentile" : 0, "Get_99th_percentile" : 0, "Get_99.9th_percentile" : 0, "Get_TimeRangeCount_600000-inf" : 4, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 66522, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 2, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "FlushTime_TimeRangeCount_600000-inf" : 2, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 19, "ScanSize_min" : 0, "ScanSize_max" : 0, "ScanSize_mean" : 0, "ScanSize_25th_percentile" : 0, "ScanSize_median" : 0, "ScanSize_75th_percentile" : 0, "ScanSize_90th_percentile" : 0, "ScanSize_95th_percentile" : 0, "ScanSize_98th_percentile" : 0, "ScanSize_99th_percentile" : 0, "ScanSize_99.9th_percentile" : 0, "ScanSize_SizeRangeCount_100000000-inf" : 19, "slowGetCount" : 0, "flushedMemstoreBytes" : 267140, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 2, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "FlushOutputSize_SizeRangeCount_100000000-inf" : 2, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-02-04 11:10:40,490 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-0] ipc.AbstractRpcClient(349): Not trying to connect to ubuntu/127.0.1.1:39063 this server is in the failed servers list 2018-02-04 11:10:40,491 WARN [RS_CLOSE_REGION-regionserver/ubuntu:0-0] regionserver.HRegionServer(2371): Unable to report fatal error to master org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: java.io.IOException: Call to ubuntu/127.0.1.1:39063 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$0(AbstractRpcClient.java:324) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:571) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.reportRSFatalError(RegionServerStatusProtos.java:13568) at org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:2368) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.abortRegionServer(MiniHBaseCluster.java:208) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$1(MiniHBaseCluster.java:207) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$2.run(MiniHBaseCluster.java:201) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.abort(MiniHBaseCluster.java:198) at org.apache.hadoop.hbase.executor.EventHandler.handleException(EventHandler.java:188) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:106) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Call to ubuntu/127.0.1.1:39063 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$3(AbstractRpcClient.java:375) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:1) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:423) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:328) ... 17 more Caused by: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.getConnection(AbstractRpcClient.java:352) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:420) ... 18 more 2018-02-04 11:10:40,710 WARN [LeaseRenewer:zhangduo.hfs.0@localhost:44491] hdfs.LeaseRenewer(468): Failed to renew lease for [DFSClient_NONMAPREDUCE_426488853_13] for 40 seconds. Will retry shortly ... java.net.ConnectException: Call From ubuntu/127.0.1.1 to localhost:44491 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732) at org.apache.hadoop.ipc.Client.call(Client.java:1480) at org.apache.hadoop.ipc.Client.call(Client.java:1413) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy28.renewLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:595) at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy29.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:372) at com.sun.proxy.$Proxy30.renewLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:892) at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:423) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:448) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304) at java.lang.Thread.run(Thread.java:748) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:615) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:713) at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:376) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1529) at org.apache.hadoop.ipc.Client.call(Client.java:1452) ... 21 more 2018-02-04 11:10:40,895 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-2] helpers.MarkerIgnoringBase(159): ***** ABORTING region server ubuntu,39267,1517713227316: Unrecoverable exception while closing region tabletest,ddd,1517713232695.68c9676b485038d63fd2cf8a435420e5., still finishing close ***** org.apache.hadoop.hbase.exceptions.TimeoutIOException: Failed to get sync result after 300000 ms for txid=146, WAL system stuck? at org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:136) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:718) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:605) at org.apache.hadoop.hbase.regionserver.wal.WALUtil.doFullAppendTransaction(WALUtil.java:154) at org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeMarker(WALUtil.java:132) at org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeRegionEventMarker(WALUtil.java:97) at org.apache.hadoop.hbase.regionserver.HRegion.writeRegionCloseMarker(HRegion.java:1103) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1615) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-02-04 11:10:40,896 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-2] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-02-04 11:10:40,898 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegionServer(2350): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 881852416, "init" : 197132288, "max" : 2794455040, "used" : 376001152 }, "NonHeapMemoryUsage" : { "committed" : 124674048, "init" : 2555904, "max" : -1, "used" : 122782088 }, "Verbose" : false, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : -1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 6227078, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 207, "TotalCallTime_min" : 0, "TotalCallTime_max" : 0, "TotalCallTime_mean" : 0, "TotalCallTime_25th_percentile" : 0, "TotalCallTime_median" : 0, "TotalCallTime_75th_percentile" : 0, "TotalCallTime_90th_percentile" : 0, "TotalCallTime_95th_percentile" : 0, "TotalCallTime_98th_percentile" : 0, "TotalCallTime_99th_percentile" : 0, "TotalCallTime_99.9th_percentile" : 0, "TotalCallTime_TimeRangeCount_600000-inf" : 207, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 207, "ResponseSize_min" : 0, "ResponseSize_max" : 0, "ResponseSize_mean" : 0, "ResponseSize_25th_percentile" : 0, "ResponseSize_median" : 0, "ResponseSize_75th_percentile" : 0, "ResponseSize_90th_percentile" : 0, "ResponseSize_95th_percentile" : 0, "ResponseSize_98th_percentile" : 0, "ResponseSize_99th_percentile" : 0, "ResponseSize_99.9th_percentile" : 0, "ResponseSize_SizeRangeCount_100000000-inf" : 207, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 207, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 0, "ProcessCallTime_mean" : 0, "ProcessCallTime_25th_percentile" : 0, "ProcessCallTime_median" : 0, "ProcessCallTime_75th_percentile" : 0, "ProcessCallTime_90th_percentile" : 0, "ProcessCallTime_95th_percentile" : 0, "ProcessCallTime_98th_percentile" : 0, "ProcessCallTime_99th_percentile" : 0, "ProcessCallTime_99.9th_percentile" : 0, "ProcessCallTime_TimeRangeCount_600000-inf" : 207, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 9, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 207, "RequestSize_min" : 0, "RequestSize_max" : 0, "RequestSize_mean" : 0, "RequestSize_25th_percentile" : 0, "RequestSize_median" : 0, "RequestSize_75th_percentile" : 0, "RequestSize_90th_percentile" : 0, "RequestSize_95th_percentile" : 0, "RequestSize_98th_percentile" : 0, "RequestSize_99th_percentile" : 0, "RequestSize_99.9th_percentile" : 0, "RequestSize_SizeRangeCount_100000000-inf" : 207, "sentBytes" : 2283235, "QueueCallTime_num_ops" : 207, "QueueCallTime_min" : 0, "QueueCallTime_max" : 0, "QueueCallTime_mean" : 0, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 0, "QueueCallTime_90th_percentile" : 0, "QueueCallTime_95th_percentile" : 0, "QueueCallTime_98th_percentile" : 0, "QueueCallTime_99th_percentile" : 0, "QueueCallTime_99.9th_percentile" : 0, "QueueCallTime_TimeRangeCount_600000-inf" : 207, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:56841", "tag.serverName" : "ubuntu,39267,1517713227316", "tag.clusterId" : "990c040f-15f4-4bef-a30c-08f39bc273d8", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "regionCount" : 8, "storeCount" : 8, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 0, "memStoreSize" : 534358, "storeFileSize" : 0, "maxStoreFileAge" : 0, "minStoreFileAge" : 298656, "avgStoreFileAge" : 298880, "numReferenceFiles" : 0, "regionServerStartTime" : 1517713227316, "averageRegionSize" : 66794, "totalRequestCount" : 32, "totalRowActionRequestCount" : 34451, "readRequestCount" : 9844, "filteredReadRequestCount" : 0, "writeRequestCount" : 24607, "rpcGetRequestCount" : 4, "rpcScanRequestCount" : 19, "rpcMultiRequestCount" : 5, "rpcMutateRequestCount" : 2, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 0, "staticIndexSize" : 0, "staticBloomSize" : 0, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1116962296, "blockCacheCount" : 0, "blockCacheSize" : 819720, "blockCacheHitCount" : 0, "blockCacheHitCountPrimary" : 0, "blockCacheMissCount" : 0, "blockCacheMissCountPrimary" : 0, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 0.0, "blockCacheExpressHitPercent" : 0.0, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 0, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 0, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 0, "l1CacheMissCount" : 0, "l1CacheHitRatio" : NaN, "l1CacheMissRatio" : NaN, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 10545, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 400710, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 19, "ScanTime_min" : 0, "ScanTime_max" : 0, "ScanTime_mean" : 0, "ScanTime_25th_percentile" : 0, "ScanTime_median" : 0, "ScanTime_75th_percentile" : 0, "ScanTime_90th_percentile" : 0, "ScanTime_95th_percentile" : 0, "ScanTime_98th_percentile" : 0, "ScanTime_99th_percentile" : 0, "ScanTime_99.9th_percentile" : 0, "ScanTime_TimeRangeCount_600000-inf" : 19, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 2, "Put_min" : 0, "Put_max" : 0, "Put_mean" : 0, "Put_25th_percentile" : 0, "Put_median" : 0, "Put_75th_percentile" : 0, "Put_90th_percentile" : 0, "Put_95th_percentile" : 0, "Put_98th_percentile" : 0, "Put_99th_percentile" : 0, "Put_99.9th_percentile" : 0, "Put_TimeRangeCount_600000-inf" : 2, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 2, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "FlushMemstoreSize_SizeRangeCount_100000000-inf" : 2, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 45, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "PutBatch_TimeRangeCount_600000-inf" : 45, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 4, "Get_min" : 0, "Get_max" : 0, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 0, "Get_75th_percentile" : 0, "Get_90th_percentile" : 0, "Get_95th_percentile" : 0, "Get_98th_percentile" : 0, "Get_99th_percentile" : 0, "Get_99.9th_percentile" : 0, "Get_TimeRangeCount_600000-inf" : 4, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 66522, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 2, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "FlushTime_TimeRangeCount_600000-inf" : 2, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 19, "ScanSize_min" : 0, "ScanSize_max" : 0, "ScanSize_mean" : 0, "ScanSize_25th_percentile" : 0, "ScanSize_median" : 0, "ScanSize_75th_percentile" : 0, "ScanSize_90th_percentile" : 0, "ScanSize_95th_percentile" : 0, "ScanSize_98th_percentile" : 0, "ScanSize_99th_percentile" : 0, "ScanSize_99.9th_percentile" : 0, "ScanSize_SizeRangeCount_100000000-inf" : 19, "slowGetCount" : 0, "flushedMemstoreBytes" : 267140, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 2, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "FlushOutputSize_SizeRangeCount_100000000-inf" : 2, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-02-04 11:10:40,899 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] ipc.AbstractRpcClient(349): Not trying to connect to ubuntu/127.0.1.1:39063 this server is in the failed servers list 2018-02-04 11:10:40,900 WARN [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegionServer(2371): Unable to report fatal error to master org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: java.io.IOException: Call to ubuntu/127.0.1.1:39063 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$0(AbstractRpcClient.java:324) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:571) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.reportRSFatalError(RegionServerStatusProtos.java:13568) at org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:2368) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.abortRegionServer(MiniHBaseCluster.java:208) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$1(MiniHBaseCluster.java:207) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$2.run(MiniHBaseCluster.java:201) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.abort(MiniHBaseCluster.java:198) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:115) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Call to ubuntu/127.0.1.1:39063 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$3(AbstractRpcClient.java:375) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:1) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:423) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:328) ... 17 more Caused by: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.getConnection(AbstractRpcClient.java:352) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:420) ... 18 more 2018-02-04 11:10:40,900 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-2] executor.EventHandler(186): Caught throwable while processing event M_RS_CLOSE_REGION java.lang.RuntimeException: org.apache.hadoop.hbase.exceptions.TimeoutIOException: Failed to get sync result after 300000 ms for txid=146, WAL system stuck? at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:117) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.hbase.exceptions.TimeoutIOException: Failed to get sync result after 300000 ms for txid=146, WAL system stuck? at org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:136) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:718) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:605) at org.apache.hadoop.hbase.regionserver.wal.WALUtil.doFullAppendTransaction(WALUtil.java:154) at org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeMarker(WALUtil.java:132) at org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeRegionEventMarker(WALUtil.java:97) at org.apache.hadoop.hbase.regionserver.HRegion.writeRegionCloseMarker(HRegion.java:1103) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1615) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) ... 4 more 2018-02-04 11:10:40,901 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-2] helpers.MarkerIgnoringBase(159): ***** ABORTING region server ubuntu,39267,1517713227316: Caught throwable while processing event M_RS_CLOSE_REGION ***** java.lang.RuntimeException: org.apache.hadoop.hbase.exceptions.TimeoutIOException: Failed to get sync result after 300000 ms for txid=146, WAL system stuck? at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:117) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.hbase.exceptions.TimeoutIOException: Failed to get sync result after 300000 ms for txid=146, WAL system stuck? at org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:136) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:718) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:605) at org.apache.hadoop.hbase.regionserver.wal.WALUtil.doFullAppendTransaction(WALUtil.java:154) at org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeMarker(WALUtil.java:132) at org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeRegionEventMarker(WALUtil.java:97) at org.apache.hadoop.hbase.regionserver.HRegion.writeRegionCloseMarker(HRegion.java:1103) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1615) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) ... 4 more 2018-02-04 11:10:40,901 ERROR [RS_CLOSE_REGION-regionserver/ubuntu:0-2] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-02-04 11:10:40,904 INFO [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegionServer(2350): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 881852416, "init" : 197132288, "max" : 2794455040, "used" : 377064360 }, "NonHeapMemoryUsage" : { "committed" : 124674048, "init" : 2555904, "max" : -1, "used" : 122773816 }, "Verbose" : false, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : -1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 6227078, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 207, "TotalCallTime_min" : 0, "TotalCallTime_max" : 0, "TotalCallTime_mean" : 0, "TotalCallTime_25th_percentile" : 0, "TotalCallTime_median" : 0, "TotalCallTime_75th_percentile" : 0, "TotalCallTime_90th_percentile" : 0, "TotalCallTime_95th_percentile" : 0, "TotalCallTime_98th_percentile" : 0, "TotalCallTime_99th_percentile" : 0, "TotalCallTime_99.9th_percentile" : 0, "TotalCallTime_TimeRangeCount_600000-inf" : 207, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 207, "ResponseSize_min" : 0, "ResponseSize_max" : 0, "ResponseSize_mean" : 0, "ResponseSize_25th_percentile" : 0, "ResponseSize_median" : 0, "ResponseSize_75th_percentile" : 0, "ResponseSize_90th_percentile" : 0, "ResponseSize_95th_percentile" : 0, "ResponseSize_98th_percentile" : 0, "ResponseSize_99th_percentile" : 0, "ResponseSize_99.9th_percentile" : 0, "ResponseSize_SizeRangeCount_100000000-inf" : 207, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 207, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 0, "ProcessCallTime_mean" : 0, "ProcessCallTime_25th_percentile" : 0, "ProcessCallTime_median" : 0, "ProcessCallTime_75th_percentile" : 0, "ProcessCallTime_90th_percentile" : 0, "ProcessCallTime_95th_percentile" : 0, "ProcessCallTime_98th_percentile" : 0, "ProcessCallTime_99th_percentile" : 0, "ProcessCallTime_99.9th_percentile" : 0, "ProcessCallTime_TimeRangeCount_600000-inf" : 207, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 9, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 207, "RequestSize_min" : 0, "RequestSize_max" : 0, "RequestSize_mean" : 0, "RequestSize_25th_percentile" : 0, "RequestSize_median" : 0, "RequestSize_75th_percentile" : 0, "RequestSize_90th_percentile" : 0, "RequestSize_95th_percentile" : 0, "RequestSize_98th_percentile" : 0, "RequestSize_99th_percentile" : 0, "RequestSize_99.9th_percentile" : 0, "RequestSize_SizeRangeCount_100000000-inf" : 207, "sentBytes" : 2283235, "QueueCallTime_num_ops" : 207, "QueueCallTime_min" : 0, "QueueCallTime_max" : 0, "QueueCallTime_mean" : 0, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 0, "QueueCallTime_90th_percentile" : 0, "QueueCallTime_95th_percentile" : 0, "QueueCallTime_98th_percentile" : 0, "QueueCallTime_99th_percentile" : 0, "QueueCallTime_99.9th_percentile" : 0, "QueueCallTime_TimeRangeCount_600000-inf" : 207, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:56841", "tag.serverName" : "ubuntu,39267,1517713227316", "tag.clusterId" : "990c040f-15f4-4bef-a30c-08f39bc273d8", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu", "regionCount" : 8, "storeCount" : 8, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 0, "memStoreSize" : 534358, "storeFileSize" : 0, "maxStoreFileAge" : 0, "minStoreFileAge" : 298656, "avgStoreFileAge" : 298880, "numReferenceFiles" : 0, "regionServerStartTime" : 1517713227316, "averageRegionSize" : 66794, "totalRequestCount" : 32, "totalRowActionRequestCount" : 34451, "readRequestCount" : 9844, "filteredReadRequestCount" : 0, "writeRequestCount" : 24607, "rpcGetRequestCount" : 4, "rpcScanRequestCount" : 19, "rpcMultiRequestCount" : 5, "rpcMutateRequestCount" : 2, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 0, "staticIndexSize" : 0, "staticBloomSize" : 0, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1116962296, "blockCacheCount" : 0, "blockCacheSize" : 819720, "blockCacheHitCount" : 0, "blockCacheHitCountPrimary" : 0, "blockCacheMissCount" : 0, "blockCacheMissCountPrimary" : 0, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 0.0, "blockCacheExpressHitPercent" : 0.0, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 0, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 0, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 0, "l1CacheMissCount" : 0, "l1CacheHitRatio" : NaN, "l1CacheMissRatio" : NaN, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 10545, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 400710, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 19, "ScanTime_min" : 0, "ScanTime_max" : 0, "ScanTime_mean" : 0, "ScanTime_25th_percentile" : 0, "ScanTime_median" : 0, "ScanTime_75th_percentile" : 0, "ScanTime_90th_percentile" : 0, "ScanTime_95th_percentile" : 0, "ScanTime_98th_percentile" : 0, "ScanTime_99th_percentile" : 0, "ScanTime_99.9th_percentile" : 0, "ScanTime_TimeRangeCount_600000-inf" : 19, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 2, "Put_min" : 0, "Put_max" : 0, "Put_mean" : 0, "Put_25th_percentile" : 0, "Put_median" : 0, "Put_75th_percentile" : 0, "Put_90th_percentile" : 0, "Put_95th_percentile" : 0, "Put_98th_percentile" : 0, "Put_99th_percentile" : 0, "Put_99.9th_percentile" : 0, "Put_TimeRangeCount_600000-inf" : 2, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 2, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "FlushMemstoreSize_SizeRangeCount_100000000-inf" : 2, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 45, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "PutBatch_TimeRangeCount_600000-inf" : 45, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 4, "Get_min" : 0, "Get_max" : 0, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 0, "Get_75th_percentile" : 0, "Get_90th_percentile" : 0, "Get_95th_percentile" : 0, "Get_98th_percentile" : 0, "Get_99th_percentile" : 0, "Get_99.9th_percentile" : 0, "Get_TimeRangeCount_600000-inf" : 4, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 66522, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 2, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "FlushTime_TimeRangeCount_600000-inf" : 2, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 19, "ScanSize_min" : 0, "ScanSize_max" : 0, "ScanSize_mean" : 0, "ScanSize_25th_percentile" : 0, "ScanSize_median" : 0, "ScanSize_75th_percentile" : 0, "ScanSize_90th_percentile" : 0, "ScanSize_95th_percentile" : 0, "ScanSize_98th_percentile" : 0, "ScanSize_99th_percentile" : 0, "ScanSize_99.9th_percentile" : 0, "ScanSize_SizeRangeCount_100000000-inf" : 19, "slowGetCount" : 0, "flushedMemstoreBytes" : 267140, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 2, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "FlushOutputSize_SizeRangeCount_100000000-inf" : 2, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-02-04 11:10:40,904 DEBUG [RS_CLOSE_REGION-regionserver/ubuntu:0-2] ipc.AbstractRpcClient(349): Not trying to connect to ubuntu/127.0.1.1:39063 this server is in the failed servers list 2018-02-04 11:10:40,905 WARN [RS_CLOSE_REGION-regionserver/ubuntu:0-2] regionserver.HRegionServer(2371): Unable to report fatal error to master org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: java.io.IOException: Call to ubuntu/127.0.1.1:39063 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$0(AbstractRpcClient.java:324) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:571) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.reportRSFatalError(RegionServerStatusProtos.java:13568) at org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:2368) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.abortRegionServer(MiniHBaseCluster.java:208) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$1(MiniHBaseCluster.java:207) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$2.run(MiniHBaseCluster.java:201) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.abort(MiniHBaseCluster.java:198) at org.apache.hadoop.hbase.executor.EventHandler.handleException(EventHandler.java:188) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:106) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Call to ubuntu/127.0.1.1:39063 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$3(AbstractRpcClient.java:375) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:1) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:423) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:328) ... 17 more Caused by: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ubuntu/127.0.1.1:39063 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.getConnection(AbstractRpcClient.java:352) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:420) ... 18 more 2018-02-04 11:10:40,911 INFO [RS:0;ubuntu:39267] regionserver.HRegionServer(1406): We were exiting though online regions are not empty, because some regions failed closing 2018-02-04 11:10:40,911 INFO [RS:0;ubuntu:39267] regionserver.HRegionServer(1104): stopping server ubuntu,39267,1517713227316; all regions closed. 2018-02-04 11:10:40,913 ERROR [RS:0;ubuntu:39267] wal.AsyncFSWAL(685): The wait for close of async writer is interrupted 2018-02-04 11:10:40,913 DEBUG [RS:0;ubuntu:39267] ipc.AbstractRpcClient(483): Stopping rpc client 2018-02-04 11:10:40,913 INFO [RS:0;ubuntu:39267] regionserver.Leases(147): RS:0;ubuntu:39267 closing leases 2018-02-04 11:10:40,914 INFO [RS:0;ubuntu:39267] regionserver.Leases(150): RS:0;ubuntu:39267 closed leases 2018-02-04 11:10:40,914 INFO [RS:0;ubuntu:39267] hbase.ChoreService(327): Chore service for: regionserver/ubuntu:0 had [[ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS]] on shutdown 2018-02-04 11:10:40,915 WARN [Close-WAL-Writer-0] wal.AbstractProtobufLogWriter(236): Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:424) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:513) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$2(AsyncProtobufLogWriter.java:210) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:166) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:201) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:233) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:143) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$executeClose$8(AsyncFSWAL.java:701) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-02-04 11:10:40,916 INFO [RS:0;ubuntu:39267] regionserver.CompactSplit(394): Waiting for Split Thread to finish... 2018-02-04 11:10:40,916 INFO [regionserver/ubuntu:0.logRoller] regionserver.LogRoller(195): LogRoller exiting. 2018-02-04 11:10:40,917 WARN [Close-WAL-Writer-0] wal.AsyncProtobufLogWriter(146): normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:521) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:144) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$executeClose$8(AsyncFSWAL.java:701) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-02-04 11:10:40,917 INFO [RS:0;ubuntu:39267] regionserver.CompactSplit(394): Waiting for Large Compaction Thread to finish... 2018-02-04 11:10:40,918 INFO [RS:0;ubuntu:39267] regionserver.CompactSplit(394): Waiting for Small Compaction Thread to finish... 2018-02-04 11:10:40,921 INFO [RS:0;ubuntu:39267] ipc.NettyRpcServer(143): Stopping server on 0 2018-02-04 11:10:40,923 WARN [RS:0;ubuntu:39267] regionserver.HRegionServer(1141): Failed deleting my ephemeral node org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/rs/ubuntu,39267,1517713227316 at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:166) at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1231) at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1220) at org.apache.hadoop.hbase.regionserver.HRegionServer.deleteMyEphemeralNode(HRegionServer.java:1586) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1138) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:187) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$0(MiniHBaseCluster.java:186) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:171) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:168) at java.lang.Thread.run(Thread.java:748) 2018-02-04 11:10:40,926 INFO [RS:0;ubuntu:39267] regionserver.HRegionServer(1150): stopping server ubuntu,39267,1517713227316; zookeeper connection closed. 2018-02-04 11:10:40,927 INFO [RS:0;ubuntu:39267] regionserver.HRegionServer(1152): RS:0;ubuntu:39267 exiting 2018-02-04 11:10:40,929 INFO [Close-WAL-Writer-0] util.FSHDFSUtils(195): Recover lease on dfs file /user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/WALs/ubuntu,39267,1517713227316/ubuntu%2C39267%2C1517713227316.1517713230044 2018-02-04 11:10:40,938 WARN [RS:0;ubuntu:39267] util.Threads(115): Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@67dd779; joinwait=30000 java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1260) at org.apache.hadoop.hbase.util.Threads.shutdown(Threads.java:113) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:181) at java.lang.Thread.run(Thread.java:748) 2018-02-04 11:10:40,938 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@67dd779] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(225): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@67dd779 2018-02-04 11:10:40,942 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(121): Starting fs shutdown hook thread. 2018-02-04 11:10:40,942 WARN [Close-WAL-Writer-0] util.FSHDFSUtils(293): attempt=0 on file=/user/zhangduo/test-data/74fb0349-7e94-45e2-a5c4-57748a5c67c3/WALs/ubuntu,39267,1517713227316/ubuntu%2C39267%2C1517713227316.1517713230044 after 13ms java.net.ConnectException: Call From ubuntu/127.0.1.1 to localhost:44491 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732) at org.apache.hadoop.ipc.Client.call(Client.java:1480) at org.apache.hadoop.ipc.Client.call(Client.java:1413) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy28.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:608) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy29.recoverLease(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:372) at com.sun.proxy.$Proxy30.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:1245) at org.apache.hadoop.hdfs.DistributedFileSystem$2.doCall(DistributedFileSystem.java:279) at org.apache.hadoop.hdfs.DistributedFileSystem$2.doCall(DistributedFileSystem.java:275) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:291) at org.apache.hadoop.hbase.util.FSHDFSUtils.recoverLease(FSHDFSUtils.java:283) at org.apache.hadoop.hbase.util.FSHDFSUtils.recoverDFSFileLease(FSHDFSUtils.java:216) at org.apache.hadoop.hbase.util.FSHDFSUtils.recoverFileLease(FSHDFSUtils.java:163) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:147) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$executeClose$8(AsyncFSWAL.java:701) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:615) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:713) at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:376) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1529) at org.apache.hadoop.ipc.Client.call(Client.java:1452) ... 31 more 2018-02-04 11:10:40,943 WARN [Close-WAL-Writer-0] wal.AsyncFSWAL(703): close old writer failed java.io.InterruptedIOException: Operation cancelled at org.apache.hadoop.hbase.util.FSHDFSUtils.checkIfCancelled(FSHDFSUtils.java:330) at org.apache.hadoop.hbase.util.FSHDFSUtils.recoverDFSFileLease(FSHDFSUtils.java:218) at org.apache.hadoop.hbase.util.FSHDFSUtils.recoverFileLease(FSHDFSUtils.java:163) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:147) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$executeClose$8(AsyncFSWAL.java:701) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-02-04 11:10:40,944 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(135): Shutdown hook finished.